Techie traced cables from basement to maternity ward and onto a roof, before a car crash revealed the problem Welcome once again to On Call, The Register's Friday column that shares reader-contributed tales of tech support terror and triumph. This week, meet a reader we'll Regomize as "Geoffrey," who told us about the time he was called out to fix a hospital's access control network – a setup that controlled physical access to …
You missed a category. Some of us "stale pale (male)" Brits may have missed the reference as well. We were at work paying taxes so that you post-boomers could sit at home watching it.
Just returning a favour to those who had allowed us to sit at home a few years earlier than that watching John Noakes, Peter Purves and Valerie Singleton, and the work of the great Gerry Anderson.
You're welcome.
It's always so easy to detect a network loop, but solving it is a whole different matter!
My first question is always "has someone installed a Sonos speaker?", because those bloody overpriced pieces of junk cause network loops by default if you have the audacity to use them with wired ethernet.
I've also had issues with certain mesh-capable access points. A very specific scenario where a cable was bad in a way that allowed PoE to power the device, but the link took ages to negotiate (eventually it did do either 100m or 1g. The device went into mesh-mode because it saw no uplink, then the link came up causing a loop, the device rebooted to return to normal operating mode. On boot the network was down, so it booted back into mesh mode, ad infinitum. That one took a while to figure out.
Also, bless switches with STP, even if they don't fix the network loop, they help you identify the problematic ports saving you *a lot* of troubleshooting with potential downtime.
Fortunately SONOS has started maliciously breaking their products EXACTLY when the warranty expires....remote bricking of even their newest models with "get lost loser" and "well buy a new one then" being the built-in customer service responses. Their popularity is roughly the same as a job unblocking mcdonalds toilets using a rubber glove....
Company is currently going down in flames internally, and trying desperately to hide its collapse from investors with vague statements and (potentially illegal) fudged financials.
"Those wouldn't be from a certain 'ubiquitous' vendor would they?"
If you mean the company whose owner pumped and dumped the stock enough to buy a basketball team; who still can't make a working L3 switch; whose wireless engineers (all the good ones) fucked off a decade ago; who still can't cope with IPv6, who've had equipment banned on every continent on earth (faked compliance tests) and who have no s/w QA at all (fanbois "test" things") then its a distinct possibility ;)
Apparently....
Sonos Problems
Yes, Sonos has faced several issues that have affected its reputation and operations. In May 2024, the company released a redesigned app that was met with widespread criticism due to bugs and performance issues.
This update led to connectivity problems for many users, causing frustration and dissatisfaction.
As a result of these problems, Sonos experienced a decline in sales and faced significant financial losses.
The company has also undergone restructuring, including layoffs and organizational changes aimed at improving efficiency and addressing the issues.
Despite efforts to resolve the issues, Sonos has struggled to regain customer trust and has been criticized for a lack of transparency regarding the root causes of the problems.
Some experts suggest that the company's reliance on proprietary, closed systems has contributed to these difficulties, and propose that an open-source approach could offer a more sustainable solution for the future.
Currently, Sonos continues to work on fixing the app and related issues, but the company's challenges persist
Sonos started off with a really good but limited range of products, namely a wireless mediabox (to play your source material, stored on a NAS, with your hifi system) plus another mediabox with a built in 2-channel audio amplifier, which did the same job, once connected by speaker cables to a pair of hifi speakers.
And it was all controlled by a funky handheld CR100 controller with a great display.
But they got a bit drawn into expanding their range...and by doing so, they stopped updateing the older firmware (they claimed due to limited memory in the devices) and for a while they've been marketing and selling a centre-channel speaker for TVs and a subwoofer.
They also love selling direct, undercutting those retailers who have supported the brand and the margin offered to resellers has always been pretty poor.
The lack of repair capability has also affected them, with customers being asked to return faulty products for assessment, and when told they were not repairable, customers would be offered a rather derisory discount off the full retail price of a new device.
So, what was once a great cutting-edge product is no longer leading the field and they've lost many previously happy customers due to their failure to maintain support.
Vlans and other appropriate (broadcast rate limiting springs to mind) segregation config would have alleviated many of those performance issues, and thus Geoffrey roof visit might have never happened.
i'd be uneasy about running unrelated stuff through a switch in a maternity ward, especially without at least vlan segregation but i'm told i have old fashioned out dated methods of working so........
Never underestimate the data bandwidth of a teenage boys favourite sock! That helical data storage capacity is huge, and they seem to produce it so regularly!
It's only once they start working that they can afford proper latex barriers!
It is said that 90% of people are caused by accidents!
--------> Yup, the latex one please!
You have to know the font characteristics and the precise grapheme: are we talking hyphen, endash, emdash or "horizontal line extension" (U+23AF)?
But if you find it is all tildes then best to seek the help of a professional.
I know a family with 3 kids, where EACH is from the failure of a different form of birth control - the method's failure, not theirs! (One was a vasectomy... done 5+ years before.)
i bet that resulted in some awkward questions
We had a 2mb (wow) infra red link between two buildings. We noticed often it would just go down for about 30 minutes in the morning for apparently no reason.
So myself and the network manager got up nice and early for a few mornings, and saw nothing wrong and the link stayed up. Then one cold morning there it was, the issue staring us right in the face, or should that be blinding us in the face.
On a clear day the sun rose up behind building two directly behind the dish, network went down. As it got higher, the network restored.
Not quite that drastic, but a few years ago we had an optical link with a client, and I had enough time spare to be playing with Nagios and Cacti. So naturally enough, I started collecting link stats from the optical units. When you looked at the graphs, there was a clear diurnal variation in signal strength - but I was never able to pin it down to 1) sunlight into the face of one unit, or 2) tiny variation in unit aim due to thermal movement of the steel structure it was attached to.
Other things to consider include sun reflecting off other things. If it's random enough, start looking at nearby buildings with opening windows that may only "blind" the optical transceivers for a short time when the angle of incidence is just right for 5-10 minutes *if* the window is open that particular day :-)
As for "Geoffrey", I think he did a sterling job of identifying the root cause. I wonder how much longer it would have taken and/or what other rabbit holes he'd have gone down without the info related to him while on the roof though? It wasn't a really obvious flaw and without knowing about replaced equipment and past experience with the supplier, I bet many of us would have a had lot more difficulty find that root cause.
I worked for a company with 6 DCs that were connected to one another over a couple of major metropolitan areas. The drive between DCs could be take some time given the time of day, and most of the drives were more than 40 minutes. On a weekly basis, we'd get calls from users who lost access to an application at a DC that was about an hour's drive from where I worked. The downtime would last about 15 minutes and then the application would come back up. Once we noticed the pattern, the operations manager and I drove over to the DC and waited to see what was going on. As we stood there, a housekeeper with a floor buffer walked into the room, unplugged a server, plugged his buffer in, and proceeded to wax the floor.
Can also be a problem with some satellite positions. For a brief period twice a year the sun is directly in line with the satellite and the ground receiver and drowns out the downlink.
A few years back one of the India stock exchanges would halt trading during that time I presume because some remote offices were on satellite data links. They don't do it anymore. Japanese TV operator WOWOW would also have a similar "Sun Outage".
Yup, sun conjunction, and it hits each site one day twice a year. You can always tell what stations forgot to change their antennas to memory track instead of autotrack. The funny ones were the ones who forgot as they'd follow the Sun, then switched to the memory track to get back on track. The funnier ones were the ones who then forgot to switch back to autotrack after reaquiring, and 24 hours later would veer off again.
There's also the thermal inversion that affects microwave comminications. When the dew point is right, it'll act as a block for microwave communications and your signal will go away for a out half an hour every morning. The fix is to set up dual antennas with one 6-12 feet higher than the other. The inversion will only ever affect one at a time.
We had similar issues at my first support job nearly 30 years ago. It turned out the network blips were caused by the pigeons who roosted up on the building room and their takeoff and landing paths were in the direct line of the optical link.
I've worked at a few places with interesting cross building connections. My favourite being the offices on an Oxford street side road where the connectivity was achieved via using a toy bow and arrow firing a string across the road via top floor windows and a CAT5 cable strung between the buildings, I still have no idea how we got away with that, but it ran happy for a few years before the company owner was caught committing fraud and the company was wound up.
Aargh! the sun! It played havoc with some super fancy touchscreens that had cameras in the four corners of the screen. One camera would be blinded by the light, and the others couldn't triangulate the touch point properly. Blame it on the architect, they wouldn't allow curtains in their fancy new building.
There are sunglasses somewhere in this coat...
There needs to be a Special Place In Hell for the class of architect who designs stuff to look good in the pretty drawings and fuck the poor bloody building users.
Over the years I've worked in dozens of school buildings. You'd be shocked ( maybe not) by stupid, unthinking or simply uncaring designs they can come up with. As examples, in no particular order
A whole glass wall, which looked lovely in the drawings, light and airy etc. From mid-March to mid-October all that side of the school was intolerably hot. In the Summer term the desks near the window were too hot to touch
An outdoor quadrangle that had to be crossed by staff and students between lessons, with grassy sections and narrow paths that met at right angles. Narrow to the point that there was no possibility of even a couple of pairs of people getting past each other- let alone 40 or more kids and a few teachers passing another 40 or so coming the other way all rushing between lessons. As to corners....
Another school with lots of South facing windows- and the nice architects had designed them with blinds (oh goody!) on the outside operated by ropes and pulleys, which inevitably had become useless, twisted, snapped, tangled, knotted etc years before. And any that hadn't were ripped to shreds by the combination of movement and weather. And the inevitable unaffordability of maintenance for these contrivances.
A modern school meant to be designed for the modern age, with the only mains plugs ( single) by the classroom doors. Even 20+ years before this place was built classrooms had needed TVs, OHPs video recorders, projectors etc. And by then PCs were appearing, so they were totally predictable- even if they only expected one per classroom.
A school with a lovely staff work room in the roof space, with a big roof window to let in lots of light. So much light, in fact that apart from the heat ( at least it could be opened- if you could reach it by standing on a chair) anyone in the room would be dazzled.
A school with lovely bright shiny walls and low ceilings- that magnified and reverberated sounds of kids but made the teacher's voice difficult to hear. (I also did a postgrad course on educational inclusion in a university room that was also like that- oh the irony)
Those old Victorian 3 decker primary schools are still working well compared to these wonderful modern buildings. Not just because they were well built and robust, but because the design and layout is good (though not inclusive for disabled kids- don't know if lifts can be fitted.. ).
> Those old Victorian 3 decker primary schools ... not inclusive for disabled kids
They were perfectly adequately designed to be inclusive for the Victorian kids, at least those in a position to go to a three-decker: "Smythe, nip out and round up a couple of guttersnipes, it's time to get Jenkins up to the Art Room".
Of course, that was just primary schools. For secondary, the First Form were all available for fagging duties, including sedan chairs for School Bully and his unwed Filipino women.
Ugh, schools. Back in the 1980s I was in high school, and some blithering idiot chose what they said was a Florida school design. We were quite a bit north of Florida, up where it snowed most years and was freezing cold every year. Large square ring with classrooms in the ring, open in the middle with large courtyards, lockers on the inner wall of the ring. It was really pleasant about a month in spring and about 2 weeks in fall. There was nothing better than telling the teacher that you can't get your books because the entire wall of lockers froze shut when the rain, then sleet, then ice, then snow came through over the weekend. And naturally, the HVAC could not keep up with outside temps so, coats on all day long. It was lovely walking through the rain and wind between classes as well. I guess I should have been thankful it was only one story. Remarkably, the school didn't smell of teenage hormones and lust so I guess there was a large benefit to it.
"stupid, unthinking or simply uncaring designs"
I went into to one school do some PC repairs and the school had only been open for a year or so. They were using the mezzanines as classrooms because there weren't enough actual classrooms for the predictable local demand.
The architect that designed the building I work in is one of those. First, he made the building curved, like 2 bananas back to back. So nothing is square. There's a room that is occupied 24/7/365. The bathrooms are on the other side of the building in the other banana, and it didn't have a lunch room to prepare food. There's a three story atrium with walkways on the 2nd and 3rd stories with bridges to cross from side to side and a glass ceiling with an overhang from the roof on one side that drops filth onto the glass ceiling. Oh, it also has big can lights mounted in the floor that have never been used because they would blind anyone walking there. Those can lights were also installed outside in the courtyard to light trees and bushes and flags. I think you can see where this is going... Yes, there have been issues with water leaking into the some of the light fixtures outside and filling the can until the circuit shorts out for all the outside lights. Apparently the architect didn't think about installing some kind of drainage system under the lights so any water would flow away instead of collecting. There is more, but those are the highlights.
High-ceilinged Victorian school rooms are a nuisance to heat if you're inclined to have heating. Otherwise, I'd like to ask how many buildings of that date (let's say whenever universal education was introduced, relatively late I think) are no longer standing, either from bad construction or because of a pressing need to knock them down and build something better suited to the purpose. What we see is the survivors, some of which are only intact because if you knock them down then you let the asbestos out. According to Wikipedia, "The use of fire-proof asbestos became increasingly widespread toward the end of the 19th century", with a list of building trade uses mentioned. Then it got stopped - in most of the world - because it causes nasty disease, mostly in the lungs and slowly fatal if you're lucky. "Outside Europe and North America", says Wikipedia - and specifically in Russia - asbestos use continues. Hmm.
Jusf up the road from here there is a school but in the early 1990s which isn't a dreadful design, with large classroom windows, designed to give children a nice view out across the valley.
Idiot builders quite literally built the thing backwards. 180 degrees out. Large windows now face the side of a hill, any class doing sports in the playground causes noise disruption and from the road outside all you see is a huge expanse of roof, a couple of Veluxes and the kitchen extractor fan(s).
M.
We had the opposite problem, laser link between two buildings which failed due to freezing fog (the beam was refracting off the ice crystals in the air.
We also had an architect designed special with bare concrete walls in a meeting room, the poor installation guy was driven mad trying to get the av mics to work without reverb, eventually I think they did go out and buy some heaven curtains to hang on the walls as sound deadening.
During the second world war the RAF kept seeing on their new fangled RADAR huge numbers of German aircraft forming up for a raid over the continent early in the morning.
This led to us readying/launching many many squadrons who then couldn't find any formations and the RADAR signal disappeared.
When it was realized that they were seeing the Sun, scientists went "that's interesting", radio astronomy was born.
Hmm. I think that versions of this problem that I've seen described previously included birds perching on your hardware - that's what I expected - and trees either breaking out in leaves seasonally, or simply growing tall enough to stand in the way of he signal. I don't think we've had "a queue of hay wagons at the traffic lights" before, but cows maybe in a field that was cowless when the link was installed? And I don't remember the one with fog, either.
My last job was at a large boarding school that had buildings spread across the village where it was based, these buildings were connected back to the IT department using fibre or CAT6
When Barclays closed its branch, our facilities department decided to buy it and use it as their office.
There was no budget to physically extend the network to this building, so we decided to use a wireless bridge, with one end of the bridge on the outside of the nearest boarding house, the other end on the outside wall of the new office.
We fired up the link on a Saturday afternoon and everything was hunky dory.
The staff moved in on the Monday and chaos ensued. The network was dropping at seemingly random intervals, checking the logs didn't tell us anything, so the staff were willing to live with random drop-outs, except sometimes when the network dropped, the printer died, needing a reset.
Eventually, we decided to sit in the office and wait for a drop-out, to see what was causing it.
The cause was farmers' hay wagons, they were high enough to block the signal as they drove past, and if they were stuck at traffic lights, the network was down for long enough for the printer to demand a reset.
That weekend we hired a cherry picker and raised the bridge on poles, which stopped the drop outs, and provided a resting place for a peregrin falcon while he watched for mice in the school grounds...
Never underestimate the helpfulness of users who see a lonely network cable end which "should obviously be plugged into something". Said network cable was there for a laptop user (back in the days when they weren't 2-a-penny), and the "something" was a spare port on a small switch in the corner of the office because the developer put too few connections in. Cue network falling over, and me getting sent to site when their new techs were unable to diagnose the problem.
I possibly have posted this before, in which case my apologies for the repetition.
Many, many moons ago when I was not long in sorting out weird IT issues, in the days when in-house Exchange was a thing, I was sent to investigate an issue with an Exchange server that would connect and disconnect. This was for the London office of a Japanese bank, so I pitch up and start running diagnostics, the server is fine, no obvious issues, but yes, connectivity was up and down like a whore's underwear. My investigation was not helped by being continuously interrupted by a senior member of said bank who was 'involved with a multi $100 million deal, and how it was Imperative that his email worked and don't I know how important he was and how vital the email worked NOW!
Alas, no I didn't have a clue who he was or how important he was, I was however aware that perceived importance has no bearing on finding a solution. What I did was to watch the 'blinkenlights' on the switch and noticed one that looked a bit odd*.
Disconnected it and the connection to the Exchange server was miraculously restored. All good, job done, except I thought I should find the root cause, so tracked down the other end of the connection, to a wall port in a little used office with a PC plugged in.
Above the port was a little sign which read "FAULTY, DO NOT CONNECTED ANYTHING TO THIS SOCKET"
Now I'd love to now say that the person who did it was the aforementioned VIP, it may have been, but nobody owned up - still we can't have everything, can we?
* Yes I know, but many (literally) decades of experience has now told me that this all was screamingly obviously a bridging loop, or (love the expression), a jabbering NIC; but at the time, we all have to learn!
> screamingly obviously a bridging loop, or (love the expression), a jabbering NIC
Reminds me of the network I modernized from BNC co-ax to this new twisted-pair stuff. On nearly no budget so lowest-price network cards. Historically it was at the point that asian copycats could make a fine card 99 times of 100. But then there is that 100th card. Worked fine the first week. Then hours of no useful network. Most of the PCs were back to the wall, and the hub (not switch!) in a box, so didn't see the blinkinlights. But when box was opened it was clear there was a panic going on. And then sometimes it would clear up perfectly. Well, it was bad at times when Judy was in her office. Her net-card jabbered bad.
Of course if I'd labeled the lines I woulda blamed her a lot sooner.
how it was Imperative that his email worked and don't I know how important he was and how vital the email worked NOW!
"I understand how important your email is and how frustrating it must be for you. If you want somebody to listen to you vent your frustration I'm quite happy to do that if that makes you feel better. But wouldn't it be more useful in the long run if you just let me get on with fixing it?"
This post has been deleted by its author
A few years back I wrangled a fax service for a health care IT company. (Long story omitted about how fax messages were considered secure where email was not.)
One of our hospital customers was a long-established institution in a cold climate. Every mid-March the fax messages our system sent them would slow down dramatically for a couple of weeks. It seems all their incoming fax messages slow down, and our service was a convenient neck to choke.
At any rate, the problem cleared up as April warmed things up.
I suggested they have their hospital telecom krewe trace their phone lines. He opened up a conduit and water poured out.. It turned out their fax machines were on copper POTS lines, in conduits. In winter those conduits filled with ice. In March the ice melted and intermittendly shorted the lines. POTS is constant-current, so the shorts drove the power supplies pretty hard.
By April the water evaporated and faxes started working again.
It was the nifty little system that detected the 90th percentile of minutes per page that helped identify this.
Fun stuff.
My village is on the top of a hill with a 1:6 valley down and then hill back up the other side where the exchange resides about 2 miles away.
The phone lines had been OK for more than 80 years, but then in the 1990's some of us started using modems. Speed was alright in the early years and then many of us noticed that as the modem technology started getting much better, our transmission speeds didn't. Over the next 10 years or so, we and BT worked out that some lines were better than others, and when you complained you were given a better one at the cabinet in the village - until the point that the 25 better lines had been used and when someone got a better line, another person was dropped to a bad line.
I caught an engineer doing this once, after I ran up the road when my line dropped, and he confessed. Being very interested, I didn't bore him out, but commiserated and pumped him for information. It turned out that the telephone lines were paper insulated and installed in a lead pipe stretching to the exchange. The ones in the middle of the cable were the faster ones. He showed me where the lead pipe could be seen at the bottom of the valley (going over a stream) and he agree to put a little nick in it. It ran with drained water for at least 2 weeks! Unfortunately it didn't make any noticeable difference to either the achievable speeds or the number of faster lines.
I worked at Guys Hospital in London yea many years ago. In medical imaging and (then) high bandwidth networking.
We had a research group from the medical school decanted into offices in a tower block acrosss the street (Weston Street).
A laser link pointing out a window was set up to connect their LAN.
One London day was foggy.... sure enough we watched as the fog grew in depth , gradually creeping up the building.. till it reached the laser. No more connection!
I spent an afternoon following a PCM link through (almost) all the DDF of a $TELCO building housing several PSTN exchanges once.
That PCM link from Hell didn't want to go up ( at Layer 2, so we knew the issue was physical ) and was a link towards another $TELCO building a few Km away also hosting several PSTN exchanges...
In the end it was a misalignment in one pair on one of the DDF, 20 second of unwrapping and rewrapping the cable and the issue was solved once located.
I’m sure many of us have found that determining the actual root cause of a problem is the hard part; the solution then becomes obvious.
Unfortunately so many people think they’ve solved a problem when they’ve made one of the symptoms go away.
Politicians spring to mind as examples of both this type of person and the root cause of so many problems!
Yes, indeed, as I would think most of us on here would want to.
Unfortunately most of us on here might not fall into the 'normal people' category!
Problem - this alert is going off. Solution - suppress the alarm! Well until something really bad happens, but then the people you mention have probably moved on and it's no longer their issue.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
