Fed up with slammed servers, IT replaced iTunes backups with a cow of a file Greetings, gentle reader! What a joy it is to see you, as once again we delve into Who, Me? wherein Reg-reading raconteurs recall rascally rebellions at work. This week our rapscallion of record is one "Roy" (not his real name) who once served as IT manager for "a well known environmentally conscious international organization …
Genius indeed: a cow has four stomachs - and the crud (cud) was following it's own four-stages: iPlayer --> laptop --> server --> backup. Our man was simply...err...reverse-engineering the process in delivering back the moo - the raw material, if you like. The users should have thanked him for such dedication ;)
Well.. I've just added a note to my dairy.
Today is the day we herd all the cow puns.
Just keep scrolling and pasturise they go, each delivered with silent aplomb.
Udder geniuses everyone of you... you've milked them dry.
We should celebrate this. Let the cow bells ring!!!
(Too much??)
A former boss (a long long time ago) forced me to write down IT proposals on paper before he'd evaluate them. A pointless exercise when you know they do not read the paper. Together with the PFY we wrote a new proposal that ended with:
There are no direct costs involved to this project, but there may be some non-IT related monetary expenditures while performing the tasks as explained.
Of course, without reading, the boss said OK for the project. The PFY and I went straight to the store and bought as much cola as we could carry (a lot) and I presented the bill to the boss for a refund, mentioning the appropriate wording from the proposal. The boss gave me a bunch of paper to repay the debt. However, it was more than owed and I did not have any change. The boss said: keep it, that is for the next time you pull such stunt.
:-)
This post has been deleted by its author
I was a systems administrator for a small accounting firm (100 accountants).
We had a contract that each employee re-signed every year laying out the IT policies.
One policy was that no personal music was to be stored on company computers or their network drives. This was back in 2003 so mp3 was running rampant, and.much of it acquired from shady sharing sites.
I had permission from the senior partner of the firm to do random scans of the network and delete any mp3 files I found. No warnings to be given to the staff, just find and delete.
It took a few months but the accountants learned their lessons.
:)
One policy was that no personal music was to be stored on company computers or their network drives
At one point I had capacity management as one of the tasks I had to do - in a time when storage on the network was expensive. We also had a 'no music or movies' rule (apart from the Marketing team who actually made short films and used music to add sound-tracks.)
So we would regularly scan the network shares (excluding the Marketing share although they had been warned, from one of the Very Senior directors that misusing their freedom would result in Bad Things..) and email individuals where we found music. They would get one warning.
Next scan, if it was still there, then the delete button would get used and an email would be generatted as to why. If the content came back, their manager would get involved then, if it came back again, the next one up, with an invoice for the storage used.
it worked reasonably well - although we did have a few of the scraming "don't you know who I am!" type phonecalls from people whose self-importance vastly outweighed their value to the organisation.
Not my favourite job! Especially when you'd discover that someone had managed to make multiple backups of their XP machine to the file server and then complained when it ran out of room..
Similar experience at a Contract Research Laboratory that had storage servers, when hard drives were still measured in megabytes.
No free space... Add more storage... Free space evaporates... No free space...
One day the IT manager threw up his hands, went rogue, and started perusing people's home directories. And found...
Games. Thousands upon thousands of games.
A games purge and a tersely worded memo fixed the storage problem.
"Games. Thousands upon thousands of games."
We had the same issue - personal space on the server was for work-related files only, no private documentation, no mp3 files, no games and no .exe files.
we used to do a sweep once a month and issue emails to people who stored personal stuff on the server telling them to remove it. Music and executables were deleted, although when work was slack we used to run quake over the network (before deleting the naughty files from the users' drives)
> I had permission from the senior partner of the firm to do random scans of the network and delete any mp3 files I found. No warnings to be given to the staff, just find and delete.
I wanted to do the same thing while working for a .govt.nz - just before pulling the trigger, fortunately for me, someone pointed out some of the Policy Advisors had accessibility issues, so they had (approved) dictation software, which, of course, saved as .mp3 ....
It proved too hard to search "bad" mp3s from "good".
In the end we just scanned for large file/folders manually and culled obvious music files.
I was mooved by the udder genius of this story. Simon would be so proud, although I strongly suspect he'll be using a different sound or combination of sounds.
"Hey, why have my moo-sic turned to cow?"
"Have you read the memo?"
"What memo?"
Y'all know what? This need more... COWBELL!
I'll hoof it out of here now.
Y'all know what? This need more... COWBELL!
This is an example of an expression originating in comedy which was so successful it made it into common parlance (like a large amount of Monty Python) - and then its origin sadly faded away in time, other than in limited clips which DO refer to it as one of the best SNL sketches ever (I would agree), but do not show the clip in its entirety which is a shame - probably to avoid the copywrong monstern wandering around and striking randomly to pretend it's any good. There should be a place where such things continue to live in full, with permission of the originators.
I don't understand this. iPods synched via iTunes, not directly with files on disc and certainly not with "iPod backups", so the files being backed up were the music files in the iTunes library. Changing the track in the company backup would only change the track on the user's PC (and iPod if they synched) if they did a recovery from the company backup to their PC, wouldn't it?
Am I missing something.
It does sound a bit odd; I'd guess that perhaps people were using their work laptops as their primary iTunes machines?
I do remember, back in the very early days of compressed music, we had a local unix sysadmin, who had a thing for "underground" D&B music, which came on home-made CDRs with inkjet-printed covers.
As he had a lot of these CDs, he decided to import them all onto his work Windows PC, and then get rid of the physical disks. And he did the sensible thing of putting them onto a separate drive, so they were safe in the event of a windows reinstall.
However, these were the days when DRM was all the rage, and whatever software he was using (I *think* it was the built-in Windows stuff, though it may have been RealAudio?) defaulted to encrypting all of the audio files with a key which happened to be stored on the windows drive.
So one day, the inevitable happened, and our poor sysadmin found himself left with hundreds of files which had effectively become complete junk...
(assuming Windows laptops - 2000 or XP) I would guess the iPod synced its files into the roaming part of the user profile. This is copied to the server roaming profile folder at logoff and copied back at logon. This would also have slowed down the logon process if each user had a couple of GB of MP3 files that all need to be shunted across a 100Mbps LAN at the same time each morning..
Alternatively, redirect the users' default file location to their server home folder and then sync using off-line files (although that was really not a good idea in practice, as it only works properly during carefully crafted demos).
If memory serves then I think the files were on disk, at least for the local library. The ITunes "library" was in effect a big XML file that logged the location of the tracks and some metadata about the them, like your rating (it used to be out of 5 rather than like-dislike). It also logged the playlists and such.
> The ITunes "library" was in effect a big XML file
At least these days, it's a binary file with an ITL extension.
You can export your library as an XML file, but there's a few potential gotchas with this.
In the first instance, if you attempt to restore this XML file, there's a few bits of metadata (number of times track has been played, date that the file was imported into iTunes, star rating, etc) which don't get reimported.
Secondly, iTunes will merrily generate the XML file using metadata from your tunes. And it doesn't attempt to validate or sanitise said metadata.
So, if you have files containing characters which can't be encoded in XML, iTunes will happily export an XML file which it will then refuse to import...
iTunes would also decide to overwrite the copy on the iPod or duplicate the backup with something not necessarily quite the same for no apparent reason.
I ended up with three copies of half my music on the backup drive before spotting that was happening. Cleaning up that mess took ages.
Worse, it occasionally decided to replace your own tracks with something else, much to the ire of a few friends of mine who discovered it had erased all of their self-written, self-recorded music. Took them ages to restore from the CD-Rs.
I never did understand why it did that, other than Jobs hubris.
They seemed to sync themselves with what was on the PC pretty enthusiastically in the early days, in my recollection.
I recall the school where I worked at the time getting some shiny new servers in summer 2003 with a whopping 72GB of storage on the main file server (minus the capacity needed for the Windows system drive, which was probably about 8GB at the time for Server 2003). After the old 18GB NT4 box, that seemed pretty roomy. All staff were also issued with laptops for the first time, rather than having to share a departmental desktop.
A few months later, the free space on the file server suddenly dropped from a healthy 40-odd percent to just 1%. I deployed SequoiaView that evening to see where all the space had gone and immediately spotted a 25GB chunk of media files which turned out to be someone's iTunes library. As it wasn't a music teacher, who might have been able to justify having an extensive music collection on the system (although I would certainly have approached them and arranged to reconfigure iTunes so that the files were stored locally on their laptop, as indeed I did when a boy at another school where I volunteered did a similar thing), and the situation was a bit of an emergency as people would soon be getting "disk full" messages when saving, I simply deleted the folder. No problem, I thought, it'll still be on the iPod and I can go and see them tomorrow to move the library folder before it backs up again.
They came and saw me first! They had plugged their iPod into their laptop when they got to school and it had promptly synced from the empty folder!
I made the appropriate soothing and apologetic noises while pointing out that they had nearly "crashed the server" (easy to understand even if not terribly accurate) and I'd had to take emergency action. Thankfully, they were very understanding and agreed to let me move their iTunes library before they restored their collection to it.
I filed that bit of knowledge about iPods away under "useful, for good or evil".
In the mid 1990s, I worked in the USENET admin team for a fairly large (5,000 employees+) Canadian based corporation.
Bandwidth was metered in those days, and rates differed by usage and time of day. Much of what our team did was automate tasks to optimize bandwidth usage so that our Toronto to New York backup ran at 3AM, when the bandwidth cost was under 10% of what it would cost to run at 3PM, that sort of thing.
Although the web existed, it will still in its' infancy. Most of the traffic was FTP, and there were many Gopher, Archie, and Veronica users within the company. There was also USENET.
Originally, USENET was brought in as a programming resource for the software developers. We brought in the Comp, News, and Sci hierarchies to start with. Then Misc got added, and Rec, because (a) a lot of the users were book nerds, and (b) using Rec groups got them interested in USENET enough to learn it, and then they started using the News and Sci groups for their actual jobs.
But then, there was Alt. The Alt.* hierarchy was (and is) a snake pit that we tried to keep far, far away. Unfortunately, it was inevitable that some particular newsgroup was needed from it, and it wasn't easy to bring it in without bringing in the entire hierarchy, although we tried. We wanted to restrict it as much as possible, but were overruled, and so the entire thing came in.
USENET bandwidth usage exploded. It went from being ~3% of company bandwidth to ~70% in about two weeks. What was worse was that it didn't just increase by a factor of 25, it was also being used during peak business hours, when bandwidth rates were highest. Our $800-$1200 a month bandwidth cost, that we'd been trying to get under $500 a month, ballooned to something like $18,000 one month, and was projected to be over $30,000 the next month.
Management freaked and ordered an investigation.
What I found, not surprisingly, was that the bandwidth was almost all USENET, from the Alt.* hierarchy. Specifically, the Alt.sex.* hierarchy. Yup, people were downloading porn videos. Instead of 2kb text articles, lots of 40MB video clips were being downloaded. Repeatedly.
The next step was to identify who. And lo an behold, 90% of the usage was attributable to six users. Five users were responsible for about 20% in total, with the other 70% being from a single user. A single executive. An executive of the "reports directly to the Board of Directors" variety.
Awkward....
So, without naming names, we sent out an email blast to world+dog in the company, with a USENET traffic graph, showing how much was being transferred from the different groups.
Of the top 20 groups, 18 were alt.sex groups. Fortunately, there was nothing illegal, and we didn't have to deal with child porn or anything like that, thank god.
Unsurprisingly, when the users realized that we could see the login credentials and knew exactly who was transferring what, our bandwidth went back down to about $1100 a month again.
So, without naming names, we sent out an email blast to world+dog in the company, with a USENET traffic graph
One of my orkplaces we had a packet-switched connection to th outside world with fairly strict policies about what people could and couldn't do.. (this was in the days of Napster - so that was the biggest "don't do" of the lot).
One day I noticed that our link was pretty saturated and the devs were complaining that with code sync jobs with the Paris HQ were failing and that email was very, very slow.
So I fired up the trusty Packeteer to see what was going on - to find that rtsp was taking about 80% of our (fairly expensive) bandwidth. Turns out one of the devs had discovered that he could stream Radio 1 (can't remember whether it was a legitimate stream or a pirate one). He was warned and stopped - but as it turned out, not for long.
Sure enough, in a couple of days the problems resurfaced, and remonstrating with him didn nothing and his manager didn't seem to care.
The Packeteer was a wonderful appliance and had a very useful function - you could control bandwidth allocation on a per-protocol basis..
So, after a few days (and with the nod and wink from my boss) I reduced the bandwidth usage of rtsp by 5%. And by another 5% the next day - until it would eventually just sit there 'buffering'.
Because it wasn't an instant cutoff, he never seemed to realise what we did. To rub salt into the would, we managed to find an old AM transistor radio and put it promenantly on his desk - at which point he got severely told off by the electrical safety team for having a non-PAT tested device. It wasn't planned that way but was the final cherry on top of the cake.
Ah yes, the old alt.sex.(how did they misspell "binaries" again?) hierarchy. The bane of many Internet uplinks over the years. I don't know how many mornings I'd be greeted with a wall of green on the MRTG graph, and know that it was time to update the settings on the news server.
I used backup software which had a remove feature. I could, and did, have it look for specific files and types of files in specific folders and nuke them. Certain people insisted on playing Doom, Wolfenstein, and Marathon on my network, and _not telling me_. Say bye, boyz…
Now, those who were polite and at least asked first, they got their very own partition which was not visible to the backup app. As long as they didn’t abuse the system, I had no problem.
Parking lots of iTunes files on my servers would have been abusing the system. Those files would have lasted one day. Just one. Any complaints would have been met by my showing the abuser a copy of the system usage protocol, which they had allegedly read and signed.
When I worked for an bank, we had lots of AV and mail AV to protect our stuff.
We also had lots of issues with file server storage always being low - back when 100-200 GB of fileserver space was a huge amount
Whilst on holiday, we had that virus pass thru that zero lengthed all media files, that came from our owner back as we didn't (until then), have mail AV on those links as we thought they were trusted.
Anyway, we recovered about 70% of our file server disk space pretty much immediately (only reason I knew was someone called me whilst I was in France).
When I got back we got so many requests for restores where when checked responses were "please provide a business justification to restore madonnas greated hits" or some such.
only area that got media restored was marketing.
We were much more aggressive in identifying and removing music/films without notification after that.
I remember that a version of iTunes would delete music files from local storage and use the "cloud" version instead. It worked from the song title and artist, and so it upset people who had an alternate or rare version of the music, and found that it had been deleted forever and replaced by a pointer to the mainstream version.
I remember being amazed that everyone just accepted that user files could be irrrevocably deleted in such a cavalier fashion. "Hey, the thing I bought to organise my collection has just destroyed that collection! Hey ho, never mind, I shall continue to buy more of their products forever."
> The Smiths split up 14 years before the first iPod was launched.
> Immediately made me think of Atom Heart Mother - but that’s my age.
Today I ripped a track from 1975 to deploy on my 2023 Toyota. I should rip that 1938 track too. Don't tell me my music is too old for my player.
> 90% of the usage was attributable to six users. Five users were responsible for about 20% in total, with the other 70% being from a single user.
Back when the university had ~~20k connected machines, the net-ops compiled a weekly Top Ten of high-traffic users. Known data-suckers were white-listed, but our 'Connie' shot from #9,999 to #9 overnight. Practically computer-illiterate. WTF was she doing??
> ...one of the devs had discovered that he could stream Radio 1
Homesick for Canada, she had discovered CBC online streaming. Left it on in her office. Day and night. At maximum datarate.
"We" hit the Top Ten again a few years later. 'Mike' hit the chart on a slow week. So not huge data but still odd. I confirmed that 'Mike' ran the dept. computer lab and may have been downloading updates. My net-ops contact sent a couple URLs and warned me to close my door. YOWSA! Even the home pages (before the "18 years of age?") were hard-core porno (that was still a new thing). The explanation involved a visiting nephew left alone in the office for a few days. I didn't know 'Mike' had a nephew, but we agreed that 'nephew' would limit his access.
Used to have the same issue with the iTunes library in users Windows profile which was stored on a server getting clogged up with music.
The IT manager at the time had a policy of "if its clearly music, just delete it. No warning, no notification, just delete.".
Then to prevent it coming back we just denied the user write permissions to the folder.
I was testing network switches at the time and wanted to load the devices.
I had many carefully curated video streams stored as MP4.
When all ports were fully loaded with the videos, they made a pleasing pattern across the 8 PCs monitors. (All turned away from the rest of the team to avoid annoyance.)
Any disturbance in the traffic flows caused a disturbance in the pattern which was easily spotted.
Two of the PC's went bang overnight, and when I started to rebuild them from the archives on the server, I found that IT had deleted the MP4 files "as there was no legitimate company need for video files!" Luckily, the 8 test PCs were clones, so I had local copies.
Years ago, when I ran a student computer lab, one of our techs came with with a script that, every night, logged in to each machine, copied every image file to a shared area on his machine, then deleted the original.
He didn't tell me he'd done this, and when several of the applications we support suddenly started failing randomly, I started investigating. I found that they were missing all the image files. Important because they were what was then called "multimedia" software, so relied on Image files being installed.
When I asked the other technicians if they'd noticed any missing files, this technician admitted to creating this script, and justified it by saying that he was trying to stop the hard drives filling on the student machines. Of course, that wasn't the real reason. It was a side benefit. The real reason he wrote the script is that we knew that , particularly during quiet times, the sites students visited were often quite pornographic. He used to build a nice collection of these porn images, write it to cd and take it home and use it as you'd expect.
That may have existed back then too for really expensive storage. But for more than ten years the standard cheapo fileserver offers classification by extension, and simply deny write access. The better ones can look at the "magic" too to get the renamed versions.
I had a problem with storage on my servers being constantly full, and not because we had underspecced the storage. Turns out one of our users had stored his p0rn collection on his user drive, complete with category folders, luckily no kiddyP but still not what you expect in a professional environment. He was tapped on the shoulder and told that it needed to be removed COB next day or it would be deleted. We also chatted to a number of staff that were storing music and videos in their user areas. Our storage usage dropped by half.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
