33 publicly visible posts • joined 1 Sep 2009

In redneck heaven, internet outages are the American Way


I shot the Fibre....

but I did not shoot the transceiver

3,500 servers go down – so my FIRST AID training kicks in


Hi there

Pat here

@ Joe Drunk:

I guess your sleep habits aren't the same as mine. I can sympathize with the effect it had on your body and mind. Not everyone is built the same.

I hate to tell you this regarding your dynamic, fast-paced environment. I work 24/7/365 and the reason things seldom went awry is simple: I'm dam good at my job, I hate it when things go wrong and hate surprises. So if your experience has been one of high stress, blame and finger pointing, it is unfortunately something I cannot relate to as my experience in the same "dynamic and fast paced environment" has been one where I fix problems and people come to me because I'm the last line of defense and if I cant fix it, it cannot be fixed. Like that day the Dutch Consulate website (which was hosted on a machine that the Hosting company I worked for in this story hosted) went down because a borked Parallels update that borked the server the site was hosted on went so bad, all that was left after the kernel panic was a blinking white cursor of death in the upper left corner after POST because the RAID that WAS the system drive no longer existed. Cue 8hour rebuild of the RAID meta data, RAID, EXT3 FS and Parallel's container (read OpenVZ) object. My boss was so impressed, his only comment was "if I had gotten that server, I would have sent it to be reinstalled and restore from backup". Not one iota of lost data, Dutch consulate website online and functional.


Re: Sounds like 'fun'

Hi all

Pat here

to clarify a few points

- the DC that died, has 3 power zones, one of the 3 zones didn't switch, cue massive server death

- the DC in question has about 10K servers, so we lost about 10% to hard drive failures

- the senior that left as I walked in, was doing more of a transfer of shift than anything else. Having 2 seniors on shift may have helped but what I needed were people to take the tickets and link them to the master outage ticket for the DC to do their work. Also to be honest, I didn't need a jittery and otherwise nervous and tired CSR senior working along already stressed, overwhelmed and tired front line agents. Calmness and cool heads prevail, we can't make rash, disconnected decisions that lead to further mistakes. In a situation like this, having one head instead of 2, like a chain of command in the army, is the preferred way of dealing with a crisis while keeping the communications open and efficient and flowing in a straight line, up and down the chain.

- Management was contacted right away and they were more focused on getting the DC back online. What I omitted (since this is a customer contact center centric story, since that's where I worked) was that at the DC, there was a small army of techs and management working the floor to get things back online. The customer contact center was in an entirely different part of the city. Since management was focused on the DC and the phones were being answered, understandbly management was focused on where the real work was needed. Long live front line!

- Regarding my/the social scene of an overnight shift. Montreal has a very vibrant Rave scene. EDM for the rest of you. If you're not a raver, then sadly, night shift kills your social life. If you are into the EDM/Rave scene, all night dance parties are the way to go. I've met so many people at these parties and made great friends. It's better then leaving work at pub-o'clock, leaving to the club for 21h, returning home at close which is around 3AM for most. All said and told, this makes for almost a 20h day once you come home from the party/club/pub. Where as if your party life follows night shift hours, you wake up at the same time your normally do and go to bed when you normally do, without having to have a massive sleep debt. I found I had less stress and I was getting 10h of sleep on a regular basis. For "day walkers" (read: everyone else in a 9-5) this is something people cannot understand because to them a "social life" ends at 3AM, where as mine only starts around then!

- finally, no mental scars from the event. Other than fatigue, I genuinely enjoyed the experience. I have a strong leadership drive that kicks into autopilot when the poo hits the fan. Plus with first aid training, you're given a heavy dose on how to with stressful and traumatic experiences. Anyone here who knows an EMT/Ambulance Tech can attest to with the training they get.

The 'fun-nification' of computer education – good idea?


If 'funnification' ends up in websters...

the world is doooomed.....DOOOOOOOOOOOOOOOOOOMED!

/queue golden bender.

Amazon yanks SimCity download from store


Why I Got some WINE

and CitiesXL

playing happily offline on my Mac whenever I want to.

‘That’s not art’ says Apple as it pulls nudes from AppStore

Paris Hilton

what we all love.... I <3 BOOBS!

BOOBS! We all love boobs! Everyone need's boobs. Cept for the lads who love lads but even thoes lads have boobs, a bit flat boobs...

just to top it off...

Jay: Yeah. Silent Bob, you're a rude motherfucker, you know that? But you're cute as hell. I could go down on you, suck you, line up three other guys, make like a circus seal.

[Jay makes a rude head gesture and car horn honks]

Jay: Ewww, you fucking faggot, I hate guys. I LOVE WOMEN!

Paris: Because we've already seen her Bulgarian Airbags and much more!

Canadians revolt over draconian internet privacy bill


Im a pedofile and a child molester....

I believe in privacy rights!

do I now get to be part of the special wing in prison?

An NT-powered Windows Phone? Not so fast...


AAPL's Approach...

you just nailed it on the head.

AAPL doesn't make fist loads of money and having done a complete round-end without the curtoursey of a reach around on the mobile industry by not having clearly declared their intentions


AAPL's Approach - love them or hate them, AAPL has a plan, its doing, *uck you all!

Two PROTECT IP sponsors drop support for their own bill


██ ████████ ██████ ██████████ ██ ████ ██ ████ ██████████ ██ .███ ███ This comment has been found in violation of H.R. 3261, S.O.P.A and has been removed.

I have you now! Top 10 Star Wars Xmas presents


Motercycle Suits beat your stinking poster


Who wants style with substance over empty slogans?

How Apple's Lion won't let you trash documents

Black Helicopters

Different from VMS/VAX


I delete the most recent version, OpenVMS wont delete the entire revision history. I have to do that myself. Granted you get to see all the revisions in the same location on the filesystem with a nice little increment tagged on to every previous revision with the current revision the actual file-name instead of a hidden folder. I will also wager my entire yearly salary that no OpenVMS Admin does a full delete of the revision history on every file that gets updated, I know quite a few that say with pride "I can roll back my entire OS to the first day I installed it, then roll it forward"

Nanny state to from DEC/HP? Does DEC/HP really think they knew better then the Sys-Admin Administrating it?

Nanny easily removed from AAPL if the delete function asked to delete the revisions too. I guess this marks the 2nd "bug" in AAPL's delete() function. Be happy you iMac and MacBook/Pro/AIR doesn't have a GPS Chip....yet.

Torvalds dumps Kernel.org for Github after breach


Obviously Authentication....

and security along with the basics of how SSH function are lost on you.

you can have one or the other or both so no redundancy, rather though and complete authentication reset at all levels.

a Sys-Admin you not be, me suspect you are the Troll from above calling OSX and Windows Not secure with no concept of Security-in-depth best practices.

I beleive thy requiereth a hat and coat

did you read the sshd_config file?


Linux haiku

Rootkit, Torvalds, Hack

The trolls need attention, lulz

Windows are now closed

London Olympics journey planner crash effort launched


Outsource to Child Services...

then Give each driver a nice rooted OuchPad with the data files on a flash drive.

Web Based - Check

Interactive - Check

Cheap - Check

Lost Drivers due to Lost Data on Lost Flash drive - Check

No clogged roads around Olympic sites. - Check

OLDfIT Policy at its best

(Olympic Lorry Drivers for IT)

DNS hijack hits The Register: All well


Biting the hand....

that feeds it....


Obligatory hat and coat already donned.

Apple vanishes MySQL from Mac OS X Lion Server


a REAL server OS?

BSD Does not make?

I suggest thy revieth thy BSD and UNIX History. The billions riding on that code is a good indication that it meets whatever you consider a REAL OS. Flashy GUI not considering. SCO-Group need not apply.

I work for a hosting company, Our Russian customers get a kick out of BSD-Derived systems. I also never really hear from them on the support lines either, accept when a drive fails.

would you like to know more?

Thumb Up


would be proud of your VAX roots!

Entire London 2012 Olympics' cultural events database held on Excel


Its only...

SQL Lite anyone? (ok better then VBA granted. Same principal, only the tables are cute tabs!). Be happy that they aren't sending it to AAPL thanks to bugs in their delete() function. Give it to Child Services, I hear they are great keepers of data. Im sure its stored on a Flash Drive to "take off site" for "backup" purposes. Perfect for their security practices!

how long before this is floating around the forums?

Yahoo! downs! mail! servers! for! essential! and extensive! work!


If only...

I do these kind of migrations all the time for my clients. It isnt just as simple as you describe unfortunately. It shows you dont manage complex clusters.

Most of these changes are not just to the front end HTML/PHP/JS they often involve complex changes in the bak end as well.

If yahoo is anything like some of my very large clients, we're talking 10's of LB's and 1000's of Servers, geographically dispersed.

So I'll give you the USB key and the magic RSYNC command. you go update my farm perfctly the first time. HUP HUP there youngan, get me a working upgrade!

Apple stuns Wall Street with 95% earnings surge


Your metal fatigue may vary...

Proportionate to the amount of force you're over-using stripping those screws. Having a proper Jewelers screw driver set for torx, Philips and flat-head and a little patience really does wonders.

Also have you tried taking appart a Dell, Acer, HP, et al. Laptop or the crap-top aka netbook? Same screws and flimsy metal..kinda need it in such tight spaces.

I have taken apart my Mac Mini and sent it from a 512 Core SOLO 1.5ghz 5400 60GB spinner entry level to a 2.4 Core2Duo 7200 Series (4MG Cache) with 2GB Ram and a 146gb SSD. Very nice filer I must say.

all screws played nice, the only brick that shat was the plastic pop pin screws that keep the heat sink on...that was a bitch. I found it entertaining using a Plaster Knife to open the thing, it was different then unscrewing case screws, not to be confused with being a pain in the ass when you're tired though.

I has a Gen1 Nehelam based MacPro with 32gb ram. Running RAID-0 with the RAID card I do CUDA research (so 2x VDO cards) and have about 20 VM's running, I have yet to pull a load of more then 4 on this thing. Only running SETI do I hit 8. Expected total re-sale valule, 3K minimum in 2 years. nice investment if I say so. Better then govt bonds....

and finally my White Macbook 13" entry level, runs SC2 just fine once I set the GFX levels at a reasonable level that the hardware can run. Do I need to see all the cute little lighting effects. No. Not important to PLAYING the game. I grew up in 8bit Monochrome Leisure Suit Larry in the Land of the Lounge Lizards time.

You sir are one spoiled child to complain about GFX over playing the game. Kids these days.




why is our economy in the crapper?

SMOKIN' HOT BABE gets job as computer geek

Paris Hilton


Baby Drill?

Paris because we all know why.

Sarah Palin calls for US to stand by North Korea


The Big Red Button should have STOP in white on it

and to think this dim-witted bimbo could have been VP...now im scared....that she can see Russia from her back yard.

Then again, Clinton actually lost the code for a few months, I'd rather that then backyard nuking.

Most coders have sleep problems, need 'hygiene and care'

Thumb Up

$0.02 for added value to this

I work night shift 23:30-08:00 and during the day I am stressed out beyond belief and get no sleep on a "normal world 9-5 schedule". On my "Extreme Owl" schedule im in bed by noon, up by 22:00. So by following what my body wants, I get 10h+ sleep versus maybe 6 on a "normal world" schedule, maybe the coder does not listen too much?

it also helps that my Social Life starts at 00:00 and doesn't end till 10:00 the following morning.

The weekend has landed. All that exists now is clubs, drugs, pubs and parties. I've got 48 hours off from the world, man. I'm gonna blow steam out my head like a screaming kettle, I'm gonna talk cod shit to strangers all night, I'm gonna lose the plot on the dancefloor. The free radicals inside me are freakin', man! Tonight I'm Jip Travolta, I'm Peter Popper, I'm going to never-never land with my chosen family, man. We're gonna get more spaced out than Neil Armstrong ever did, anything could happen tonight, you know? This could be the best night of my life. I've got 73 quid in my back burner - I'm gonna wax the lot, man! The Milky Bars are on me! Yeah!

9-5 is for the birds.

Apple signals disk free notebooks way to go


@Marcus dubious

I surprises me the surprising amount of oxygen that gets sucked out of the room by the Internet At Large who believe bashing an opinion because it dosnt fit their reality distortion field.

they be-little thoes who speak passionately about something they like, You bemoan that it actually sounds intelligent and see the issue and post not for what it is but for what your reality distorion thinks it is, through your green jaded eyes.

get over yourself, my 3 y/o can make a more constructive argument and she can barely say "dada"

Bull-Horn for blowhards like you!


Jealousy and Inferiority Complex? Maybe Synonymous?

So here you are, reducing something because you find it expensive.

Not because of Engineering you have yet to experience nor understand.

Not because of an AntennaGate public Reality Distorting style brou-ha-ha going viral with comments likes yours.

Not because the "Main Market" has done what it did to the Zune, similar price as the "iPod Clasic" but never understood that rapid, successive refreshes keeps things cool

(random side note, Look up how Jean-Chretien, Canadian Liberal Prime Minister during the 90's (suck it Harper-ites and Neo-cons), he ran this country; the same way Jobs sells his vision and people lap it up like soup, elections every 3 years. Constant brand F5. Funny, business logic applies to politics too, might want to look up social engineering.)

Not because you actually held it, dis-liked it and said no THEN decided to post your drivel.

you bash it because you can't afford it. Whine B**ch Moan and complain, why not save. Now counter with predictable "I no like nothing coming from fake prophet Jobs." *dueling banjos* and we can all go home.

Dont interpret this as pro-Jobs either, he's got the business sense to charge 200% markup on the same pool of chips as Dell, Acer, HP et al. are not so they charge 299 and make nothing. Their Bread and butter is Services like IBM. Everyone wants to be like Mike. The race to the bottom is not a race you want to win.

Rather lacking your post was. Please return the electrons wasted on your post. I'll wait 'till AirGate starts.

ACS:Law's mocking of 4chan could cost it £500k




+1 cowbell as well please.

Indiana judges dismiss girl's nipple exposure appeal

Paris Hilton

But north of the 45 you can!


The Ontario Court of Appeals made it legal for women to be shirt-free in 1996 when it overturned a charge against Gwen Jacob, a University of Guelph student who was arrested for being topless in public.

Crinklaw and Webb say women may have won the legal freedom to be topless in public, but they don’t have the social freedom. They want their event to help desensitize the masses to the female breast.

Paris because Britney isnt an option for a follow-up to her Beaver making an appearance!

Ranting Ohio Republican scares interwebs

Gates Halo

Steve Balmer

Does it better by screaming for people to get up and praising developers.

Bill Gates because he's an angel compared to the Jock running the place now, at least he had a vision that wasn't blurred by beer goggles.

Drunken employee pops cap in server


I shot the Server...

But I did not shoot the Tape Array.

Microsoft Windows glider crashes


There is the Microsoft way to Fly...

And of course it would look like a Monolithic Tower with Wings and crashes on first boot. But an Apple Designed plane would be the epitome of DIY Flight. With Majestically Swooped Wings and a one piece white body. When it did crash we'd be told we were looking at it wrong.

Florida woman prangs car while shaving her privates


I 2nd that!

Playmobil FTW!

This should rival the Swedish Dykes Rompy Pompy!

im sure Billy C and a Blue Dress could make a Cameo!

Microsoft sneak peeks embedded Windows 7


like all the kids are doing

and access to marketplaces to buy phone apps like all the kids are doing. ®

So Microsoft has become a crack dealer with new whiz-bang features that all the kids gotta have dood. see little jimmy there, see how he looks cool...

McRuff The Crime Dog says:Say no to WinMo Kids, its bad, mmmmkay.