* Posts by Jim 43

55 publicly visible posts • joined 13 Jun 2015


Surviving Hurricane Katrina: A sysadmin's epic DR (as in Didn't Realise) odyssey

Jim 43

Little Update

Sadly, after ten years, the marriage didn't work out.

On the plus side, with nothing holding me in the US, I decided to move someplace with great weather and political stability.

I now make my home in Edinburgh.

I still suck at evacuation planning.

Jim 43

Re: Don't forget

Mike Barnett has some serious stones and I was glad to be reading his accounts from a safe place since he was holed up about two blocks from where I worked at the time. I can't imagine enough money that would have convinced me to trade places with him.

Jim 43

Re: "I never got a chance to speak with him again"

It was well over a month before I thought of it again. At that point, given the time that had lapsed, I just let it lie. Thinking about it now I feel like an ungrateful prick...

Jim 43

Re: DR is like backups...

80+ people distributed across the country. I was also a billable asset and spent a small amount of time in the field and time supporting other consultants.

When you're up to ass in alligators it's hard to remember that your primary mission is to drain the pool.

As it turns out, they had no trouble hiring someone to replace me. There's always some idiot out there that thinks they can do it all. Just look at me :)

I'm much happier working in a place with 40 SA's across three shifts. I'm too old and bitter to go back to a situation where it's just me and Google running everything from printers to servers.

Jim 43

Re: DR is like backups...

Yeah, I didn't have the time to do it right. But even if I could have found the time I didn't have the experience. I inherited most of those systems and didn't understand what was needed to recover or rebuild them. There was no documentation or change control. The tape backups were good but no decrypt keys meant that they were functionally useless. Realistically, even if I'd had the keys I wouldn't have been able to restore them as they were going onto new hardware, and, having never tested... You get the idea.

As to today? On each of the ~1600 machines I'm responsible for there's a job that runs three times a day to validates that files are being backed up to the master catalog server. We've got a dedicated enterprise backup team, but I still make sure that I've done everything I can to ensure things are working. Every year we have a DR exercise and every year I volunteer to be on the team.

Jim 43

Re: I'm very sorry

No new people, no promotion, no raise. I stuck around long enough for the company to recover and moved on to truly better things.

I got a wife I love very much out of the deal, so I figure I'm ahead.

Jim 43

Re: Refugee sex

It's also possible that the quoted bit was painstakingly crafted.

Jim 43

Re: rewards?

Nope, not a thing.

Jim 43

Re: I'm very sorry

No sympathy needed. Ivan WAS a huge wake-up call for me and if Ivan had gone the way Katrina did then I doubt that the company would have survived. As it turned out, while I wasn't able to do anywhere near as much as I wanted to, it was enough to get us through.

I tried to present this story as it happened without trying to cover up my mistakes. But then, that's the point, we don't learn anything unless we focus on our errors. An AAR with all 'sustains' is a crap AAR.

Sysadmin jeered in staff cafeteria as he climbed ladder to fix PC

Jim 43

Re: So ...

Three times you've had C-level fired for plugging unauthorized kit into the network?

Please come up with more believable lies.

$310m AWS S3-izure: Why everyone put their eggs in one region

Jim 43

Re: A third option

I like the Azure storage redundancy options as well -- It's one of the places that I think they beat AWS. With Amazon S3 you only have two redundancy choices out of the box: Standard and Reduced Redundancy.

Standard claims 99.999999999 data durability and 2 concurrent facilities (in the same region).

Reduced Redundancy claims 99.99 data durability and single facility.

If you want your data available in multiple regions then it's up to you to write that data to multiple places (buckets or containers). What's more, a lot of people get confused about this as the top of the console page says "Global" where you would normally select a region (I can forgive casual users for thinking that their data is stored in multiple regions by default). If you actually read the docs (or create a storage bucket) then you should understand that each storage bucket resides in a single region.

You're Donald Trump's sysadmin. You've got data leaks coming out the *ss. What to do

Jim 43

Government (and most corporate) orgs have their own trusted CA -- these are added to the OS cert chain. Outbound port 80 and 443 connections are routed through a proxy server which serves as a man-in-the-middle. Instead of your browser seeing the webserver SSL cert you get the dynamically generated cert from the proxy server and since it's using a trusted (by your OS/browser) CA, you don't get any errors.

FBI drops bombshell, and investigation: Clinton still in the clear

Jim 43

Re: Early vote??

When a cabinet member tell you to do something, you fucking do it or you're gone. You'll be fired two words into "I'm refusing for your own protection".

And! it! begins! Yahoo! sued! over! ultra-hack! of! 500m! accounts!

Jim 43
Thumb Up

It's going to be an expensive Fall for the orgainization.

Well played!

Sysadmin sticks finger in pipe, saves data centre from flood

Jim 43

Re: The Power of Power

I'm guessing oven controlled crystal oscillators. They drive the radios transmitters that play Jefferson Starship

Use Brexit to save smokers' lives and plug vaping, say peers

Jim 43

Re: Ah, e-cigarettes

I've heard them referred to as douche-flutes by many of the twenty-somethings I interact with. It's a wonderfully evocative name.

I'd still say that smoking analog cigarettes is more effective at birth control.

FBI won't jail future US president over private email server

Jim 43

Re: Female Privilege

Gender has nothing to do with it, political party has nothing to do with it.

20+ years of political favors and debts is all that matters here.

Hillary Clinton: My promises to America's tech industry

Jim 43

There's a lot of horrible stuff in here as well

But first, this could be promising

"She will also promote open-licensing arrangements for copyrighted material and data supported by federal grant funding, including in education, science, and other fields."

-- on to the crap

"People of all ages need continued access to a range of higher education and training opportunities—early career, mid-career, and even late-career—so that they can keep up with changes in technology and industry" -- What a fucking joke. Mid career and above technology workers should be making enough money that they're at the bottom of the list for government help. Frankly, if you haven't figured out how to keep up with changes in your field, you're in the wrong field.

"Employers also need a better mechanism for communicating to job seekers and educational institutions what sorts of skills and competencies they are looking for." -- Really? There's no possible way the US Federal government can help with this.

"Hillary’s College Compact dedicates $10 billion in federal funding to enable students to participate in promising new programs—such as nanodegrees, accelerated learning programs for computer coding, career and technical training, certificates for “specializations,” and online learning." -- I suppose this will go to funding 16 week 'boot camps' to learn ruby on rails.

Defer student loans -- So long as you open a business in the right neighborhood. Better to make non-federally backed student loans dischargable via bankruptcy, stop providing Federally backed student loans to those attending overpriced or under-performing institutions, or anything else to reduce the total cost of education rather than kicking the payments down the road.

"There is fierce global competition in the global tech economy. And there are many countries that would rather regulate than innovate, or who do not shy from closing off markets, forcing technology transfer from U.S. innovators, or even shutting down the internet." -- I owned and wore an RSA munitions t-shirt while Bill Clinton was President.

"Hillary will support efforts to strengthen cybersecurity, both for government networks and for the private sector." hahahahahahahaha

"When Hillary was Secretary of State, the United States led the world in safeguarding the free flow of information" -- Has to be a Pvt. Manning reference.

"Hillary will make it easier for the federal government to find, try, and buy innovative technology—including open source software." -- Useless until they drop the requirement that all software in use in production environments have a support contract.

"She will prioritize the enforcement of well-known cybersecurity standards, such as multi-factor authentication, as well as the mitigation of risks from known vulnerabilities." -- NIST/DISA requires all of this already. Federal agencies have 1 year or less to comply with all NIST directives.

2FA? Required for over 10 years.

Mitigation of risks from known vulns? Most have to remediated within 30 days.

25,000 malware-riddled CCTV cameras form network-crashing botnet

Jim 43

Re: Just wait until you have

More likely penis shaped bright spots.

Utah sheriffs blow $10,000 on smut-sniffing Labrador

Jim 43

See any of the Star Trek series

Jim 43

Jared Fogle (The 'Eat Fresh' Subway spokesman) had his child porn stash found (hidden in a wall if I remember correctly) by the same sort of dog. I remember being shocked by that at the time.

It seems legit. Anyway, it's nothing that strong encryption can't solve.

12 years of US Air Force complaints lost in database crash

Jim 43

Re: Well done BOFH

This is a very efficient way to condense all the existing tickets to one ticket.

Jim 43

Re: Wait..

Lockheed understand that all federal systems are accountable to FISMA/NIST.

There was a detailed recovery plan that was certified annually by a small team of experts.

Someone had to certify annually that recovery tests had been performed and validated.

Lockheed probably billed for the time to update and test recovery plans. No bonuses for this one.

Jim 43

Re: They probably lost all the records from the

Every Federal agency is required to comply with NIST 800 series guidelines within a year of publication. NIST 800-34 is the relevant document here.

Sysadmin 'fesses up to wrecking his former employer's IT systems

Jim 43

He's probably better off in prison for the housing, meals, and new career training.

Admin fishes dirty office chat from mistyped-email bin and then ...?

Jim 43

Odd moral dilemma he created for himself

Unless someone in management asked Flash to do this then he's way out of line and should be fired. I'd also advise Flash to find an IT job that does not put him in a position to access his co-workers personal data as it looks like he has some serious issues with respecting boundaries.

I'd love to know what's wrong with letting mail bounce. Why on earth did Flash feel the need to set up a 'catch-all' account? While I'm at it, why would anyone want to manually route email?

Google tries to run from flailing robotics arm

Jim 43

Here's a remix you may like https://www.youtube.com/watch?v=sAmyZP-qbTE

Top rocket exec quits after telling the truth about SpaceX price war

Jim 43

When I first read about this I was positive he was getting sacked over the 'two fiancees' analogy he used.

Yahoo! kills! search! APIs!, games! and! Astrology! site!

Jim 43

I'd love to know what percentage of their searches are for 'google'. There have to be tens, maybe hundreds of thousands that simply don't know they can change their search engine.

Microsoft has made SQL Server for Linux. Repeat, Microsoft has made SQL Server 2016 for Linux

Jim 43

Re: I can't wait to try it out

It worked fine for IE4. I remember it being slower than I expected. But it was beautiful while it lasted.

Jim 43

I can't wait to try it out

I remember how well IE4 worked

Bill Clinton killed off internet taxes, says Australian politician

Jim 43

Re: People won't vote to be taxed

Sure they will. In the US we call them bond issues. I've seen communities in Texas push through a 58 million dollar bond issues so the local high school can get a football stadium. 120 million for a library on an amazingly expensive piece of river-front property? Sure, why the hell not.

Every last one of those bond issues is a property tax increase.

Rejoice, sysadmins, there's a new glamour job nobody understands

Jim 43

Re: Roll up, roll up! You don't even need to study!

I'll explain the joke for you.

The job description quoted from Gartner reads like a bullshit bingo card. For fun, I imagined someone who perfectly fit the bill. I imagined that they would have no idea that things like Zigbee had been around since the late 90's, because, you know, Gartner said it was new.

Cryto for communications and testing. Cool. Wonderful. No idea why you'd be proud of that, but hey, we each have our own standards for success.

As to my hyperbole... Do you follow security news at all? Maybe we just have different thresholds.

Anyway, sorry I struck a nerve. Have a nice weekend.

Jim 43

Re: Roll up, roll up! You don't even need to study!

Pish! With such a young, rapidly evolving technology you rely on your IoT architect to keep up with the hottest security trends. Your IoT architect will be the first to tell you that the technology is simply too new to have "'best' practices".

Brit firm unleashes drone-busting net cannon

Jim 43

Re: Fire control system integration

You're going to need an IoT Architect for this and I hear they're recently become terribly expensive.

Backup bods at Microsoft lose CA audit data after server crash

Jim 43

The wrong person wrote that message

Missed an opportunity to make a joke about root cause analysis of the root CA audit data loss.

Addressed to "Many of you" -- So, uh, guys, we're not 100% sure which roots are impacted (is that important?).

This is likely an error on our side -- We don't know how this happened and the one person capable of reading the log files is not present, or we don't log, or it was malicious activity from some unknown third party.

It looks like it rolled back to an old backup -- again, we have some logging issues to resolve. No idea what happened, but it's possible that the system spontaneously decided to initiate a restore, or maybe Todd in storage was messing with snapshots...

lost data for about 147 roots -- I know, specific huh? Anyway, the mail log guy was available and he tells us that there were 146 similar emails sent along with the one we got a panicked reply about. So, yeah, we think 147 is about right. But maybe not.

If you received a message -- But the mail log guy is not quite as competent with regex so, while he was able to identify the message content match, he might have munged the To: field. Alternately, they discovered that the initial message was poorly crafted and likely caught in spam filters.

Please don't panic -- You really should panic

Sorry for the confusion -- Really? Sorry for the confusion. How about "Sorry for the fuck ups"

Gopher server revived after 15 years of downtime

Jim 43

I would have paid good money to be sitting next to the network/firewall tech who picked up the service ticket to allow that traffic.

Dell PowerEdge R730: Reg rack monkeys crack smiles over kindness of engineers

Jim 43

Re: Nice kit...

This one simple tip is shocked my last three bosses: You can have your Dell sales team replaced faster than a bad stick of RAM.

Every email from a Dell sales rep contains contact info for their boss in the footer. Contact that boss and explain that you're not happy with your sales team and would like a replacement. They'll fix you right up.

The number of times I've started a new job and gotten an earful of bile about how their hardware sales team sucks is amazing. One fscking phone call fixes it right up and people stare at me like I've cured cancer or something equally difficult.

You've heard of Rollercoaster Tycoon – but we can't wait for Server Tycoon

Jim 43

Re: I can see it's use in Job Interviews

Does the lowest or highest score get the job?

Jim 43

Re: very specific

They're in Italy. The $109,242 looks like a EUR to USD conversion.

Sysadmin's former boss claims five years FREE support or off to court

Jim 43

No kidding, they just don't understand that the morale will continue until the beatings improve.

Chef launches Compliance: Server security policy as code

Jim 43

Relax there Sparky, it was an example. If it's a compliance tool, then, yeah, of course all that other stuff should be incoming.

Linus Torvalds fires off angry 'compiler-masturbation' rant

Jim 43

In other news

I sent a nasty email to the cow-orker who used cat before grep

11 MILLION VW cars used Dieselgate cheatware – what the clutch, Volkswagen?

Jim 43

Re: Confused....

In most states, agricultural vehicles are not subject to emissions rules. IOW, we're aware of the problem and are quite happy to ignore it.

Techies! Shadow IT means you need to up your game

Jim 43

If your users are able to implement shadow IT in an enterprise environment you've got bigger problems than shadow IT.

FBI may pillory Hillary with email spillery grillery

Jim 43

Re: Rand Paul (R-KY)?

I'm not falling for this again

Asimov's ghost! Oil and gas rigs could be taken over by robots

Jim 43

Re: Finger-tapping!

Catch that Rabbit

GIGANTIC galaxy-chomping black hole rips boffins a new one

Jim 43

Re: There's something heartening...

You should visit an Astronomy On Tap event if you get the chance.

Evil NSA runs on saintly Linux, Apache, MySQL

Jim 43


Linux Abrogating My Privacy

US police to throw big balls in criminals' faces

Jim 43

I can't wait for the headlines

The .01% of cops with no sense of career preservation are going to find amazing new ways to get their departments sued with this thing.