Microsoft secure Azure Storage goes down WORLDWIDE

Microsoft's Windows Azure storage cloud is having worldwide problems with secure SSL storage, probably because Redmond let the HTTPS certificate expire. Being 'in the pink' is not good news for Windows Azure, as this screenshot from the Windows Azure Service Dashboard attests (click to enlarge) The problems were first …

[stanimir]

Not the 1st SSL accident by Microsoft, it happened (iirc 2000) once w/ hotmail / password. Does anyone have fresher memory?

[Peter2]

That was my first thought.

http://news.cnet.com/Good-Samaritan-squashes-Hotmail-lapse/2100-1023_3-234907.html

You'd think that a company the size of Microsoft could afford to run a system with better uptime during designated working hours than the majority of us lot manage to deliver to our businesses. Then again, size is probably the problem. I would imagine that Microsoft is one of those tremendously process managed places and "it wasn't my job" can now be heard echoing from a thousand voices crying out in terror.

[Anonymous Coward]

@Peter2

"I would imagine that Microsoft is one of those tremendously process managed places and "it wasn't my job" can now be heard echoing from a thousand voices crying out in terror."

Lets not be too hard on them; I for one wouldn't be surprised if Win8 also had some influence in all this.

You see; when Outlook starts on my PC it pops up a window with a list of chores to keep in mind (todo items), which I then work on during the day. I can well imagine that in the full-screen Win8 interface such a pop-up could end up somewhere in the background, thus easily missed and never looked at.

[jonathanb]

From memory, they forgot to renew the domain in that case.

[Levente Szileszky]

"we are so sorry"

http://forums.theregister.co.uk/forum/1/2013/01/04/windows_media_center_epg_fixed/

[Anonymous Coward]

>"It is the opinion of The Register that to have a core service fail in every data center across the world simultaneously is an extremely bad thing to happen to a cloud provider."

Now that was funny!

[Dazed and Confused]

Proves its secure

So secure we won't let anyone access it

[Bob Vistakin]

For crying out loud! How many times????

This is a whole different kind of incompetence for which a word has not yet been invented. Seriously - this is the same me-too comedy "Cloud" offering from Microsoft which fell over last time because it didn't know about leap years? So hows that "lessons learned" thing coming on then?

[Captain DaFt]

Re: For crying out loud! How many times????

I hereby propose (and apologise to Matt Groening), that such stupidity be called a "ballmer". *

[eulampios]

@Captain DaFt

Nice one,

I'd even suggest three forms of this concept: bummer, balmer (pronounced in the British way: bɑ:mer) and ballmer

--What a bummer, I forgot the keys in the car!

-- It's Feb 29th, and who'd ever have predicted that, what a balmer?!

-- Your Azure is down? Don't worry it's a planned ballmer!!!???

[Levente Szileszky]

Re: For crying out loud! How many times????

Err, I'm sorry, I claim MY COPYRIGHT as I'm ALREADY USING "BALLMERIAN" as adjective as well as a noun, look up my posting history. Also just recently started using it as a verb (on another forum) eg. ballmering up = utterly, royally ****ng up something that supposed to be trivial...

...but hereby I grant a royalty-free 'use as is' license to ALL ElReg posters on this forum, effective immediately, until revocation.

[Anonymous Coward]

single point of failure

there must be an attack or two to be developed here? enquiring minds want to know.

[Anonymous Coward]

Re: single point of failure

>"there must be an attack or two to be developed here? enquiring minds want to know."

Two attacks, in fact: username = "sballmer" or "bgates", password = "password".

[Anonymous Coward]

Coincedence?

Now, obviously the SSL expiration is a plain out schoolboy mistake, anyone could have foreseen that. Still; in our small Dutch country those kinds of stupidities even happen at government level.

But the other aspect is intriguing, especially if you take in mind that Microsoft has also recently released version 4.5 of their Web platform installer. Highlights in this version 'spotlight' are fully aimed at Azure; "Azure SDK for PHP", "Azure SDK for Node.js", "Windows Azure PowerShell" and obviously at first place the Azure SDK for .NET.

Those .NET SDK's are for both VS2012 and 2010, both dated at Februari 2013.

Heck; they even added a new software section: "Windows Azure", even though one could argue that Azure is basically another framework and as such should be listed as such (under 'framework' you'll also find stuff such as Python, PHP, obviously the .NET framework, Node.js and so on...).

Could it be that... ?

Microsoft is really pushing Azure as of late. And it seems either their 'success' or something else has now come to haunt them. And if you look at the prices which they charge then I think some customers have every right to be upset about all this.

(for those of you unknown to the 'web platform installer': its a sort of package manager for Windows which allows you to quickly install specific Windows products. From IIS to SQL Server Express right down to MySQL, PHP, Python, Silverlight and their Visual Studio express versions. If you're interested in (web) development on Windows then this tool is IMO the best & easiest place to start looking).

[asdf]

Re: Coincedence?

>Now, obviously the SSL expiration is a plain out schoolboy mistake, anyone could have foreseen that. Still; in our small Dutch country those kinds of stupidities even happen at government level.

Wow in most places these kind of things are expected out of governments but not so much multinational enterprises.

[Denarius]

Re: Coincidence?

I take it you have not worked at a multinational ? In IT anyway. Mostly starting with the well known phrase, "We have a standard process, what could possibly go wrong ?" Another instance; quote "A lot of good local decisions could be bad for the company"

[AndrueC]

Re: Coincidence?

I take it you have not worked at a multinational ?

Well yes, fair point. However it's the OP's apparent faith in governments that is so shocking.

[Anonymous Coward]

@asdf

"Wow in most places these kind of things are expected out of governments but not so much multinational enterprises."

Governments usually rely on (Enterprise-like) companies to handle the IT setup for them.

Well, except for the time when someone within the government really felt good about DigiNotar I suppose ;-)

[itzman]

Re: Coincidence?

Oh dear.

I remember one of our engineers being asked to do a security audit for a large company.

The firewall was fine.

The 37 private modems attached to DDI lines allowing staff to phone in from home were not.

When asked what would be the easiest way to steal information he replied 'Id simply attack, or bribe the junior member of staff who takes the nightly tape backup off site every night in her handbag'

Of course these days, you just nick the laptop the sales director carries with him everywhere with a full copy of all the corporate data on it.

Or better still, stick a trojan horse on it.

[Gert Leboski]

Hahaha

Mwuhuhuhahahahaaaaa!!!!

That is all. :)

Couldn't have happened to a nicer company.

[Anonymous Coward]

Re: Hahaha + Hahaha = Double Ha Ha.....

To bad the entire Microsoft ecosystem and all the data on it, didn't get wiped out in one huge fucking unrecoverable corrupted erasure as well.

Ooooooooooooooooo that wooda stung.

[Mikel]

Re: Hahaha + Hahaha = Double Ha Ha.....

Google "Microsoft danger data loss".

[Tel]

Oh Rly?

And they want us to subscribe to Office 365 and store all our documents that run our businesses in the cloud if we don't want to pay through the nose for a licence to use only once and then throw away a physical local install of Office?

Just who do Microsoft think they are and just how f***ed up is their business model these days?

I've *never* trusted cloud computing and I never will. At least I can get at the culprits if my network goes down.

How did Microsoft get this big of a deathwish without anybody in their corporate structure noticing?

[Fatman]

Re: Oh Rly?...At least I can get at the culprits if my network goes down.

Which is the attitude of my boss. She know exactly whose ass must be properly motivated¹ to insure that the network functions. She also know whose ass needs to have a fire light under it, in the event network services goes down. Rarely has she ever needed to even threaten to get out the flame thrower. The network has issues, they get fixed. End of story.

¹ Includes purchases of the latest shiny-shiny to keep motivation high. It also includes the requisite monetary compensation that keeps acquired knowledge in-house, and deters pilfering poaching by other employers.

[Steady Eddy]

Re: Oh Rly?...At least I can get at the culprits if my network goes down.

Fatman, where do you work, and how to I apply?

[AndrueC]

Re: Oh Rly?

I've *never* trusted cloud computing and I never will

Admiral Kirk once said that about the Klingons. Mind you a few years earlier he also said "Double dumbass on you" and that applies here as well :)

[cpreston]

Re: Oh Rly?

Yes, because having all of your IT under your direct control always guarantees that you will never have an outage. That's why no company in the world ever reported an outage until cloud computing came around.

But I probably shouldn't bother you with facts. You're mind is obviously already made up.

[Joe Montana]

Re: Oh Rly?

It's not about guaranteeing you won't have an outage, that is impossible...

It's about knowing what you have, how its setup, and if something fails what's gone wrong and what you can do about it.

It's all about knowledge, control and understanding risk. With a third party cloud provider you have no idea how well (or otherwise) setup their infrastructure is, how resilient their hardware and power is, how it all fits together and you have no ability to fix anything if it breaks. With your own infrastructure you do know all these things, and you can make your own decisions according to how much resiliency you need vs available budget.

[itzman]

Re: Oh Rly?

You miss the point.

I ran a small ISP once. I had a customer escalated to me screaming that the mail was down and no one would tell him for how long.

I asked why he needed to know that. 'Because I am trying to meet a legal deadline for filing some changes to a contract: I need to get the stuff there in 2 hours, if I cant email it I can fax it or get a courier'

I said 'two hours I cannot guarantee'. THANK YOU he said. I will fax it instead.

The moral is its not the outage that's the problem. Its knowing how long and what is wrong that allows you to take sensible steps to minimise the impact. That is FAR more available when its YOUR IT department who have the server in bits on the floor peering at the smoking CPU chip..

a third party company simply doesn't want to tell you.

[Stewart Cunningham]

Re: Oh Rly?

I think you'll find that it is now Office 364, possibly soon to be renamed Office 363

[adnim]

Some people...

haven't noticed.

If it wasn't for fsckups, pointless devices and trying to convince serious computer users that a finger is the most accurate input device. I would have forgotten Microsoft existed,

[Gray]

Oh, mercy! So now the process goes: a disruption is a "tits-up," a calamity is a "f***k-up," and now a total meltdown is a "ballmer!" .... oh, Mother, spare us!

Thank you, Captain DaFt! "I hereby propose (and apologise to Matt Groening), that such stupidity be called a "ballmer".

[Numpty]

Xbox Cloud Storage is definitely down as well (and presumably related) -- http://support.xbox.com/en-GB/xbox-live-status

[Robert E A Harvey]

Altered mind

I entered the story thinking 'shit happens' and being quite sympathetic to the microsofters.

Then I saw the first paragraph. " let the HTTPS certificate expire."

Tossers.

[amanfromMars 1]

Global Control Systems Meltdown .....?*

Are the world's brainwashed masses, ignorant of ITs novel virtual abilities at extraordinarily rendering colossal change, and in the new worlds emerging, strident and confident and disruptive, and also in cyberspace, being misinformed and diverted in thought to believe that China is an enemy and anonymised individual hackers, ..... invariably subjects of establishment entrapment whenever objects of APT prosecution and persecution, both adept and inept, actively engaging and thoroughly concentrated on phantoms and ghosts of sacrifice, .... are responsible for Microsoft's mistakes, and compounding code errors, in securing their operating systems as pimped and pumped to governments as a means with various platforms with which to control realities deemed suitable for media presentation with an invisible hand control ........ http://www.bbc.co.uk/news/technology-21556611

Methinks the wheels have fallen off that broadband wwwagon before it even gets to the starting line in the New and Surreal IntelAIgents Space Race, don't you?

*And only shared as a question for those who are unable to believe what can nowadays be easily done by remote powers with virtual control of true hearts and smarter minds.

Note to Patten and the BBC .... Get your FCUKing act together, please, or clear the decks of dead wood and old hacks. GCHQ expects and all that jazz and razzamatazz ...... http://en.wikipedia.org/wiki/England_expects_that_every_man_will_do_his_duty

[Novex]

There are so many reasons...

...why cloud isn't a good idea that outweigh its one or two plus points. This kind of balls up is one of them.

Some of the others:

Lack of guaranteed privacy (just who else really can see my data, and that of my contacts and my calendar?)

Lack of truly robust security (there's no such thing, all it takes is one disgruntled insider and my data along with god knows how many others' is in a hacker's hands)

Reliability (not just of the cloud system, but the internet to it - what if my ISP has a problem, or my physical internet connection is in some way down?)

The only good use I can personally think of is as a place to put strongly encrypted backup files.

[cpreston]

Re: There are so many reasons...

1. If you think you have privacy of your contacts & email within your company, I think you'd be surprised with the reality.

2. You know absolutely nothing about what "the cloud" offers as security, do you? I've seen some of the offerings and was more impressed with that they offered than what I see in typical shops.

3. If your ISP is down, your business is down. Cloud or no cloud. But to your greater point of reliablity, I again say that just because something is in your datacenter doesn't make it more reliable.

Amen on encryption. But I have to disagree that it's the only thing the cloud is good for.

[Paul Crawford]

Re: There are so many reasons...

I think for (1) they mean outside of their company.

For (2) please provide more, most of the stuff I have seen mentioned is only secure between your PC and the servers, not 'secure' on their servers in the sense of encrypted by a key they do not hold. Also when it comes to encryption I would ONLY trust an open source implementation of the client side, otherwise how can you check at all if it has a back door? (Yes, I know there could be subtle flaws introduced to make GCHQ/NSA/etc's job easier, but its about the best you can ask for).

(3) depends on your business, for some work yes it is but for a lot of design and manufacturing work you can work quite well (in some cases better!) without an internet connection for a day or so.

[Novex]

Re: There are so many reasons...

1. I'm the only person in my company, so I suppose I could leak my info, but I'd have to sack myself if I did so...

2. The cloud only offers as much security as 'they' say it does - we don't actually know for sure how each provider of cloud-type services ensures that data doesn't get into the wrong hands. It's certainly true that it's in the providers' interests not to treat users' data badly, but that doesn't mean things can't go wrong. And as far as I'm concerned, better the dickhead I know in my company than the dickhead I don't know in someone else's.

3. My business isn't reliant on the internet - I can use my computers offline to get work done, which is pretty much impossible if the applications and data is all up in the cloud.

[Vic]

Re: There are so many reasons...

> 3. If your ISP is down, your business is down

No it isn't.

Communications is a little harder, but I can do plenty without net access.

Vic.

[itzman]

Re: There are so many reasons...

..and data that you couldn't care less if the world and his dog DOES see.

I.e. 99% of domestic data.

NO company should let a cloud service anywhere near their corporate data. Especially a BIG company with so many sysadmins you will never know which one sold your data to a third party.

[mmeier]

Re: There are so many reasons...

Privacy can be an issue for companies not to mention legal reasons preventing storage of data in foreign countries. For my privat data l do not care, nothing I keep on a computer is secret or needs privacy so l could as well put it in a cloud

Data security breach by insiders can be done with local storage just as well

Reliability depends on what you need or want / need. If you need data access from offsite than cloud is likely more available on the average due to it's distributed nature having no single point of failure unlike storage in a single data center as it is common in private use and smaller companies

[Mikel]

Azure waves of pain

Now is a good time to suss out exactly how many Microsoft properties trust their own "enterprise class cloud provider" enough to eat the dogfood.

[Anonymous Coward]

SSL Certs - the new single point of failure

There must be a name for the law that says "whenever you think you've eliminated all single points of failure, another one arises"?

[PhilipN]

Re: SSL Certs - the new single point of failure

Murphy's Law

[itzman]

Re: SSL Certs - the new single point of failure

smiles.

Like the time the digger on the M1 dug up all the 'diverse' optical fibres that connected the UK south to the UK north and cut the UK internet in half?

Like the computer in charge of monitoring and switching connections between diverse links failed to connect ANY of them?

Like the fire that destroyed a company HQ taking all its data with it, and the offsite tapes turned out to be blank..

[Jamie Jones]

Re: SSL Certs - the new single point of failure

"Like the fire that destroyed a company HQ taking all its data with it, and the offsite tapes turned out to be blank.."

I used to work for a company that was affected by the IRA Manchester bomb around 20ish years ago. Equipment was trashed, but the backups were in the firesafe, and were fine..... It just took over 2 weeks before police/forensics/health and safety would let anyone in to get them!

In the meantime, all the disaster recovery team could do was rebuild the server (as new) and create new accounts for the staff.....

[Piro]

.. and nothing of value was lost.

Oh Microsoft. Why would anyone trust Azure anyway? It's not like they already tried storing a bunch of customer's data for Danger.. and lost a bunch of it. Whoops.

[cpreston]

Re: .. and nothing of value was lost.

I think your facts are a bit off there. The danger systems were not running on Azure, or even on updated MS equipment. And it appears that it was an outage that was reported as a data loss story. I have seen no evidence that anyone actually lost data after the restore was completed.

[Piro]

Re: .. and nothing of value was lost.

I never suggested Danger's data was on Azure, that predates this of course, I'm just suggesting that Microsoft have a proven failure at software as a service. Just ask anyone who's running Office 365!