back to article Microsoft's own code should prevent an Azure SSL fail: So what went wrong?

Server 2012 is the Microsoft operating system that, in my opinion, makes cloud computing a reality. As far as I am concerned it is as big a leap over Server 2008 R2 as that OS was over Server 2003. With it you can build anything from a small cluster to a service as big as Microsoft's own Azure platform. Which is why I am …

COMMENTS

This topic is closed for new posts.
  1. RyokuMas
    Joke

    What went wrong?

    They took Eadon's advice and brought him in as a consultant...

    1. TheVogon
      Mushroom

      Re: What went wrong?

      "Even if it was the cryptographic certificate upstream from the end nodes that expired, why wasn't the CSC server auto-renewing from elsewhere?"

      The answer to that would be that a properly designed and secure PKI root certificate server architecture is not connected to the LAN.

      In Microsoft's case, the approach is publically documented: http://blogs.technet.com/b/aviraj/archive/2011/10/04/microsoft-downloads-public-key-infrastructure-at-microsoft.aspx

      1. Rushyo
        Stop

        Re: What went wrong?

        Quite. Whilst it's certainly possible to automate certificate management, it's sort of like maintaining service uptime by sending service restart commands over telnet. Insecure, undesirable and just a little bit quaint.

    2. Anonymous Coward
      Anonymous Coward

      Re: What went wrong?

      I remember when Amazon's storage went down across multiple regions and it took them a lot longer than 9 hours to fix it!

      http://www.nytimes.com/2012/12/27/technology/latest-netflix-disruption-highlights-challenges-of-cloud-computing.html?_r=0

  2. Anonymous Coward
    Anonymous Coward

    Will this work with Symantec (Verisign) certs?

    I buy a 1 year cert, but it will have an expiry date in 2018.

    However it will be revoked this time next year if I don't pay them for another year.

    I would like to buy the 5 years together, but beancounter says no.

    1. Mr Anonymous

      So don't buy an over priced Verisign cert with an expiry date you cannot track.

    2. Fatman

      RE: I would like to buy the 5 years together, but beancounter says no.

      The solution is right out of a BOFH's playbook.

      Invite the beancounter into the IT lair, and motivate said beancounter with the exposed ends of a mains cord. His experience will be 'shocking' (to say the least), and hopefully it will instill a newly found sense of respect (or fear as some might put it) for the powers of IT.

      If that doesn't get the beancounter to see the error of his ways, then arrange for a "data corruption" to occur in his payroll records. A salary that mysteriously changes to ZERO; and resists any attempts to "correct" it.

      I mean, use your imagination.

  3. Phil O'Sophical Silver badge

    > I'm not fully sure of the underpinnings of Azure; does it run on Server 2012?

    According to wikipedia (yes, I know):

    "Windows Azure has been described as a "cloud layer" on top of a number of Windows Server systems, which use Windows Server 2008 and a customized version of Hyper-V, known as the Windows Azure Hypervisor to provide virtualization of services. "

  4. koolholio
    Holmes

    SysAdmins versus Ops versus Sec

    System admins deal with systems, ops usually deal with servers and infrastructures, sec deals with the security of them...

    I'm sure back in the older days, certificate administration was someones full time occupation?

    Theoretically, you would need triple the amount of Sec to cater for infrastructure, server and system?

    A basic maths principle, I would have thought?

    1. the-it-slayer
      Facepalm

      Re: SysAdmins versus Ops versus Sec

      You'd think that would be the case. But what's stopping the Sys Admin creating a certificate validation spreadsheet/database of all the cert's in production/test, location etc and then assigning calendar reminders of up to 4 weeks before the cert becomes invalid? Surely that should be setup on birth of a certificate? It's not exactly rocket science; it's common sense and effective administration.

      1. Just a geek
        Windows

        Re: SysAdmins versus Ops versus Sec

        Why mess around with a spreadsheet? Surely networking monitoring/syslog/SNMP must have the ability to say <8 weeks remaining on this cert and send out an alert/change an icon to yellow/something to draw attention to the issue?

  5. Anonymous Coward
    Windows

    One problem with the article...

    "Not only can Microsoft sign its own damned certs, Server 2012 makes this whole process so simple web administrators will weep."

    Which is of course assuming that Microsoft actually uses that stuff themselves. Be very careful there because Microsoft has a solid history of telling the world "A" while doing "B" themselves. For example; when they started pushing Exchange forward as the big Windows MTA which could easily rival Unix environments their own e-mail facility remained hosted by Unix for several years to come. Simply because Exchange wasn't capable to handle their load. Something which they tried hard to keep under wraps of course.

    Just because Microsoft has released a new server doesn't mean they immediately started using it themselves, that line of thinking is IMO a bit silly. In fact; I would deem it much more likely that parts of MS are currently running on hybrid solutions; a Windows core which is being maintained in-house themselves and as such its no longer really server 2003 yet also not really server 2008; but instead a 'hybrid' sitting somewhere in between which got all the solid enhancements to their server line of products yet without the bloat.

    Because just like any other Enterprise environment Microsoft knows that with every change you apply you always risk introducing a certain danger to the system. Considering that they also maintain their own OS updates; why not utilize that themselves as well ?

    1. 1Rafayal

      Re: One problem with the article...

      I would argue that you are pretty much right, but with one caveat; I dont think MS would use hybridised versions of its own server OS in mission critical platforms like Azure (for example).

      I completely agree that these systems exist, and that they are being used, but MS needs to present a business use case for a lot of their new offerings, I think someone has already said Server 2012 will let you run an Azure like cloud if you wanted to do so. In my mind it would make a lot of sense to me for MS to run its cloud in the same way other would or could.

      But, this is just my opinion, nothing more.

      1. TheVogon
        Mushroom

        Re: One problem with the article...

        It is a known fact that Azure runs on a version of Hyper-V Server 2012....And that Microsoft almost always 'dogfood' their products internally before releasing them publically.

        1. Anonymous Coward
          Anonymous Coward

          Re: One problem with the article...

          >It is a known fact that....Microsoft almost always 'dogfood' their products internally before releasing them publically.

          Que? No comprehendo.

          You seem to have buggered up your grammar and spelling there RICHTO. Allow me to help you:

          Microsoft almost always releases 'dogfood' well before it should be releasing the products publicly.

          1. Richard 12 Silver badge

            Re: One problem with the article...

            Maybe you've not heard the term before.

            "Dogfooding" means "using your own products internally".

            It is almost universally a good thing, as it saves the supplier money and helps find subtle bugs.

            Aside from that, would you trust a supplier who doesn't trust their own products enough to rely on them for their own business?

            It comes from "Eat your own dogfood".

  6. Anonymous Coward
    Joke

    "Server 2012 is the Microsoft operating system that, in my opinion, makes cloud computing a reality"

    Thanks for the laughs!

  7. ColonelClaw

    Er,

    Probably a stupid question, but is it possible that MS were indeed running a single CSC server, and it went titsup?

  8. oldcoder

    easy failure - designed to fail.

    More like the root CA expired.

    Regenerating one requires a manual intervention somewhere.

    Of course, as soon as the root CA expires, so do ALL certificates signed by that root CA.

    The next problem is how long is the root CA valid? If it is longer than the persons job, how does the responsibility get passed...

    And if the root CA lasts 5 years, and you change personnel 3 times, how likely is it that the current person in charge knows they are responsible for a root CA that "just works".

    Point and click admins don't know how certificates actually work. And that makes it easy to skip a simple message "the certificate is about to expire" as they just click to update one... Unless it is the root CA.

    I would almost bet the person doing this spent the last 12 hours before the CA expired trying to find the documents on how and where to generate the new certificate, AND get all the updates distributed world wide.

    1. TheVogon
      Mushroom

      Re: easy failure - designed to fail.

      The process and people are almost certainly well defined. However obtaining the required approvals, and getting the right 2 security people out of bed to the right datacentre and to physically issue the required Certs from the offline root server would probably take a while...Requesting a certificate is not supposed to be something that you need in an emergency.

    2. Fatman

      Re: Point and click admins

      AKA, your average Windows Click Monkey.

    3. pixl97

      Re: easy failure - designed to fail.

      Designed to fail, because it's a single point of failure. I bet about everything else with the system is redundant.

      The entire cert system needs to be ran off 2 different CA's, the entire system can run off one, but has a total fit about it (leading a person to correct the problem). Oh, and make sure the CA's expriy dates are significantly different.

      1. TheVogon
        Mushroom

        Re: easy failure - designed to fail.

        Thanks for demonstrating your lack of understanding as to how PKI works. You can't run an "entire cert system" off 2 different CAs. That would by definition be two entirely separate and different cert systems...

        The best you can do is cluster the CAs (If you run Windows you can anyway).

        For offline root CAs, a regular secure backup is the best option.

        1. Michael Wojcik Silver badge

          Re: easy failure - designed to fail.

          Thanks for demonstrating your lack of understanding as to how PKI works. You can't run an "entire cert system" off 2 different CAs. That would by definition be two entirely separate and different cert systems...

          There's no reason you couldn't provision a network of SSL/TLS servers from a dual X.509 PKI hierarchy with two CAs, a primary and a backup.

          Simply give the servers two certificates, one signed by each CA. Give the root and server certs distinct expiration dates - 24 hours apart is probably safe, but there's no reason not to provide a wider margin, say a week. Have the servers check the expiration dates of their own "primary" certificate and its root (and any intermediate certs in the chain, of course) and if the expiration is dangerously close (as defined by an appropriate safety margin, based on things like your SSL/TLS session-resumption timeout), send the "secondary" certificate instead.

          Obviously that means two root certs to distribute to the clients, rather than one, but typical clients have so many damn roots installed it hardly matters.

          Then, a server that has to switch to the secondary should send an alert - SNMP, SCOM, whatever your operations console uses.

          There's nothing magic about the "one certificate per server" rule. The client wants to see a server cert that has a subjAltName or CN that matches the FQDNS name of the server it's trying to connect to; that was signed by an issuer it recognizes (or is at the end of a chain that etc); that hasn't expired; that isn't listed on a CRL or dissed by OCSP; and so forth. It shouldn't care if it's not the same certificate it got the last time, and while there are clients that will notify you of the fact (eg Firefox with the Cert Patrol extension), it's quite acceptable.

  9. Cipher
    Holmes

    Inexcusable

    Fill in the blank with any excuse you like, but the fact remains this was a simple thing to look after and totally avoidable. Its not like Microsoft doesn't have the resources to make sure the simple things are taken care of. Apologists be damned, Microsoft is poorly managed to allow this t happen...

    1. Anonymous Coward
      Anonymous Coward

      Re: Inexcusable

      Yup. Totally pointless article. The bottom line is Microsoft is badly managed and promotes a culture of incompetence, much like the software they unfortunately keep trying to inflict on people who don't know any better.

  10. Tom 7

    The real reason would be that, like any other organisation, they cant afford

    to have their IT staff off having an expensive lecture in where the sodding icon has been moved every time the software is shuffled, sorry upgraded.

    1. 1Rafayal

      Re: The real reason would be that, like any other organisation, they cant afford

      I think the issue simply boils down to the fact there should have been either a system or process or both in place to have caught this well in advance.

      How many people here need to deal with licence credentials for the software they work with? Spotting when they expire is a fairly run of the mill job for most people, even if it comes down to a calendar entry in Outlook.

      1. Michael Wojcik Silver badge

        Re: The real reason would be that, like any other organisation, they cant afford

        I think the issue simply boils down to the fact there should have been either a system or process or both in place to have caught this well in advance.

        Yes. Ironically, Microsoft notified me back in October that my Azure ACS 2 certificate was going to expire - a few days from now. Apparently they're better at tracking certificate expiration for Azure customers than they are for themselves.

        Also ironically, I haven't gotten around to replacing that certificate yet. It will most likely expire before I do. (I only have it for development purposes, and I haven't needed to do any ACS work in months.) In my defense, rolling the certificates over would take at least three or four minutes, which I could use instead to post comments no one really cares about on the Reg.

  11. Anonymous Coward
    Anonymous Coward

    They don't?

    Microsoft doesn't have to worry about licensing Microsoft's own kit, so how exactly did this happen?

    In my experience from a large Swedish Telecom business: It is *exponentially* harder to license another departments products that any external vendors!

    "Aiding and Abetting The Enemy, that's what that is" -

    Relying on stuff controlled entirely by the competition, those other managers looking for the next vacant slot in the hierarchy, is a *dumb* move.

    Building an empire of vertical stove-pipes with own-versions of everything needed to ship the product one is responsible for, is the smart move.

    Who cares that this costs money, those are *shareholders money*. Leaving too much of them on the table only makes the top executives more popular, thus difficult to remove, limiting the stream of future career opportunities.

    In fact, Open Source remains the normal, safest and most career-conscious way to share common code and applications between profit centres.

  12. Pat 4

    The answer to the question at the end of the title of this article simply resides in the very first word of said title...

    1. DJV Silver badge
      Facepalm

      You beat me to it - though I was going to say the first 3 words...

    2. hplasm
      Devil

      So if you want a good, stable Cloud coputing system do you choose:-

      a. Microsoft, A 'Software Company' or

      b. Amazon, a bookshop?

      Answers on a Valid Certificate...

  13. Henry Wertz 1 Gold badge

    Brand new code?

    On the one hand, Microsoft forgetting to renew their own certificate is a big fail, and I'm highly amused by it.

    On the other hand, no, I do not expect them to manage this using technology just released for Windows Server 2012. a) (I doubt it but...) maybe they DID use this and it failed. b) If they did use this and it failed, people would give them even more grief and jokes than they are now. Like it or not, it is essentially new and untested code. c) Azure was out before Server 2012, I for one would not rip out and replace a SSL certificate server without good reason -- and they may have decided auto-SSL renewal was not a good enough reason. (Maybe they will re-evaluate that now.)

    1. Trevor_Pott Gold badge

      Re: Brand new code?

      CSC work and it works well. There is also a straightforward upgrade path to its use that is easier than any other IIS upgrade before it. I don't buy "untested" here; it is an evolution of certman and IIS. Not a bloody metal --> hypervisor transition.

      That's what baffles me. Server 2012 is GOOD CODE. Not only that, it has the exact answer to this exact problem. You know me well enough by now to understand that I would never say this lightly, but...CSC doesn't behave like Microsoft "never use version 1.0 of anything" code.

      The whole incident is bizzare.

  14. Michael H.F. Wilkinson Silver badge

    Just goes to show

    The old adage:

    Nothing is foolproof for a sufficiently talented fool

    still holds true

  15. Hans 1
    Joke

    Azure runs on FreeBSD

    ... like hotmail used to ...

This topic is closed for new posts.

Other stories you might like