back to article STOP! It's dangerous to upgrade to VMware 6.5 alone. Read this

At a client site recently, we had to investigate why the upgrade from VMware vSphere 6.0 to 6.5 had gone wrong in that the normally rock-solid environment was a bit ill – to say the least. On-site conversation ran something along the lines of: “It’s up.” “Hang on...” “It’s down.” “Webservices say: ‘No’.” Cue PSC reboots, …

  1. Anonymous Coward
    Anonymous Coward

    RTFM

    If you have a large setup then RTFM first before doing a major job

    https://docs.vmware.com/en/VMware-vSphere/6.5/vsphere-esxi-vcenter-server-65-upgrade-guide.pdf page 43, Important. That took one simple search ("vmware 6.5 upgrade guide") and a skim read to find.

    1. a handle

      5.5 to 6.5 was quick n easy

      Upgrade vcentre then esxi's. For any software upgrade check dependancies and do it in the right order, I did double check even though it was obvious what it should be in our case.

      Love html access, we now connect straight into vcentre and even directly into hosts. But best of all we can do this from Ubuntu too. I haven't missed the legacy, Windows only c# client once, relatively it was restrictive.

      Update manager is also built into the vcentre appliance, it rocks!

      Well done VMware :-)

    2. Anonymous Coward
      Anonymous Coward

      Re: RTFM - And get some training

      Even if it’s just from one of the better online training providers. VMWare has become a hideously complex beast.

  2. A Non e-mouse Silver badge

    Too Naive

    I think your details on the upgrades of the vCentre environment was way too naive.

    Firstly, the upgrade to 5.5 to 6.0 isn't that trivial. (Mine went so horribly badly that VMWare support gave up and said delete everything and start again!)

    The upgrade from 6.0 to 6.5 isn't really an upgrade but a migration. New VMs are deployed and the data copied over.

    Finally, we get the horrible mess of clients that VMWare has: The fat Windows client, the Flash web interface and the newer HTML5 interface. No one interface can do everything. Nice one VMWare!

    Oh, and always know which hosts your PSC & VSC are running on. Just in case you need to login to the hosts direct to restart the VMs.

    1. DougMac

      Re: Too Naive

      > Finally, we get the horrible mess of clients that VMWare has: The fat Windows client, the Flash web >interface and the newer HTML5 interface. No one interface can do everything. Nice one VMWare!

      At least there is a path. eg. The C# client is dead now and has been for a year (if you are current on patching on the 6.5 track).

      The HTML5 fling UI offers full usability, and they keep wrapping more and more of the HTML5 fling into 6.5 as it ships U updates.

      Most of what I've encountered are VMware admins that are adament that they can never use anything but the C# client, even though it is dead, so they stick with the older things because they can't change.

  3. Locky

    Upgrading?

    For major versions in VMware, I find it's rarely worth it.

    Build new and migrate. Then nuke from orbit, it's the only way to be sure

    1. J. Cook Silver badge
      Thumb Up

      Re: Upgrading?

      @ Locky:

      "For major versions in VMware, I find it's rarely worth it.

      Build new and migrate."

      THIS TIMES MANY MILLIONS.

      We did that going from 4.1 to 5.0, then to 5.5, and then again to 6.0.

      It's a pain in the tuckas having to re-build the clusters (unless you are also doing hardware upgrades concurrently), but it's certainly cleaner.

  4. adam payne

    Read the documentation, read it again and then read it a third time until you completely understand the benefits and risks associated with the upgrade.

  5. Daz555

    This smacks of change management failure. Surely peer review of the planned change would have resulted in somone spotting a missed prerequisite for the change? If it was peer reviewed and no one spotted it then they need better SMEs or better training for those SMEs.

  6. drexciya

    It's a bit mean to say RTFM, but I cannot help to agree with the previous commenters, when it comes to this. It's also explicitly mentioned in the offical What's New 5.5-6.5 training course.

    To add something more useful to this discussion, I want to note that if you want to modify your environment, you should do this from the start. In vSphere 6.5 it's not supported to split the PSC and Management Roles after deploying your vCenter Servers. This was possible in 6.0u1+, but for some reason has been discontinued.

    1. A Non e-mouse Silver badge

      I'm sure it's scrawled all over the vSphere/vCentre docs: Upgrade vCentre first, then, and only then, do vSphere itself.

    2. J. Cook Silver badge
      Boffin

      Yep, that bit [RedactedCo] hard two weeks ago; we kept wondering why vCenter would randomly stop responding on the web client, and we'd have to re-boot the vCenter machine. This all started after we upgraded the 6.0.[mumble] to 6.0U2 some months previous. It got to the point where none of the services would start, and I called in VMware on a 'OMGMAYDAY' priority ticket. (because not being to manage the environment actually *IS* an emergency around here...)

      Turns out, 6.0U2 with an embedded PSC *and* a linked vCenter (also with an embedded PSC) is not a supported configuration. Your Humble poster got to build a new external PSC, A new vCenter, restore the vCenter database (after bodging the broken machine up far enough to get a database backup!) so that Distributed vSwitch would continue working, then re-connect hosts..

      That was the main site; the second site that had the linked vCenter got a similar treatment (external PSC and new vCenter).

      Thankfully, the only downtime we endured was having no automated HA or DRS shuffling while that was taking place, no granular backups of the environment, and our DR solution broke horribly as well, requiring a full reinstall. The business never noticed anything was wrong, and none of the applications they used skipped a beat.

  7. Anonymous Coward
    Anonymous Coward

    You have to read a *300 page* document just for an upgrade?!

    1. Jonathan 27

      I think the clue that you have to read it is that it's a 300 page document.

  8. johnnyblaze

    I've upgraded a few clusters to vSphere 6.5, but then reverted back to 6.0 with latest 'U' release. Until VMware sort out the management web UI and port everything over to HTML5 (which won't be until v7), they can stick it. The Flash based UI is horrible, and the current HTML5 UI is buggy and unfinished. If they're going to dump the VIClient, then at least give us something that works in it's place.

    1. A Non e-mouse Silver badge

      Er, in 6.5 they added the HTML5 client by default (You could always add it manually to a 6.0 anyway via the Fling ) They didn't remove anything. The Flash interface is there in both 6.0 & 6.5.

  9. MrBoring

    5.5 to 6.0 was painless with the vcsa with built in PSC. Don't see the point of going from 6.0 to 6.5, that's just lots of pain without any gain. It must suck for sysadmins who have to just do as their told by managers who have no clue and just want to be on the latest version of everything.

    1. A Non e-mouse Silver badge

      At one extreme there's being on the bleeding edge and upgrading as soon as a new version comes out. At the other, there's being on the trailing edge and upgrading in a rush because you're on old, unsupported software and hitting compatibility, reliability or security issues.

      Somewhere in the middle the best: Let others find the bugs in the .0 releases, and upgrade at your leisure to keep vaguely current. 6.5 has been out for a while, so now is a good time to start looking at upgrading.

      1. This post has been deleted by its author

  10. Tom 7

    I remember VMware

    or rather I didnt. Had to read the manuals when it went wrong. MS networking I could recite the manual after 14 pints.

    1. Anonymous Coward
      Anonymous Coward

      Re: I remember VMware

      Tom7,

      Not sure whether that is a Good Thing or not !!!

      It could be because you have 'Deep Dived' the MS Docs so often to fix issues that you know them by heart or that VMware once correctly implemented tends to 'Just Run' and you don't need to live in the Docs as much. <Grin>

      YMMV :)

      Also could explain why were you were drinking the 14 Pints to begin with ???

      [Runs, ducks and hides]

  11. SiFly

    Validation

    Whatever happened to software having doing a validation check prior to running an upgrade, i.e. are all my upgrade requirements in place, does everything look right prior to the upgrade and can the rollback data in place.

    For heavens sake isn't modern software capable of preventing such FAILS, rather than giving some poor sap a 300 page document to follow, or are the writers of the software just being cheap ?

  12. cjbeckwith

    C# 4 ever

    The only thing worse than having the C# client pried out of my cold dead hands is trying to administer a solution and having to use the dreaded flash web client in combination with a not yet complete HTML5 client. Until such time as complete functionality is added to the HTML5 client I simply can't justify the upgrade. Get your "stuff" together VMware.

  13. Anonymous Coward
    Anonymous Coward

    Upgrading from 5.5

    to Hyper-V like everyone else

    1. DougMac

      Re: Upgrading from 5.5

      I know of nobody that has changed to Hyper-V and liked it.

      Hyper-V has its own worms and problems.

      Everywhere I've encountered it in production is in basic mode (eg. single hosts) because full-on clusters is unatainable for just about eveyone.

      1. cjbeckwith

        Re: Upgrading from 5.5

        I think the biggest problem is that administering a cluster of Hyper-V is simply "cludgy". I will give VMware that, even at its worst the flash web client is still better than the cluster management of a Hyper-V cluster.

        1. Franco Bronze badge

          Re: Upgrading from 5.5

          Hyper-V has come on leaps and bounds, but is still let down by Management. Even now, you can't do every single thing in SCVMM, there's still the occasional need to dip in to the Hyper-V or Failover Cluster Manager tools. Don't agree with the comment about Hyper-V clusters being unattainable though, I've built clusters for sites ranging from 8 VMs running on entry level HP LeftHand SANs right up to massive clusters running on Dell Compellents and 10Gig Cisco Nexus switches.

          I can't comment on this personally, but I have heard from guys I know who work with VMware more than Hyper-V that the support levels are not where they were pre-Dell either.

    2. kryptylomese

      Re: Upgrading from 5.5

      Or even better, to ProxMox!

  14. Bitsminer Silver badge

    I remember VMS

    Back in the days before rtfm was a word....

    The boys in the project down the hall paid several kilobucks for an in-field upgrade of the VAX-11/780 to a dual processor '782. Downtime was about 4 or 5 days while the extra cabinet was bolted on.

    And then came time to boot up. It didn't.

    After a couple of days on the phone, tech support asked what version of VMS was installed. Oops, should have upgraded the software before doing the hardware! VMS 4.1 [iirc] doesn't support SMP! Meanwhile 10 programmers were stuck reading magazines and wandering the hallways.

    [In those days, one CPU at 0.005 GHz could support 10 coders on their VT100s.]

    1. Anonymous Coward
      Happy

      Re: I remember VMS

      Ironically, VMS clustering and DR was the gold standard for high availability. You could take a rocket launcher to a VMS site and still stay in business. Bit late to help you now, but I mention it to remind the kids that the old timers have seen and done it all before. On less hardware. With a CLI experience best described as two dictionaries colliding. Oh - and get off my lawn!

  15. Anonymous Coward
    Anonymous Coward

    They're helping us to RTFM...

    They're making it easier for us...

    https://vspherecentral.vmware.com

    vCenter --> vCenter Lifecycle --> vSphere 6.5 Upgrade

    https://www.vmware.com/uk/products/vsphere/upgrade-center.html

    It's pretty straightforward stuff really. Plan and execute. Simples!

    That said, I prefer build new and fresh and then migrate VMs to it (SRM eval anyone?).

    Cleaner, allows you to take advantage of new features (i.e. architect better!) and satisfied my OCD tendencies. :-)

    (Also, just as your account reps... they will be able to help)

  16. Rob D.
    Happy

    Life learning

    > The actual process isn’t that bad as long as you follow the documentation.

    Just made me smile.

  17. J. Cook Silver badge
    Pint

    Another gotcha with 6.5 VCSA...

    ... is that it *requires* a properly functioning DNS in order to not mess it's pants halfway through the install; if one is doing a bare metal build, one has to either use IP addresses for everything, or manually build a DNS server that is authoritative for the address it's living in.

    This week saw me trying three times to get the appliance to install to some old hosts for a test lab that [RedactedCo] is building for a major application update, and I had to port a copy of the production domain controller into test so that I had DNS of 'some form' operational. (I'll spare you the shenanigans of finding out that the flash-based site needing a fully patched and updated workstation to keep from crashing and a patently useless HTML 5 client for such things as setting up permissions; fair to say that it took me about two days of actual time spent to get the thing stood up to the point of being able to start building actual infrastructure in the test environment.)

    1. J. Cook Silver badge
      Pint

      Re: Another gotcha with 6.5 VCSA...

      That having been said, the actual *process* for deploying a VCSA is nicely streamlined, along with the 6.0 U2 installer. (the earlier installers were... less than helpful regarding the order of what needed to be done.) The deployment wizard, while not very verbose, is straightforward enough, and the vCenter installer is very straight forward about what you need to do.

      The Update Manager installer still sucks, though.

      I also found that if one is using the embedded database for Update Manager, then it should probably go on it's own machine (NOT! the external PSC!), at least for a moderate deployment such as ours (~20 hosts and ~400 or so vms)

      YMMV.

  18. Terafirma-NZ

    don't forget the firmware

    Also before the ESXi step make sure you have run whatever hardware updater your vendor has as 6.5 moves to newer drivers that requires a newer minimum firmware version.

    Even more fun if you have Broadcom NIC's and what to pick between the Broadcom and Qlogic drivers.

  19. Anonymous Coward
    Anonymous Coward

    Staying current is still essential - 5.5 goes end of GA support this September !!

    I hope people are not forgetting that vSphere 5.5 goes end of GA support in September this year?

    So yeh, get planning now. Plan well and it's pretty straightforward.

    Lots of help is available - just ask.

    (anon as I work in this area)

  20. Doogie Howser MD

    Sounds to me

    Like it was the author's fuck up and he's writing it in the third person to pretend it was someone else.

  21. Talleyho

    Upgrades are NOT expensive

    Version upgrades are included if support is current.

  22. Anonymous Coward
    Anonymous Coward

    Physician, heal thyself

    You think you've got it bad. Try an in-service upgrade of an OpenStack deployment.

  23. Anonymous Coward
    Anonymous Coward

    A Brief History

    Much of the current vCenter 5.5->6.0->6.5 angst is because of the introduction of what we now know as the PSC together with the (optionally) distributed vCenter model. If are moving from 5.0/5.5 to 6.5, today there are tools to assist but the easy button is likely to deploy new and migrate servers from old to new.

    Flash sucks but VMware is caught in a tough place with many vCenter third party plugins written to support the Flash API. Migration is proceeding as we know with the two UI choices and ongoing updates to various plugins to support native html5. Change is pain but pays the bills... Oh yea... Read the release notes/upgrade guides.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like