back to article Watchdog slams Pentagon for failing – for a third time – to migrate US military to IPv6

The US Department of Defense (DoD) has been shamed for its appalling IPv6 migration efforts in a formal probe by the Government Accountability Office (GAO). The auditor’s 37-page report [PDF], dated June 2020, noted that the DoD is actually on its third attempt to migrate to the expansive internet protocol, having tried and …

  1. Anonymous Coward
    Anonymous Coward

    The last time I was involved in these issues - a few years ago and for a different nation's defence organisation - the problem described as "DoD ... decided not to complete the upgrades due to security concerns" related to issues with security-related application software (for example mail filtering, reverse proxy agents).

    These applications either didn't support IPv6 at all and had no published plan to do so; or the supplier said that IPV6 support would be along Real Soon Now(r) but without a committed date, and/or that, when available, IPv6 support would be 'an additional cost feature' but the quantum of such cost was not yet established.

    Put on top of that, the issue with at least one major firewall supplier whose then brand-new, wonderful, most technically excellent IPv6 software was single threaded and as a result choked & died at not really very many Mb/sec (certainly not 1Gb/sec) whereas their crappy old IPv4 software was multithreaded and didn't even get a bit sweaty at twice or three times the throughput.

    :(

    1. John Brown (no body) Silver badge

      It sounds like any transition process of such magnitude and with such a large customer should be starting with looking at the infrastructure and software, determining what doesn't support IPv6 and immediately putting that out to tender for stuff that does support it. If the incumbents want to remain the incumbents then they have to update their kit in a similar time-frame to those who can deliver working kit now.

      1. eldakka

        While in an ideal world, that's what you'd do, the problem with government agencies, especially if they have any security role, is certification. Especially security-related devices (firewalls etc.) could have a very short 'certified devices' list, with certification taking years, and being quite expensive to undergo therefore you only tend to find larger companies on it as a small organisation isn't wlling to stump up the $20k+ it could cost just to get certified, with no guarantee of sales volume.

        2 years ago, an organisation starting widely using an internal CA for internal environments (dev, testing etc.). But getting a certificate signed was a very manual process. So someone had a bight idea, why don't they buy a Hardware Security Module (HSM) to store the CA keys and make signing automated. There are decent ones availabe for only around $2k, perfectly suitable for lower value dev/test cert signing.

        But then someone pointed out to that any such device had to be purchased from an approved supplier with a certified device. So the catalog was found and perused, and the only single certified HSM on it, certified 2 years prior, that was $40k.

        Well, that bright idea faded fast.

        And, if it goes to tender, you'll have fun issues (if it's of any significant value) like with the current cloud tender that's been going on for a couple years now with multiple court cases. Or the Boeing v Airbus KC-45 tanker fiasco.

    2. vtcodger Silver badge

      Plus which ...

      On top of everything else, the DOD has real and long-standing security concerns and elaborate systems for dealing with those concerns. Industrial security failures can and often do have serious consequences. Military security failures, however, can have immediate and lethal consequences. Plus which, they are likely to be career limiting for those involved. Assuming they survive.

      I moved on from DOD related work before PCs became ubiquitous, but let me tell you that I can't imagine any security officer in any military related organization having any enthusiasm whatsoever for a technology that potentially allows folks anywhere in the world to browse the contents of its computers if any of thousands of things go wrong.

      That said, some DOD work is just ordinary business operations. Keeping track of employee vacation days, paying payroll taxes, running the Base Exchange. There's no conceptual reason that can't support ipv6 as well as it does in non-defense businesses. Trouble is that the same facility may well also deal with material of tactical or strategic importance. How does one keep the two separate? And how does one keep mistakes from happening that, for example, make the details of a Hawaiian facility's weapons capabilities and current status available to nation state hackers in Novosibirisk?

      I suspect that "upgrading" to ipv6 is right at the bottom of priority list everywhere in the DOD. And possibly rightly so.

      1. Nanashi

        Re: Plus which ...

        If you moved on from the DOD that long ago, then you may have missed that computer networks and PCs have already become ubiquitous in the DOD. They're already using a "technology that potentially allows folks anywhere in the world to browser the contents of its computers if any of thousands of things go wrong".

        All of the worries you bring up have already been addressed by their existing network teams for their existing network. They're essentially just FUD at this point.

  2. Kevin McMurtrie Silver badge

    Security concerns

    I'm guessing many years of applying superstitious IPv4 security fixes until the vulnerability scans pass. They flip the switch to IPv6 and nothing works but some vulnerabilities.

    Software for IPv6 still sucks too. It's the year 2020 and there's still no way for a Docker container, VM, or VPN to negotiate dividing the host's massive subnet.

    1. Yes Me Silver badge
      Happy

      Re: Security concerns

      Some upper layer software has sucky or non-existent support for IPv6. That isn't IPv6's fault; IPv6 support in network stacks and ISPs is getting better and better. Actually to a large extent it's DoD's fault for not doing what they are very good at: requiring specific support in their RFPs and checking that it's delivered.

      Meanwhile in the real world, IPv6 usage is up to 30% according to Google, with a measurable jump caused by COVID-19 (i.e. work-from-home generates IPv6 traffic).

      1. bombastic bob Silver badge
        Devil

        Re: Security concerns

        "work-from-home generates IPv6 traffic"

        ISPs don't necessarily support it for home connections, though, at least not in the USA. The alternative is to get a tunnel from one of the free services [which I do]. But yeah, it would make perfect sense for a workplace LAN to "not support" IPv6 because of the security issues, a good number of which center around windows boxen with publicly visible IPv6 addresses.

    2. bombastic bob Silver badge
      Meh

      Re: Security concerns

      yeah, about those security concerns: I identified a number of those YEARS ago, and came to a few conclusions on how to mitigate them.

      1. Every IPv6 is technically a publically-viewable IP address, which means your extremely insecure box (like one that is running a Micros~1 OS) can be directly accessed from "teh intarwebs". IPv4 usually has some kind of NAT that effectively firewalls most of the problems caused by a ;publically visible IP address. A similar "NATv6" would do the job and also allow for firewalling.

      2. Direct routing to the outside world must somehow be managed so that firewalling can be properly done, especially for secure things. A "NATv6" system could also do that, but it may be more easily done by implementing IPv6 tunnels or internal VPNs. As an example, I use a popular ISP for IPv6 tunneling over IPv4. It can somewhat-easily be done on individual workstations, but would require SIGNIFICANT careful firewalling if you did this for Micros~1 OSs.

      3. On Micros~1 OSs, there are a significant number of "listening ports" that you can see via the windows equivalent of "netstat -l". Like the old "Win Nuke" exploit of a few decades ago, there is nothing to stop some rogue from firing random packets at a visible port until something breaks. As such, having these ports "seen" over IPv6 is a major security risk. Normally the firewall settings would mitigate this, but a proper firewall appliance is really needed, and not something that runs ON a Micros~1 OS trying to "firewall itself".

      Each physical connection to the internet could (somewhat simply) be routed through "an appliance" that would a) firewall everything, b) provide a level of "NATv6", c) assign semi-randomized IPv6 addresses to clients through a DHCPv6 or similar address assignment [already part of the protocol], and d) prevent any client from opening an un-firewalled publicly visible listening port [a major security flaw caused by UPnP support on many NAT routers, if you don't shut it off].

      And yeah, it should NOT be that hard.

      The biggest problem I found was filtering all of the IPv6 listening ports for windows boxen. I simply don't allow incoming connections on those ports. But I don't NAT the IPv6 addresses, so in theory, a windows box COULD open up a publicly visible listening port. If that ever becomes a problem I'd set up a better firewall on the gateway to prevent it (disallow ALL incoming connections to specific IPv6 ranges or similar). This is still do-able since you know what IPv6 addresses are routeable on the local side, and which ones are gateways [so you allow incoming connections on the gateways only]. However, for my home/office setup, I chose not to do it this way. And there's only one IPv6-capable windows box running these [the rest are all Linux or FreeBSD or phones]. I could shut off IPv6 on that box if I ever needed to.

      Worth pointing out, phones and slabs do IPv6 really well on my LAN, over the WiFi. [now if I could JUST get some real speed on it...]

  3. ratfox
    Angel

    I appreciate the fact that all of these steps that the Pentagon were supposed to accomplish are preparatory. As in "scope the project". The hard part of actually doing it, ah well, miracles require a delay.

    1. Anonymous Coward
      Anonymous Coward

      RTFExcuses

      Covered under "They added that they had thought that the activities’ deadlines were reasonable until they started performing the work." ?

    2. Pascal Monett Silver badge

      I appreciate the fact that, apparently, the DoD is treating this migration just like morning exercise. Plan ? Just go out and run, numbskull. Oh, it's harder than it looks ? Maybe we should follow the recommendations ? Nah. Just put on another pair of shoes and it'll sort itself out, probably.

      These "security concerns" of theirs, they do not prevent planning. There's no reason to not plan properly because of them. Lack of personnel trained in IPv6 ? So plan to train personnel. Email not compatible with IPv6 ? Plan to replace it, or upgrade it.

      Planning is not doing, but if you don't want to plan, you won't be doing for sure.

      And the DoD quite obviously does not want to plan. Probably because really doing so will put a lot of ugly figures on an official report, and the DoD doesn't want to show those figures.

      1. katrinab Silver badge
        Flame

        Lets look at it this way:

        If they migrate to IPv6, it will cost $xbn.

        What tangible benefits to they achieve from this $xbn? What does it enable them to do in terms of their military operations that they can't do already?

        Having every device directly addressable from the internet is not a benefit, it is a security risk.

        So, what is the benefit.

  4. Anonymous Coward
    Anonymous Coward

    DoD edict?

    I spent the majority of the first decade of this century and part of the second, supplying switches and routers to the Aerospace and Defence industries. After a couple of years I could tell when the eventual customer was the US DoD. Those were the RFQ's that specifically asked if IPv6 traffic could be blocked completely, and always using the same wording.

    In practice all of my customers could be doing it, as it is a simple settings change, but only the DoD were so blatant.

  5. Griffo

    I dunno what the issue is. I mean, sure, Android doesn't support DHCPv6, Cisco gear still has major memory leaks with their DHCPv6 implementation, there's no hostname registration in DHCPv6 so you have to chase down MAC addresses and hope the client registered themselves in DNS.. It's just so easy to administer.

    1. Tom Samplonius

      You will be waiting a long time for DHCPv6 on Android

      Android will never support DHCPv6, because it is unnecessary. And a waste of battery in comparison to IPv6 auto assignment. Mobile networks are already 90% IPv6.

      DHCPv6 is needed for prefix delegation, but prefix delegation is really just for service providers. If you are using DHCPv6 in an enterprise environment, you've just chosen to make it hard. Along with registering all devices in DNS via DHCP? That was cool 20 years ago, but who actually needs every host in DNS?

      1. Anonymous Coward
        Anonymous Coward

        "but who actually needs every host in DNS?"

        Most organizations, and even my house network. Do you really wish to monitor and access devices by their IPv6 addresses? Assign many options manually? What about reservations? You still need DHCP data even when the address is auto assigned. IPv6 didn't think t about DHCP because in 1996 DHCP was quite new as well, DNS records managed manually, and most systems where client-server ones, very few servers, and client usually unable or with no need to talk to each other. Today, it's quite different.

        Only Google doesn't implement DHPCv6 in Android on an ideological stance, after all for Google Android is just a slurping endpoint, and they have all the data to uniquely identify each device and profile it. The less Android devices can be controlled by others, the better.

        And how much more power would DHCPv6 need? You're not going to renew addresses every five seconds.

      2. bombastic bob Silver badge
        Linux

        Re: You will be waiting a long time for DHCPv6 on Android

        actually, the host would be the router, the client would be the one that has to register its DNS... at least that's my understanding (from the perspective of the network schtuff).

        I support both DHCPv6 and the protocol's built-in auto assignment in order to max out the compatibility on my LAN. As I recall, iOS devices want the DHCPv6, but Android uses auto-assignment. {is that correct?) That may be different now but last time someone came to my home office with an Apple device it seemed to work ok. Laptop too.

        And, isc-dhcpd supports both IPv6 and IPv4 - you just need to run 2 separate instances, that's all.

        One wifi IPv6-related bug I did a patch/workaround for (sorta): the wifi router I have sends out advertisement packets all the time for IPv6 routing, and I can't shut them off (normally I'd disable the routing daemon and let the LAN handle it). Unfortunately, it ALSO sends them on the LAN, which means it tries to take over all IPv6 routing, regardless and send it out the WAN port. So if I don't manually config routing on the clients, it screws things up. To work around this, I manually assigned the WAN port to an IPv6 address on the LAN, and simplhy plugged it via ethernet back into the 4-port ethernet LAN block on itself (which is also plugged directly into the LAN). So it still takes over, but everything routes back through it and back into the LAN again, effectively adding a single hop to the route until the LAN advertises the route, and they fight [but so what]. I suppose I could just plug the WAN into my network as well, but THEN it wouldn't be getting the DHCPv6 etc. info from the server if I did it [I'm pretty sure the WAN port filters these things]. OK it's a hack, and maybe some day I'll just build my own wifi setup with the FreeBSD server again, but it "solved" the issues of IPv6 routing being "taken over" by an ancient buggy wifi router that still works for everything else. [if I rebuilt the OS and took that bug out it would be better, but I'm kinda lazy sometimes]

  6. katrinab Silver badge
    Meh

    What are the security concerns?

    I'll take a guess:

    Maybe something to do with the lack of people trained in IPv6. If you have someone who doesn't know what they are doing set up a network, it probably isn't going to be very secure. Also there are things you take for granted in IPv4 that don't apply to IPv6. For example, if you stick something behind a NAT router, for example a printer, people outside can't access it unless you specifically forward a port to it.

    And: We have had 37 years of using IPv4 and getting to know how to make things secure. We have had 25 years of saying that IPv6 is the Next Big Thing, and maybe we should start using it at some point.

    Also:

    IPv6 is not a new technology. It came out 25 years ago. If a 25 year old technology hasn't reached mass-adoption yet, it probably never will.

    Maybe we should have an IPv4.1 that is exactly the same as IPv4 except that it has more numbers.

    1. Paul Johnson 1
      Stop

      NAT is not a firewall

      Repeat after me: NAT IS NOT A FIREWALL.

      If you want a firewall in IPv6 you can have one. In practice NAT and firewall functionality have such big overlaps that NAT boxes generally include firewall settings too, but creating an IPv6 firewall with default settings that resemble IPv4 NAT is a trivial job (basically, block all incoming connections but allow all outgoing).

      1. DropBear

        Re: NAT is not a firewall

        This is not about firewalling not being possible on IPv6 (which it clearly is) - it's about a device on a LAN being naturally impossible to reach unless you specifically take steps to make it reachable in IPv4, and the same device being naturally reachable unless you specifically take steps to prevent that in IPv6. It's not a difference of what is possible, but a bloody large (and unpleasant) difference nonetheless.

        For an ideally perfect sysadmin with an ideally perfect arsenal of tools, there would be no difference between the two - but in practice I'm willing to bet it will end up mattering quite often, IPv6 leading to loads more scantily protected stuff ending up exposed to anyone with an interest than it would have on IPv4.

        1. Paul Johnson 1

          Re: NAT is not a firewall

          Home Internet ISPs will carry on providing router/firewall/Wifi boxes so that Grandma can connect to the Internet as securely as she does now. IPv6 doesn't make a difference there.

          Small businesses will do likewise.

          Anyone above that level will be hiring people to look after their IT.

          1. Xalran

            Re: NAT is not a firewall

            I can't tell for UK, but for France all the Home Internet ISPs are using dual stack nowadays

            ( dual stack = IPv4 and IPv6 at the same time ). Even the triple play boxes are dual stacks...

            It's just the non conformists like me with an old IPv4 router that are left using only the IPv4.

      2. Anonymous Coward
        Anonymous Coward

        Re: NAT is not a firewall

        I've seen gamers doing their best to lower their pants in front of the world trying to bypass NAT and UPnP because they believe it will lower their ping or whatever.

        NAT does protect a large number of unaware users because it acts like a deny all rule which cannot be disabled easily, with enough technical knowledge and still for a limited number of endpoints. A gamer may put its PS4 in the DMZ or as the default host, but other system will be inaccessible still.

        On the other end IPv6 will need a proper firewall and will need to avoid users disabling it because their ping "is too slow" or they can't access their NAS from their phone, etc. etc. Expect lots of users following advice to "disable the firewall".

        It's a trivial job for those who understand how the internet works. For all those thinking that's a magic box where cat videos appear from, it's not.

        I'm fully for IPv6 being deployed, but we can't simply ignore how it will impact users and what new risks it brings.

        For example, I don't believe the DoD is unware that anybody able to monitor just its IPs with IPv6 will be able to identify how many different systems are behind the firewall, and which IP changes and which one stays the same. In turn it gives an idea of how many people could be there, etc. etc.

        And at least most systems should not now send out their MAC addresses.....

      3. eldakka

        Re: NAT is not a firewall

        (basically, block all incoming connections but allow all outgoing).

        Glad you aren't doing my firewall, that is incredibly naive for anything except a home/SMB, and even then easy but not ideal.

      4. DemeterLast

        Re: NAT is not a firewall

        NAT may not be a firewall, but it looks an awful lot like one in the way it prevents direct access to devices. You might as well say stripping all .EXE attachments from emails is not an anti-virus program. While true it's also a pretty good idea and stops a huge chunk of problems from occurring.

        The other one is "RAID is not backup!" Yes, but it's still better than nothing. Perfect is the enemy of good.

        1. Nanashi

          Re: NAT is not a firewall

          It may look an awful lot like one when combined with RFC1918 addresses, but it is not. Any inbound connection that's possible on a network that's not using NAT is also possible if you add NAT to the same network without changing anything else. NAT is applied only to outbound connections, not inbound ones.

          I'm not saying that it "only blocks most" inbound connections here. It doesn't block any of them. That's why it's not a firewall, that's why you need a firewall and that's why you don't need NAT to have a firewall or to be secure. If anything, NAT is just going to confuse you into thinking you're secure when you're not.

          Hopefully the DoD understands this better than the average El Reg commenter (who will now proceed to downvote me for calling out their misunderstandings of NAT).

    2. S4qFBxkFFg
      Trollface

      Re: What are the security concerns?

      "Maybe we should have an IPv4.1 that is exactly the same as IPv4 except that it has more numbers."

      Already in the works:

      https://packetlife.net/blog/2011/apr/1/alternative-ipv6-works/

      1. Anonymous Coward
        Anonymous Coward

        Re: What are the security concerns?

        I notice this ipv4.1 article was published on April 1st (ahem)

  7. Jim Willsher

    20 years down the line, low adoption, and even the DoD can't do it. The IETF needs to admit defeat and launch something that has the simplicity of IPv4. And that means a nice NAT setup that makes security and setup easy for everyone to adopt.

    1. Anonymous Coward
      Anonymous Coward

      They're sitting on the largest allocation of IPv4 addresses to a single entity, why should they move to IPv6? Addresses exhaustion is not a problem of theirs, and probably some of their systems are so old they probably barely work on IPv4....

      1. Nanashi

        They interact with a lot of other entities, ones that don't have the largest allocation of v4 addresses to a single entity, and those other entities use a lot of RFC1918 space. That's going to lead to a lot of instances of needing to interoperate with networks that have overlapping RFC1918 space. RFC1918 overlaps are expensive to deal with, and are a security headache. I can understand not wanting to deal with them.

        They're probably also interested in selling off some of that space.

  8. Maelstorm Bronze badge

    I can see the push for IPv6 since there are no more numbers for IPv4, and haven't been for awhile. NAT is the main solution here as you can have one public IP address and an entire A block of private addresses behind it. Besides, why does a corporate workstation in an office require a publicly routable IP address when it's behind a firewall? The problem here though is that IPv6 is missing features that IPv4 enjoys. Although it's a matter of software, getting the vendors on board to actually write standards compliant code is like pulling teeth.

    1. batfastad

      `IPv6 is missing features that IPv4 enjoys` please list. Or is NAT a feature now?

  9. batfastad
    Trollface

    IPv6

    It is always worthwhile visiting these comment pages on IPv6 articles. When I return in several hours I expect to see a deluge of predictable "IPv6 is rubbish" comments by people who give absolutely no justification for their opinion and who don't or refuse to understand it.

    I suspect a high level of overlap with people who voted for Brexit :p

  10. Version 1.0 Silver badge
    Happy

    IPv6 was a great idea when it was invented

    But that was a long time ago and the world has changed a lot since then. When IPv6 was created everyone was saying that we would run out of addresses soon and the internet would collapse ... that was about 25 years ago and we're all still sailing along just fine. I think this tells us more about the basic design of the internet than a little thing like the address scaling that has an itch or two.

    It would make sense to dump IPv6 now and look at what's happened over the years and create a new mechanism - hopefully one that could be implemented without the massive changes that IPv6 requires.

    1. Nanashi

      Re: IPv6 was a great idea when it was invented

      v6 doesn't require massive changes though, considering the scope of what it's doing. It requires something close to the minimal set of changes that are necessary to expand the address size.

      The v6 networking model is pretty much identical to the v4 model; its main difference is that the addresses are longer than they are in v4. It's true that making the addresses longer requires a large set of changes, but... isn't that sort of unavoidable? v4 doesn't work with longer addresses, and those longer addresses are the reason we're doing this in the first place. How are you going to get them without making the changes needed to add support for them?

      v6 already has widespread support and deployment. Dumping all of that in favor of something that still has to do most of the same work, from scratch with no existing support, would be incredibly counterproductive.

  11. batfastad

    IPv6 in the UK...

    ... is getting there. https://www.google.com/intl/en/ipv6/statistics.html#tab=per-country-ipv6-adoption

    30% IPv6 penetration in the UK is not too bad when you consider that of the 4-5 main UK fixed-line providers only BT and Sky have mostly complete end-user deployments for new customers.

    Are there any mobile providers in the UK that are offering it yet? Mobile IPv6 deployment is fairly advanced already in Europe, the US and Asia. Once mobile providers in the UK start rolling out IPv6 I'd expect UK stats to jump quite nicely to the 40-50% range.

    Obviously if you're an enterprise you may as well continue using RFC1918 space - noone cares about your enterprise citrix or whatever connections being IPv4. And since you're an enterprise then enabling IPv6 for customer-facing services should just be a toggle with your edge vendor, with backhaul to origin over IPv4 if you want.

    Don't be scared of IPv6, people! It's fine when you get to know it!

    1. Roland6 Silver badge

      Re: IPv6 in the UK...

      >Are there any mobile providers in the UK that are offering it yet?

      Why do mobile providers need to offer IPv6?

      Whilst I think you will find their networks are IPv6, however, the majors will present as IPv4 - I suspect from my recent investigations of EE's 4G interface, this gives greater control over the router/mobile network interface and thus allows for more billing options.

      If you need IPv6 on mobile then you'll find the networks will point you at the various overlay providers who will sell you at IPv4/IPv6 capable SIMs with fixed private or public IP addresses. Naturally these are expensive compared to the consumer offerings.

      As a consumer, if you want a fixed IPv4/v6 address for a mobile then there is alot to be said for investing in A&A's L2TP-VPN offering - it works provided your device/router supports outbound L2TP-VPN (Draytek DrayOS routers do, but their Unix ones don't...).

      1. Nanashi

        Re: IPv6 in the UK...

        Because if they don't, all of their customer traffic has to go over v4. They don't have enough v4 addresses for their customers, so that traffic has to be CGNATed, and CGNAT is expensive.

        Doing v6 lowers their costs. EE are, in fact, doing v6 to their customers today for this reason.

        There's also the part where you need v6 to reach v6-only servers, but even if you somehow thought that wasn't the case the cost situation is still there. (And no, v6 doesn't restrict their billing options. They can still see the traffic and know which customer generated it.)

  12. jelabarre59

    Expansive? Expensive?

    on its third attempt to migrate to the expansive internet protocol

    did they mean to say "expansive" or "expensive"?

    I am surprised though that the DoD didn't become involved in the specification early on, so that their issues (security & such) could have been addressed while the spec was in development. Seems precisely the sort of thing they would have wanted a say in.

    With all the difficulty of implementation so many companies have dealt in, and the seeming resistance in various areas, maybe we *do* need to go back and re-evaluate the spec, and write a new one. Not expecting such an idea to be easy or quick, but it at least needs to be considered, even if the idea gets rejected later, it could spur on some creative thinking. Or how readily could an extension to the spec be designed for the security-minded, something that looks to the outside to be regular IPv6, but enough additional routing information for the routers at the edge of their networks to know what the traffic really is (and no, I'm not talking about setting the "evil" bit).

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon