How can we recruit for the future if it takes an hour to send an email, asks Air Force AI bigwig in plea for better IT A US Air Force director of ops this week blasted the Pentagon for failing to overhaul its outdated computer IT infrastructure after his work machine apparently took an hour to send an email and completely froze when he tried to use Microsoft Excel. "I am writing an open letter echoing some recent service member frustrations …
Eye-rolling aside, though, just by way of $perspective, say half the DoD needs a new box, so ~700,000 boxes, at say $1,500 for bulk(!)buy of decent boxes that might last more than 5yrs or even run Win11, well that adds up to a shade over A Billion Dollars.
That's more like 5 to 10 F-35s.
Quite a large amount to spend on non-pointybangy things. Particularly when the people making the decisions will never actually see anyone so lowly in the normal course of things, let alone talk to them.
The government/DoD does have more money than a billionaire. But they never work out the cost/benefit ratio of the spending.
Wages for 700,000 staff at a miserly $2,000 per month = $1.4 billion per month. But they're only 50% effective due to bad IT.
It's a bigger waste of money to have those 700,000 staff working at 1/2 speed for a few weeks than it is to get new kit.
There is a difference between OpEx (Operating Expenditure) for people's pay, and CapEx (Capital Expenditure), which is buying things. Different budgets, and people shifting expenses to someone else's budget and so being raised fo 'saving' money. It happens in commercial organisations so I expect it happens in government and military.
Plus, of course there is all the bad publicity they would get when the Secret data and pr0n on the recycled and 'wiped' old IT gets onto WikiLeaks. (Not that I'm accusing the USA military of ever watching pornography on official issue IT equipment, or of not being at all unprofessional when it comes to sanitising equipment.)
Secret data and pr0n on the recycled and 'wiped' old IT
It was a while ago but the last time I saw a MOD hard drive decommissioned after it had been used to store some very secretive stuff the data destruction technician came to the office. The technician disassembled the hard drive and ground the magnetic oxide of the platters while being witnessed by one of our security guards. The bag of the oxide was handed back to us for secure incineration as he was not allowed to take anything off site.
And it's not uncommon for organisations dealing with sensitive data to have decommissioned HDDs smashed with heavy equipment.
Working in education, of course no one senior would spend money on that kind of stuff, or indeed have worked out that old HDDs contained information about vulnerable kids and families.
I used to do as much destruction as I could. I'm aware that many other services didn't even remove the HDDs when sending ageing computers to the dump/recycling centre. Because we once got a query as to why some PCs had no HDDs. The answer being that some services did take the bloody things out first- just not all of them.
I have done site inspections of a couple of the high end recycling centers. In them the disks were removed and barcoded and the empty PC's and servers were sanitized to remove any customer identifying labels. The disks then went through a wiping to the customers spec. Failed drives went up the conveyor belt into the mechanical shredder where the scrap came out about the size of door keys. The really cautious customer could opt for all their drives to be shredded, but depending on how good the kit was that tended to make the recycling cost rather than generate a small surplus (for charity in our case).
Even though they were pretty thorough I still had our disks wiped on site first.
high end recycling centers
Unfortunately Local Authority education support services ( and I'd guess Social Care too) don't have such luxuries. In my case the alternative was taking the old PCs to the domestic recycling centre in the back of my car and hoiking them into a skip - or at the very best a semi-secure rehoming container. I'd bet a couple of quid that some services just bunged them into the big refuse bins and sent them to landfill HDD and all.
Having worked with the client machine team of a large corporate who had a 3 year replacement policy for laptops and 3-5 for desktops one of the big costs as they got older than 3 years was they started breaking down. One on site call costs > £150, more if remote. That doesn't include lost productivity. Phone based call ~ £50.
As I explained to my boss 20 years ago, you supply a salesman a company car at ~£200 a month but refuse to spend £200 a year giving them a decent laptop that doesn't break down and replace it every 3 years.
Any large organisation who can't buy a decent laptop from one of the big suppliers for less than £600 needs to sack their procurement team. Even desktop replacement laptops are less than £1000. Desktops cost peanuts.
Such decisions need to come from the top.
Didn't they mention bloatware? Government computing at the workstation level includes to two agents for system and software management, accounting and patching of systems, cranked up AD interactions, add in full-boat continuous antivirus scanning, encryption, continuous network scanning, always on Teams client, always on email client, whether Outlook or O365, network access agent, Adobe Cloud always on and I am probably missing other factors. Fully 1/3 of your processor and RAM is burned in this on a well equipped workstation. My policy from years ago for laptops was i7 and minimum 16GB RAM for the "basic functionality" people- i.e: spreadsheet as database type users. 32GB for "power users" That was the only way I didn't get daily complaints about how bloody slow their computer was.
The equipment cost is not the problem,
in a secure enviroment , only "approved" computers can be used
You are not allowed / capable of connecting any computer to the network,
you have no USB ports,
It takes 6 months for any PC to go through approvals,
costs a fortune
and come out with so much "security" that
if your doing anything apart from typing a memo
its usless.
half the DoD needs a new box
To be fair, I wouldn't be surprised if half the DoD only needed to reboot their boxes.
<rant>
Almost missed a customer session yesterday, was reminded by WhatsApp. The invitation was not in my calendar. An hour later got another, unrelated, invitation by mail, accept, check - not in the Calendar, either. Pattern recognition engaged and locked on target. Check OWA - not there, either. Open a ticket with IT... Think a bit... Reboot the computer for the first time in about 10 days on a creative whim - both invitations magically appear in the local Calendar and OWA. Cancel the IT ticket...
That's a rather new (<15 months old) MacBook Pro with the latest macOS - shouldn't need to be replaced even for USAF... Still no clue why rebooting should affect the server that OWA accesses... The case doesn't look all that different from needing an hour to send an email or to open Excel, but can't be fixed by a HW upgrade.
In my defence, I am not used to rebooting computers myself. Typing this on an 8 year old personal Linux laptop with uptime of more than 86 days... Got a new battery for it a couple of months ago, see no need for any other upgrade at this point.
</rant>
"To be fair, I wouldn't be surprised if half the DoD only needed to reboot their boxes."
According to the article he restarts his PC several times a day although it doesn't actually say whether this is a reboot.
"personal Linux laptop with uptime of more than 86 days"
True, for a server 86 days is nothing. But I don't see the point of not switching off my Linux laptop when it''s not in use.
Leaving a PC on and idle for any significant amount of time - unless it's serving files* - seems remarkably wasteful. It can't take that long to come back up again once you switch it back on, surely.
*My main home PC has a nice big shared partition on one of it's HDDs. For shared files when I'm using the laptop. It still gets turned off if I'm going out/to bed/meal times/etc. Not because it's a Windows box. But because it's a terrible waste of the Earth's resources to leave them on, idling.
OK - 700,000 people need a new box. Let's assume that they are paid on average $50K a year. Let's call that a thousand bucks a week.
Now, assume half an hour a day is wasted waiting for the machine to log in, to reboot, etc etc etc. That's going to be 10 hours a month, or, conservatively, about 100 hours a year. That's well over two weeks of productivity lost. That's well over two thousand dollars.
Suddenly, a $1500 box for each person seems like a bargain.
This is the problem. Not only do the oxygen thieves making the decisions have the newest kit they also sit 2 floors up from the data centre. Put them out in the remotest office you have with double the people the office was intended for and give them the oldest PC in the fleet.
.. and 56k modems.
These people have better mobile and data connectivity in theatre than most people have at home. That also leads to interesting issues re supply chain - they have an absolutely massive energy footprint and need lots and lots of energy/fuel when deployed.
Oh yes, I recall some AOC clip where she interrogated a military supplier over a clutch disk and what they charged. Now I assume that it's easy for them to dismiss her as uninformed, but her questions were ery much to the point. I wish all of these morons put as much effort in asking questions.
"I have sneaking suspicion that what costs an ordinary person $100, the american military pays 10 - 100 times more for."
Decades ago, when the world was young(er), I chanced to be a fly on the wall listening to a conversation between a group of mid level managers at a large defense contractor discussing whether they could manufacture a single -- quantity one -- aircraft qualified toilet seat for some ridiculous price. ($3000?) Their conclusion. They couldn't. Not with all the proper paperwork and required testing and reviews.
On the other hand, you don't want some complex hardware unit failing in use because an overzealous assembler put it together with hardware store Loktite when the stockroom was out of the Mil-Spec qualified stuff. And yes, I actually saw that happen. Took them all morning to get the lid off that box.
So yes, the military pays a lot for stuff. But it's not ALL due to stupidity or laziness.
There is (or used to be) something called COTS (Commercial Off The Shelf) procurement that allowed the military to buy some stuff at the local stores sometimes. But the rules were and probably still are complex. And how do you keep Sgt Jones from buying his batallion's paperclips and toilet paper from his brother-in-law who marks supplies for the government up 500%? It's difficult.
Another issue is things like durability and temperature range. Back near the dawn of time when the US required most able bodied young men to do some time in the military I played Company Clerk for a US Army Reserve unit. The Company Clerk then and probably still today was a sort of junior assistant office manager. Did filing. Prepared the Morning Report of significant events. Ran around and found people. Got folks forms for requesting vacation and such. Made sure the completed forms got to where they needed to go. Handled the mail, etc,etc,etc. There is nothing the clerk did back then that couldn't be handled by a Z80 CPU, a few hundred K of memory, Word Perfect and maybe a spreadsheet like Lotus-123. I doubt the job has changed much. BUT if the company deployed to the field their computer might have to boot and run in desert heat (probably COTS would do that) or in a subfreezing tent. (Maybe today's low end stuff will. 20th century OTS hardware mostly wasn't fond of low temps).
Bottom line -- it's complicated.
"And how do you keep Sgt Jones from buying his batallion's paperclips and toilet paper from his brother-in-law who marks supplies for the government up 500%?"
All supplies to be purchased at no more than manufacturer's list price.
"a Z80 CPU, a few hundred K of memory"
The Z80 can only address 64K so unless you have some clever bank-switching H/W it would have been limiting.
But the basic issue here is that DoD desk wallahs' desks aren't going to be suddenly deployed to the field. In fact no rank is quoted in the article so it seems likely he was a civilian.
>But the basic issue here is that DoD desk wallahs' desks aren't going to be suddenly deployed to the field
So if the whole DoD wasn't a socialist conspiracy to provide jobs for life, free college education, and free healthcare to the unemployable you could contract-out all the non running-shooting-killing part to civies ?
There is an army dept in an office block next to me. It is obviously some sort of administration dept rather than running-shooting-killing dept.
However they do have to wear boots and uniforms while queuing at Starbucks. They are also of a physique which suggests the typical Texan rather than the Special Forces
Conclusion:
1, Somewhere in the DoD is a supply of camo trousers with a 56inch waist
2, If they ever were to be deployed their camo wouldn't really conceal them - a forest might not be enough
3, They are unlikely to strike terror into the Russian army
I'm reminded here of Sgt Bilko, who after transferring out, Col "Melonhead" Hall finds his replacement super anal & causing him more paperwork, than Bilko's under the counter deals with local service stations exchanging tires for parts rather than ordering via official channels & kept all the base vehicles running.
https://www.youtube.com/watch?v=OrUwMMP7bKI
>could manufacture a single -- quantity one -- aircraft qualified toilet seat for some ridiculous price. ($3000?)
Even without military certification (was this made entirely in the USA by US owned companies, with a certain percentage being veteran owned and all following federal labor guidelines) and aviation certification (did 10 engineers sign off that this part is only a toilet seat and can't emit RF that could trigger the nuclear bomb in bay next to )
It's incredibly expensive to design and tool a single replacement part for some 50 year old bit of kit.
How much would it cost you to make a windscreen for a 1972 Vauxhall Viva from scratch? Probably more than $3000 !
Agreed, cost for qty. 1 with all the paperwork is always going to be ridiculously high, because it includes cost of design, tooling, multiple forms/certifications/signatures/testing, etc.
Per-unit cost decreases drastically once you manufacturer in volume, so if $contractor builds 1000 of these toilet seats perhaps their cost comes down to $100/ea (and if it's FAA or other TLA certified, that may be a bargain... I have an FAA-certified noise canceling headset for which I paid around $1300, even though it's likely not that much better than consumer stuff costing $300).
This post has been deleted by its author
We have $, £, and € on our keyboards.
We achieve this by having an extra key between Z and left-shift.
This takes the key that on US keyboards is above your Enter key.
Our Enter Key covers two rows and is narrower, giving is an additional key between Enter and '
That key is the # key, and our £ sign is where your # is.
On PC keyboards, but not Mac keyboards, the @ and " are swapped round.
~ is to the left of Enter rather than the left of 1
There are some other changes relating to characters that nobody ever uses.
His complaint also cited it restarting ten times a day with updates. That sounds rather more like Win10 (without setting the never restart during these hours) than Win7 or Win8 to be fair. If so, it needs the active hours thing setting via group policy.
Ten year old hardware also isn't a problem if managed appropriately. Personally I am just retiring a bunch of core 2 duo boxes that are almost certainly as least as old as their gear. However, they had an SSD, a Quadro card for a second monitor and 8GB of RAM as a life extension 5+ years back.
4GB was adequate for Win7 but Win10 struggles with less than 8GB. Do these boxes have adequate memory in them? Most of those problems could be attributable to lack of RAM, especially with 2 AV programs arguing. Easy solution; whichever the old AV program is needs to go.
That then just leaves the logon time, which he says was over an hour yesterday and an hour 20 mins from sign on to opening outlook. Roaming profiles across the wider network or something? If so, ruthlessly trim down what actually gets stored centrally in the profile to the mission requirement as there is obviously a lot more in there than needs to be if it's taking an hour.
None of it is insoluble, even without replacing the PC's.
I recall having to make recommendations, to boost the memory of a number of machines at a County Council in Zummerzetland, this was digested & actioned by the desktop services people, clocked up a nice bit of weekends overtime doing so.
Come Monday morning the IT Service Centre based out of a wooden hut in the car park we get swamped with complaints of slow running crashing machines, because the highly trained staff*, decided to play mix & match with the memory.
Normal function was restored by mixing the machines into two groups, one group was now fully populated with the older matched RAM & the other used only two banks of the newer larger memory modules.
I made certain the call notes reflected the nature of the upgrade project's execution.
*On another occasion, we were hit with a swathe of tickets reporting over heating machines, once again the desktop boys, tasked to install a suite of new machines over the weekend & pushing the fan vents right up against the upholstered cubical walls to make them look neat. Once again the call notes reflected this.
I lived the Navy IT dream for many years. When we decided to go with outsourced IT through the Navy/Marine Corps Internet (NMCI) it was - and remains - an absolute cluster fsck. The USAF looked at NMCI and wisely said, "EDS will own all our stuff? Er, no. We will let Navy stuff it." The Marines "My Ass Rides In Navy Equipment" ditched it
Then we doubled down and added SAP ERP to the cesspool.
Through it all the Air Force still had relatively functional IT. Now, apparently, they get to embrace the suck
One of his less well known Laws is that a committee will spend next to no time deliberating on, say, a new power station - but then agonise endlessly about a bike shed. Because they understand what a bike shed is and costs.
I suspect that there is an element of that in these sorts of decisions. Big things that go Bang are pretty incomprehensible to beancounters. But a $500 PC (even times several thousand) they can question.
Strictly speaking, Parkinson's law is the adage that "work expands so as to fill the time available for its completion."
Robert's first law of management is perhaps more specific to this situation: :The amount of time and effort spent on a decision is inversely proportional to its impprtance".
There's a book. Not just that bit.
In Parkinson's Law: The Pursuit of Progress, London: John Murray, 1958 a chapter is devoted to the basic question of what he called comitology: how committees, government cabinets, and other such bodies are created and eventually grow irrelevant (or are initially designed as such). (The word comitology has recently been independently invented by the European Union for a different non-humorous meaning.)[10][11] Wikipaedia
OK, bear in mind that the military computers actually run a virtual machine with a vetted computer image on it, for security and maintenance reasons. Yes, they can take an hour to boot up if you are in a some godforsaken outpost, but they are actually secure and can be repaired and reloaded with a new secure image remotely.
My first work PC was a Heathkit/Zenith 286. We had an in-house word processing system, VT100s all over, but everybody of course wanted PCs. The only way to get one is to claim you needed it for classified processing, meaning it came in a 40 pound Tempest shield. To get off the top you had to take out about 40 screws.
Then I got clever and specified a Leading Edge Model D as a deliverable "controller" from a research contract. It had a color monitor! And I bought memory chips to bring it up to 640K!
Some time later (2005), I rescued a laptop from the turn-in pile that couldn't run all the antimalware/spyware and put Linux Mint on it for fun. I of course could only use it off-line. After that I was done with Microsoft. I moved to an off-base office with an antique PC, put Linux on it, and have never used Windows since.
Getting off Windows is the key to safety, security, and using "old" machines. They could probably get rid of half the IT staff, too. The problem is the brass have never used anything but Windows.
The business world doesn't exactly thrive on Windows, it just does the accounting so loses the costs in the 'cost of doing business'. They then shed as much surplus productive capability as possible to maintain profitability.
Its a 'cart before horse' thing.
Incidentally, I have the same experience with Linux. "It just does work". I'd even pay money for it.
It does "just work". Yesterday I needed to scan some of my wife's papers for my accountant. Could Windows find the scanner? Nooooo! Could Ubuntu running hplip? Not a problem. Scan to PDF, zip with a password, email, and job done.
As time goes on, Windows just seems to get more broken and obtuse. Stuff that used to be easy in Windows is now just broken and lost in a vast sea of unnecessary complexity, behind a bland facade of useless non-information.
Worked for a few of the big ones over the decades,
Only last year,
the company I was at had windows 7, as it had been made "secure"
Most if not all of the programs used to design chips of the last 5 years did not work,
But did manage to get an update to 8G of memory,
Also no NVMe drives, as they are "not secure" ,
has to be slow rotating rust / approved ones,
No USB ports on the computers,
test equipment designed to be connected to a network , did not work,
We fight hard to design current stuff using out of date tools,
And the worry,
the managers,
who have been in the company since youth,
Seem to think they are ahead of the curve on desing !
perhaps their best path is smart/thin terminals and some kind of remote desktop arrangement against central servers/mainframes.
Seriously, take the responsibility and authority for end-user 'computing' away from local officials and civil servants and get back to a centralized model that fits the way the 'system' is known to work. People can still have their PowerPoint and Teams. Or linux-based. Or whatever.
Plus, all of those GS 9-15 local idiots that currently hide behind their desks are made redundant in favor of local techs or contractors who do nothing but swap broken thin terminal out.
Yup, this is an issue I have strong and unpopular opinions on.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
