Re: "unforeseen technical complexities"
Yes, old farts know how things work now, how things are connected together, and all the assembled technical cludges and how organisational culture actually works. As already noted, requirements will change even if the original spec was accurate, and that's new stuff. The OF are well placed to know how to deal with it, but they don't know what they will be dealing with. This is as true of the private sector as public, but the public sector are subject to transparency requirements that companies are not.
There is a fundamental reason why the public sector keeps getting shafted that I've not seen mentioned yet, and that is not just because (a) requirements are incorrect, and (b) further change is inevitable, it is because no attempt is made to understand the commercial model of the supplier.
We're all familiar with the big services provider commercial model to bid low, even loss making, knowing for certain that change will happen over any 3-5 year period (as well as a good chance of change because the spec wasn't correct), and then make a fortune on variations and non-standard service requests. If anybody isn't familiar here's a fine private sector example: Huge corporate with 20,000 employees signs deal with big US IT services provider to TUPE across the in house team. Obviously the deal was scrutinised by the buyer team for the big corporate, by the legal team, and the big corporate believed it would save money. Now, with many employees leaving, it wasn't unusual at that time for departing employees to get a PAC number to port their corporate mobile to a private contract. Managers knew this only involved a phone call and the telcos couldn't charge for it. But because that activity had to go through as a service request, and hadn't been written into the contract, it was charged at the non-standard service request rate, of £440 each. The company had paid over quarter of a million quid for "free" PAC numbers before it realised how it was being shafted, but there were of course many other forms of NSSR and variations that meant the vendor's tills kept jingling.
The first lesson here is that if you can do it yourself, you should. The second is that if you have to use an external vendor, understand their commercial model before negotiations start, and make sure there's NOTHING that isn't costed at a fair rate, and that the bid includes a commercially attractive margin. So no pricey NSSR rates, no "to be agreed" rates for variations, and also complete transparency. Vendors won't like this one bit, and they will try and fight back, but the first part of contract negotiations should be the buyer asking "Based on what we think we'll need, how much will that cost you, what is your resourcing model, and what is your estimated gross and operating margins on that work?" If the vendor won't answer, show them the door. Next up, "we'd like you to validate our specification as part 1 of the contract, that it is complete and can be delivered" this is to get the vendor to be partially responsible if things turn to slag. Then "What will the cost of unplanned variations and changes be, and how much margin will you make on those?". And so on - assume at every turn the vendor is entitled to and should be earning a decent margin for the relevant level of risk they are taking, also assume that the vendors are lying, thieving bastards who make the Thenardiers look like honest citizens. If the buying team haven't examined every single clause of the contract and worked out how the vendor hopes to make more money through it, then they haven't done their job properly. But of course, the sales team are vastly better paid than the procurement team, and in addition get huge bonuses. Buyers don't get anything like the same bonuses for a job well done. Then there's the fact that organisations make big IT services purchases rarely, the sales teams do this day in day out. And finally, if we wanted things to be different, we'd need to (a) exclude bigwigs on the buy side from any influence in the process (as that's one of the tools the sales team use to force buyers into poor decisions), and (b) re-write the public procurement rules.
All this could be done, but it won't be done.