Absolutely get practice with FOSS
FOSS is a good idea but isn't a drag and drop replacement for Microsoft. It would make a huge amount of sense for the government to start using FOSS to get the experience in what it can be useful for, what the limitations are, how you will manage support, etc. Part of the initiative should absolutely be bringing more IT expertise back in house to the civil service (which means paying closer to market rates for good people). Loss of IT expertise is how the government keeps getting bullied, cheated, and scammed by consultancies and commercial vendors.
Just rip out Microsoft and replace it with FOSS sounds attractive, but across such a huge estate would inevitably be very expensive, and cause outages and general chaos. A plan to erode the use of Microsoft slowly is more feasible and gives MS an incentive to start lowering prices.