But if the unit tests are thorough enough, it shouldn't matter what each unit gets
No - that's the point. A system is more than just the sum of its parts.
The easiest example to pull is timing - each unit can perform exactly as specified, but if they yunits are improperly intergrated, the syustem as a whole might generate results at the wrong time (usually late). This can be the difference betwwen the system as a whole being a tribute to its engineers, and it being worse than not having it at all.
Vic.