The GDDR6 used on the 2 systems may also have a speed difference, so a difference cost per GB and the Blu-Ray BOM cost is likely to be somewhat higher than $20. The real cost saving may come from the fee paid to AMD for the processor as the greatly reduced CU count will allow the use of a lot of dies that would have failed the testing process for the full-size system.
The real problem may come in the future if to many of the S series units are sold compaired to the X series. What will MS do if developers start to just deliver games that play correctly across both the S and X. The value of the X design will be devalued and the PS5 will become the platform for 'advanced' development.