Not fast enough
Any type of external memory is far slower than the level 1 cache memory. There are a number of unavoidable delays in accessing external memory (the speed of light being one of them). SRAM with the same cycle rate as the level 1 cache memory has a high power requirement. CPUs have only small amounts of L1 cache so the power consumption is acceptable - having gigabytes of memory with that speed would result in PCs acting as high power heaters (multiple KW).