Because the images have a hash based on their contents it doesn't matter _where_ you pull them down from as long as you've validated the hash. They're effectively signed (unless there's a weakness/collision discovered).
On the flip side of this, they should be the easiest thing in the world to cache, because you're downloading them based on a request that indicates the contents that you want to download. The image for a particular hash will _never_ change, so your caching can be based off LRU and you're sorted.
Even Microsoft have worked out how to do this, and their build agents have a certain number of popular images cached _on the build server itself_.