Reply to post: Re: Files

Want to train a dragon? You'll need 500 million files, 730TB of data, 54,000 CPU cores...

Korev Silver badge

Re: Files

It's a huge number of files.

In HPC land we'd normally use something like HDF5 to store the images (or other matrices) which are then very fast to access compared to have zillions of files all over the filesystem. Some people also use SQLite for this and I assume there are other similar tools that'd also work.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

SUBSCRIBE TO OUR WEEKLY TECH NEWSLETTER

Biting the hand that feeds IT © 1998–2020