Want to train a dragon? You'll need 500 million files, 730TB of data, 54,000 CPU cores...

Family favourite DreamWorks Animation has built a cloud platform powered by microservices that uses a graph database and Gremlin query language to guide the production of its films. This digital pipeline handles everything from early pre-visualisation to the final render and will be employed in its next feature. This setup has …

[Craig 2]

Gremlin...

Don't get them wet. (The servers)

Keep away from bright lights. (The staff)

No coding complex queries after midnight??

[Aladdin Sane]

Yeah, pretty sure this could run Crysis.

[Tromos]

730TB of data and 54,000 CPUs

Or, alternatively, a mid-range 2029 smartphone.

[Zarno]

Re: 730TB of data and 54,000 CPUs

But will that future phone still have one byte(bite) missing from close to the core?

Sorry, Apple puns, they write themselves...

[bombastic bob]

Re: 730TB of data and 54,000 CPUs

and that future phone, with all of it's CPU cores, RAM, and computing power, will still have a 2D FLATTY interface, allegedly because it "wastes CPU cycles" to make it 3D Skeuomorphic... (whether people PREFER the 3D or not, and every study _I_ have seen says it's about 2:1 in favor of the 3D-like appearance of things as far back as Windows 3 and OS/2 PM 1.2...)

and people will STILL use their uber-powerful smart phones just to look at cat videos, e-mail, and the next gen of Fa[e][cebook. And maybe a game app or two...

meanwhile, Dreamworks would hopefully settle for an integrated solution where all xxx,xxx cores can work with one another gracefully. And probably running Linux, or a BSD or Solaris derivative OS...

thinking of 'mythical man month' there's 'mythical CPU core count increase' too, for determining how effective a 'moore's law' architecture evolution is when you can't just double the frequency any more.

[Anonymous South African Coward]

Here there be Dragons.

[Pirate Dave]

Files

Million or billion? The headline says 500 million, the article says 500 billion. That's a lot of files either way (for a 2 hour movie that would have a max of 172,800 individual frames at 24FPS), but one seems a whole order of magnitude greater than the other.

[Korev]

Re: Files

It's a huge number of files.

In HPC land we'd normally use something like HDF5 to store the images (or other matrices) which are then very fast to access compared to have zillions of files all over the filesystem. Some people also use SQLite for this and I assume there are other similar tools that'd also work.

[I.Geller]

Re: Files

If they use AI technology each word in each pattern is annotated with at least one dictionary definition, which is a file; at least 3-5 million files. Plus synonymous clusters on each paragraph are separate files; 1-50 files on each paragraph.

[Kobblestown]

Re: Files

I don't know what you're talking about, but I'm pretty sure you're doing it wrong.

[David 132]

Re: Files

Technically, three orders of magnitude greater.

[Korev]

Re: Files

Do you know where to log a complaint?

[bombastic bob]

Re: Files

/var/log/complaints.log

it's a symlink to /dev/null

[I.Geller]

Apache Cassandra, a NoSQL database originally developed at Facebook, and Apache TinkerPop, a graph computing framework supported by multiple database vendors.

I hope they create synonymous clusters?

[Anonymous Coward]

Hold on a sec. Somebody mentioned “How To Train Your Dragon” and there’s no NetApp reference yet ???

Well did you know that NetApp was involved in the production of “How To Train Your Dragon” ?

One of dreamworks employees sent an email. And that email was actually stored on a NetApp system somewhere.

And then a graphic designer had a NetApp mousepad.

So proud to be part of something big and important !!! Wouldn’t be possible without the great culture.

[swm]

Wow! However they do it the end result (movie) is impressive!

[Bob Ajob]

Key messages

The key messages here are very encouraging from a DevOps perspective. They had the balls to experiment at a scale that affected their whole pipeline. Creating a dummy show probably cost less than the cloud hosting and time spent learning the new tech. Bravo to whoever had the vision to do this.