Netflix reveals massive migration to new mix of microservices, asynchronous workflows and serverless functions Netflix has revealed it’s built a new media ingestion and distribution platform and expects to spend much of 2021 on a migration from what it describes as a “large and complicated legacy system”. As detailed in the streaming firm’s tech blog, the new platform is called “Cosmos” and is the fourth generation of a tool that is …
At the scale of Netflix, I always wonder if it'd be cheaper to do it all on prem like Dropbox decided to do a few years ago. For this kind of bursty workload having a $CLOUD autoscale is nice, but for other workloads it's probably cheaper to run your own hardware. Also, most Clouds clobber you on data egress which is bag news for a media streaming platform.
I'd love to see their figures!
Tech develops strange bedfellows. When I was with hp, we brought critical drive parts from suppliers to Western digital. Just wasn't cost effective to design our own disk drive heads as the technologies became more and more advanced. AWS was the most advanced cloud service when I was using it 10 years ago. I think amazon makes more profit from AWS than selling stuff.
You'd hope that you'd persist the state of a computationally-intensive process somehwere or you've just wasted a lot of CPU cycles. Putting the state somewhere else and refusing to look at it doesn't mean it's disappeared. I periodically tidy up my wardrobe and move things around inside - and instances of clothing items might periodically be replaced - but the contents remains basically the same and on the outside it's still a wardrobe.
The adoption of any new technology is an opportunity to spring clean, but I'm always wary of claims that a specific change of design approach has transformed an existing application.
Output is very much not the same thing as state - of course you want to capture the output.
Stateless just means that the output you get depends only on the input you supplied, and not on any /state/ that the service maintained internally (which in turn means that using more instances is much easier, since there is no separate setup to do for each one, and no need to route requests consistently to the same one).
They have invested a lot of resources into FreeBSD which uses Jails rather than Docker (I know not a direct comparison).
I wonder if anyone knows if this recent infrastructure is for a different part of the system or if they have been phasing out FreeBSD?
I think it would have been very cool if they went with Jails rather than Docker, it would have been great to see any presentations they often show to the FreeBSD community on quite how the orchestrated a number of Jails.
I think the article is about the stuff they need to offload to AWS to crunch through, so compartmentalising for commodity processing on hardware you don't have to worry about is key.
But I do agree it would be interesting to see what they're doing with BSD, presumably for less ephemeral stuff on their own hardware. And isn't FreeBSD pushing for its own version of Docker?
You're still going to want to transcode to different formats and resolutions: h264, h265, AV1, HD-I, HD-P, etc. Basically what Google does every time someone uploads.
But I suspect they have other workloads they want handling. It easy to imagine them wanting to do some near-realtime crunching of the logs because the Netflix client generates oodles of data,
My take on this article (and I have no inside knowledge, this is just my guess) is that this is related to the organisation and processing of media.
FreeBSD is used for the distribution, and from the FreeBSD commit logs, the netflix guys are still working on tcp and other networking tweaks.
Besides, they use their own FreeBSD cdn for distribution - the article is referring to their cloud stuff.
Maybe some day soon they'll fix the "notify me when this is released" thing in the coming soon list. Because, for me, many of them until themselves, and if I go into the list and come out and go back in again, a fair few that did get ticked no longer are...
Secondly, it would be nice to have a watch list that is sorted by some sensible means (alphabetical, date added, etc) rather than what seems to be essentially random placement.
Finally, the ability to flag something as "seen it" so the service doesn't keep suggesting stuff you watched just last month/week/yesterday.
Netflix used to support manual watchlist ordering. They still have an option for it but, as far as I can tell, it doesn't do anything. Mine is still set for manual (and I've tried turning off and on) but I just get the crappy random ordering they seem to like so much.
The watchlist ordering and invisible limit on watchlist length are the two things that drive me nuts about their service. The "invisible limit" is when I can add a show to my watchlist and, per the show, it is in my watchlist but it does not show up when I go to "my watchlist" - I'm assuming this happens after you reach a certain length to your watchlist (and, with the random ordering, it can push shows near the end off into invisibility).
"Paying more for 4K vs HD?"
I agree that the service should try to send "best quality" like Prime Video does, but you aren't exactly comparing like with like here. The bump in quality also comes with a bump in how many concurrent users can be streaming stuff, so there is a difference other than "better looking shit".
You sound like every old person with *shit* eyesight that has ever compared HD to SD. If you can't see the difference the extra £2 a month costs you, don't pay it. Or buy a better TV set. But stop whining about it, the rest of the world can't all be wrong.
To be fair, this site covered this issue ten years ago (long before 4K in living rooms was sensible), and the data are quite interesting. From personal experience, I like having 4K, but at my normal viewing distance I genuinely can't tell the difference from 1080p (and according to my optician my eyesight is above average [+0.2 dioptre], though my colour vision is crap). Sometimes it's nice to get really close and marvel at it though.
Netflix needs to adapt deepfake technologies to make the lip movement match the dubbed language - then I could actually watch it.
My brain must use a combination of hearing and lip reading to understand speech as I can't watch a dubbed program without getting a headache. Loud background sounds don't help the situation. Unfortunately, subtitles don't work out any better for me.
I also hate the dubbing – but you can switch back to the native language with subtitles quite easily. In my opinion, it much improves the experience, but I guess that still depends on the content. In any case, I find I quickly get used to subtitles, although maybe you don't like them because you want to scroll through your insta at the same time as "watching" something... </youthOfTodaySnark>
One of the reasons for the disastrous separation from Lloyds Bank was the late decision to use the load-balancing software that Netflix had open-sourced. It turned out that a routing service that worked well for routing requests to a CDN was not that good for stateful banking services.
They might call the deployment 'Strangler fig', but others will call it 'Canary testing': Just because they are "web scale" does not mean it is any good.
I'm a bit confused by why this is such a big deal. I did a video service prototype and it wasn't that much code. FFmpeg supports intermediate stream formats so one input can feed a large tree of filters, codecs, and containers that run concurrently. A big machine could run this entire processing tree so it was very simple to scale horizontally. The only actual hard part was fixing FFmpeg bugs.
Just because you use containers or some other fangled technology does not mean that this is serverless. Underneath everything will be servers and storage, in Netflix case, huge numbers all busily burning electricity and generating heat. You may not manage any of it but if the servers were not there, Netflix would not exist.
"Serverless" appears to be yet another marketing buzzword to make something old appear cool. It is irrelevant where you run a service, on your own servers or someone else's, there are still CPUs, memory and disks in boxes doing stuff.
“A Cosmos service retains the strong contracts and segregated data/dependencies of a microservice, but adds multi-step workflows and computationally intensive asynchronous serverless functions.” San Miguel wrote.
The anti-singularity approaches!
(The anti-singularity is my take on the future of computing. One by one, every piece of technology will become more complex than is possible to maintain. And there, we will be stuck, unable to go backwards or forwards, forwever doomed to use half-functional garbage. For evidence, see any example of modern techonology).
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
