Whisper it: FFmpeg 8 can now subtitle your videos on the fly FFmpeg 8.0 brings GPU-accelerated video encoding via Vulkan – and can now subtitle your videos automatically using integrated speech recognition. At the start of the week, the FFmpeg project released its eighth major version. It's codenamed "Huffman" after the Huffman code algorithm, which was invented in 1952, making it one …
Separate tracks for subtitles and different audio tracks are older than any file formats having long been a requirement for international trade. If you're selling to a country that wants subtitles, you'll need that, and if you're selling to somewhere that dubs, you'll need the atmos on a separate track. The digital stuff was effectively codified in MPEG-II for DVDs and any format since then has kept the approach, though AVI is a bit weird.
Tell you what I find odd, having recently revived my dish and installed the current Freesat* box, whereas Freeview (digital telly on UHF through a normal aerial) and Freesat on my tellies offers multiple subtitle options on suitable channels (for example, S4C has many programmes with both Welsh and English subtitles), the Freesat box only offers subtitles "on" or "off". I know the subsets are there because the tellies can access them directly, why can't the box?
Then again, neither of my LG televisions has a "subtitles" button on the remote so I presume you are expected to turn it on permanently in the menu. Fortunately I had an older remote which does have a subtitles button and was able to programme that into a One4All. The tellies still respond to the code :-)
M.
*I suspect that with Sky seemingly bent on moving entirely to telly through the intertubes, Freesat's days are also numbered, given that there is a large degree of satellite and transponder sharing going on, and there hasn't been a new satellite launched to 28E for maybe 10 years, so when the current ones die...
Don't know if it's what you are seeing, but there are different ways of transmitting subtitles, for example it can be embedded in the video stream (hidden data off screen, basically the old way), or in a separate parallel data channel. Perhaps the Freesat box only sees one of these?
PS: Been a long time since I looked at this, so things may have changed since then!
Hey Martin, have you noticed S4C (in this case, from Kilvey Hill) with audio description on appears to transmit the slightly delayed original audio, along with the description track, making the whole station sound echoey?
I haven't had a closer look, but I noticed it on my mums TV when I was last over there. I know she has audio description on, so this is just my guess at what's happening (I didn't have the opportunity to try switching AD off or doing anything else) - it could be the TV, but it's ok on all the other channels.. I didn't think to check the other TV's! doh.
"I suspect that with Sky seemingly bent on moving entirely to telly through the intertubes, Freesat's days are also numbered, given that there is a large degree of satellite and transponder sharing going on, and there hasn't been a new satellite launched to 28E for maybe 10 years, so when the current ones die..."
As far as I remember the current satellite used for Sky UK and Freesat has an service end-of-life around 2028 or 2029 with no plans currently for a replacement satellite. So a Sky UK Satellite and Freesat turn-off may not be far off.
"Which is ridiculous, really."
How so?
I assume that by then Sky will expect all their customers to be using their Internet-based service.
As for FreeSat, well who knows, I guess people will be expected to be using iPlayer, BritTV (is that the right name for the ITV/C4/C5 streaming service?) etc over the Internet by then.
Realise I'm replying to this late, but...
...I also believe that DTT's days (Freeview) are numbered. Terrestrial transmitters don't cover the whole of the country at the moment, and some more recent fill-in transmitters have (I believe) been set up to rebroadcast TV received via satellite (possibly satellite is a backup option on others too). Making the huge assumption that this is via Astra, that puts a lot of these transmitters at risk. If a large part of viewing moves to IP then the financial case for maintaining terrestrial transmitters becomes difficult, even if many people find it difficult to swap to online delivery.
The problem with IPTV is that apart from Sky, there aren't any real "integrators" where you can get a mix of channels under one EPG. Without Sky, you'll be having to subscribe to a minimum of five different streaming services (BBC iPlayer - which also covers S4C if you don't want to use S4Clic - ITV, Channel 4, Channel 5 and UKTV) to recreate much of what is available directly on Freeview or Freesat. Just trying to work out what's on offer from each "broadcaster" will be a nightmare!
By the way, Wikipedia states that BritBox was subsumed into ITVX, and as far as I know it never actually offered the "normal" (linear) channels for viewing, just collections of programmes on demand (please correct me if I'm wrong...).
So, what am I moaning about?
M.
"Freely" is the integrator.
All channels will be on the EPG regardless of source (terrestrial /satellite/IP)
This is the replacement for Freeview, and will be instrumental to their deprecation of traditional OTT sources.
https://en.m.wikipedia.org/wiki/Freely
I share your concerns though. Seeing as multicast never took off, how will they handle millions of simultaneous live 4K+ broadcasts?
I guess a CDN in every UK ISP would cover the internet side, but that would suck for smaller ISPs.
But what about the capacity of links to local exchanges?
Even the "last mile" isn't a one to one link any more (the telegraph pole outside my parents house which used to distribute individual copper lines to 8 surrounding houses now has a fibre feed and fan-out)
I hope it’s more reliable than auto-generated sub-titles seen elsewhere or the audio- transcription Teams does.
Both are as useful/reliable as the cloth eared bints Alexa and Siri are that can’t understand clearly spoken English (other languages are available).
Teams esp. concerning given Microsoft bought Nuance a few years back.I hope its Medical notes transcription is better.
I've tried to subtitle a film before... a film that's close to my heart bu only available with German audio and, being a low-ish budget production, with poor sound quality.
If this can provide a _skeleton_ subtitle file with correct timings but lots of gobbeldigook text, that would be a huge help because then I can just fix the text and eventually translate it, so that you too can enjoy this hidden marvel of football film (hated by some fans who are too serious about football)
https://de.wikipedia.org/wiki/Nordkurve
It depends on the length, of course, but in the past I got a head-start at subtitling something properly, by uploading it as a private video to YouTube, then downloading and extracting the auto-generated subtitles.
I could then edit that file to correct the mistakes.
Would something similar work for you?
(yt-dlp can d/l subtitles only, or you can extract them from a full video with ffmpeg)
Apparently transcribes English, Spanish, French, German, Italian, Portuguese, Dutch, Russian, Chinese, Japanese, and Korean and can translate from those to English.
If it can manage better than some of the dreck served up by Open Subtitles, then that's already a great leap forward. It has been known to hallucinate "please like and subscribe" and similar, I wonder if anyone can guess where OpenAI got the training data from...
I admit that I have no interest in gaming, and therefore don't know much about GPUs. That said, I do use FFMPEG a lot, mainly convering H264 videos to H265 (for reasons, mainly disc space).
So what are the requirements for this use case? Will a £100 Radeon show a marked reduction in encoding time, or do I need to spend £1000 on a higher spec card? I'm a Debian Linux user, so obviously Vulkan 1.3 support is a given, but what else is needed? I assume more £££ = more speed, but as it's doing maths rather than outputting to a screen, I wonder if a lesser card is good enough.
Anyone?
My experience will all those GFX-card encoders is that they target streaming. Try to use higher compression setting and you may get a hard freeze on nvidia (in Windows Terms: Your mouse is stuck, you don't even get a blue screen). But since you use debian you won't have that crash issue, but possible the same limitations: It cannot beat software encoding regarding size vs. quality.
MSU encoding tests at https://compression.ru/video/codec_comparison/index_en.html may be the one you should look for since they do quite extensive testing. But does not include your specific scenario, but allows comparison hardware vs software.
Ah, so are you saying this new speech to text option will only work if using the GPU for hardware encoding then?
I was hoping it would be a separate thread and do the sub-title generation purely from the audio track leaving me to choose how the video source is encoded.
On the other hand, I think my initial reaction was "ooooh! Shiny!" and on reflection I don't really have all that much use for subtitling in the original language, more the occasional need to subtitle in English from a non-English soundtrack where I can't find one already done on t'internet :-)
Although as I tyoed that, I wondered if FFMPeG can generate textual subtitle files from non-English source and I could then run through a translator.
You might be able to script that, but you might need mkvtoolnix on top. ffmpeg internally does use the mkvtoolnix muxer for mkv. But subtitle extration, then send through some translator, save it as new subtitle file next to the old one, that sounds like mkvtoolnix for extracting to me. Which can handle various source types, not just mkv. Regarding some API to auto-translate: Sorry never needed that. English and German are always available, quite often both.
> But they don't always allow hardware acceleration to other software.
Asahi Linux supports Vulkan on the GPU in the M1 and M2. That means FFmpeg on Linux on Apple Silicon will. But only for those specific codecs I named. This does not, AFAICS, include whisper.cpp -- but I welcome correction on this.
Agree. Bang for buck Intel's discreet GPU's have the edge as they all have AV1 encoding built in. You'd need a 50x0 Nvidia or 90x0 AMD card to get GPU AV1 encoding otherwise. If you're only worried about H265 encoding I believe everything from the Nvidia 20x0 and AMD 5xx0 series forward has that built in.
1) The word is "discrete".
2) You're off by a generation on both Nvidia and AMD. Nvidia has AV1 encoding support since 40 series, AMD since 7000 series, and all Intel Arc GPUs support it.
Fun fact: AMD had apparently released RX 7400, which needs only slot power and supports AV1 encode.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
