• 0 Posts
  • 88 Comments
Joined 3 years ago
cake
Cake day: June 22nd, 2023

help-circle
  • It looks like they ran the test case and triggered the crash. Therefore the issue is not confabulated.

    Also, I’m unconvinced that use of ffmpeg inside of Google services is relevant to this. Google services can sandbox executables as much as they like, and given the amount of transcoding they do (say for youtube), it would surprise me if they’re not using gpu or hardware transcoders instead of ffmpeg anyway. Instead, they may care more about ffmpeg as used in browsers, TV boxes, and that sort of thing. That puts them in the same position as the Amazon person who said the ffmpeg devs could kill 3 major [Amazon] product lines by sending an email.

    If a zillion cable boxes get pwned because of a 0-day in ffmpeg, well that’s unfortunate but at least they did their due diligence. But if they get pwned because the vendor knew about the vulnerability and decided to deploy anyway, that potentially puts the vendor on the hook for a ton more liability. That’s what “ffmpeg can kill 3 major product lines” means. So “send the email” (i.e. temporarily flag that codec as vulnerable and withdraw it from the default build), seems like a perfectly good response from ffmpeg.

    The Big Sleep article is really good, I read it a week or so ago, sometime after this thread had died down.



  • Have any of the google-submitted vulnerability reports turned out to be invalid? Project Zero was pretty well regarded.

    Yes I know about the asm code in ffmpeg though IDK if it’s doing anything that could lead to a use after free error. I can understand if an OOB reference happens in the asm code since codecs are full of lookup tables and have to jump around inside video frames for motion estimation, but I’d hope no dynamic memory allocation is happening there. Instead it would be more like a GPU kernel. But, I haven’t examined any of it.

    Anyway there’s a big difference between submitting concrete input data that causes an observable crash, and sending a pile of useless spew from a static analyzer and saying “here, have fun”. The Curl guy was getting a lot of absolute crap submitted as reports.

    From the GCC manual “bug criteria” section:

    If the compiler gets a fatal signal, for any input whatever, that is a compiler bug. Reliable compilers never crash.

    That sounds like timelessly good advice to me.


  • Yes, but the FFMPEG developers do not know this until after they triage all the bug reports they are getting swamped with.

    With a concrete bug report like “using codec xyz and input file f3 10 4d 26 f5 0a a1 7e cd 3a 41 6c 36 66 21 d8… ffmpeg crashes with an oob memory error”, it’s pretty simple to confirm that such a crash happens. The hard part is finding the cause and fixing it. I had understood the bug search to be fuzzing controlled by AI so I referred to it as fuzzing. Apparently though the AI is also writing the bug report now, so yeah ok, maybe there is potential slop there.

    “Don’t ship software with vulnerabilities” sounds good in a vacuum,

    I said KNOWN vulnerabilities. Make it known vulnerabilities without known mitigations if you prefer.

    I wrote a few of those GNU coreutils that the Rusties are now rewriting. I don’t remember hearing of any CVE’s connected with any of them, though that is mostly because they are uncomplicated.

    Here’s all the Debian security advisories for the past year or so. There aren’t THAT many, and they are mostly in complicated network programs, the Linux kernel, etc. Also a lot aren’t actual vulns: https://www.debian.org/security/

    This was the first search hit about ffmpeg cve’s, from June 2024 so not about the current incident. It lists four CVE’s, three of them memory errors (buffer overflow, use-after-free), and one off-by-one error. The class of errors in the first three is supposedly completely eliminated by Rust. No idea about the fourth. Not claiming that a Rust reimplementation of ffmpeg is anywhere near feasible. Dunno if the current set of CVE’s are comparable but it’s a likely guess. Anyway, as SPJ likes to say about Haskell’s type system, the idea is to stop fixing bugs one by one, and instead eliminate entire classes of bugs. We can’t fix everything but we can certainly do better than we are doing now.

    I saw earlier you mentioned google keeping vulnerabilities secret, and using them against people or something like that,

    That was listed as an example of what not to do, not a proposal of an approach to take.



  • This is why compilation flags exist. You can compile software to not include features, and the code is removed, decreasing the attack surface. But it’s not really ffmpegs job to tell you which compilation flags you should pick, that is the responsibility of the people integrating and deploying it into the systems (Google).

    ffmpeg in fact comes with a default makefile that excludes a bunch of modules for various reasons, some for license incompatibility issues and some because they’re not considered to be of production grade (so theyre released for testing etc.). It’s nuts to suggest continuing to ship something with known vulnerabilities without, at minimum, removing it from the default build and labelling it as having known issues. If you don’t have the resources to fix the bug that’s understandable, but own up to it and tell people to be careful with that module.

    AI generated

    AI tools were apparently used for locating the bugs but the reports were real and legit.

    But it really should have been fixed by Google, since they brought it up.

    It would be great if Google could fix it, but ffmpeg is very hard to work in, not just because of the code organization but because of the very specialized knowledge needed to mess around inside a codec. It would be simpler and probably better for Google to contribute development funding since they depend on the software so heavily.

    You might remember libav, a now long dead ffmpeg fork from some years back. It was well intentioned and had good programmers involved, but it just couldn’t handle the technical demands of developing something like ffmpeg. ffmpeg is a messed up project in some ways but it’s extremely impressive. Being able to find bugs (say by fuzzing) is much different from being able to fix them sanely. If you’re google and have infinite hardware like they do, you can do more fuzzing than anyone else, and that’s valuable to any project with this type of exposure.

    It’s not really ffmpeg’s job to pick a sandboxing software (docker, seccomp, selinux, k8s, borg, gvisor, kata),

    Those approaches would be ridiculous bloat, the idea is just supply some kind of wrapper that runs the codec in a chrooted separate process communicating through pipes under ptrace control or however that’s done these days.

    but instead the responsibility of the people integrating and deploying the software.

    The ffmpeg CLI tool is in fact an integration of the software and should probably use the wrapper, at least for suspect modules. Note that all major web browsers already do something like this and have done it for years, for exactly this reason.



  • I don’t understand how a bug is supposed to know whether it’s triggered inside or outside of a google service. If the bug can only be triggered in some weird, google-specific deployment, that’s one thing, but I don’t think that’s what we’re talking about here. If the bug is manifestly present in ffmpeg and it’s discovered at google, what are you saying is supposed to happen? Google should a) report it under the normal 90 day disclosure rule; b) report it but let it stay undisclosed for longer than normal, due to the resource contraints ffmpeg’s devs areunder; c) not report it and let some attacker exploit it? (b) might have some merit but (c) is insane. Once some bad actor finds out about the bug (through independent discovery or any other way), it’s going to be exploited. That might already be happening before even google finds the bug.

    FFmpeg’s codebase and dev community are both exceptionally difficult and that is not helping matters, I’m sure.

    There are a bunch of Rust zealots busily rewriting GNU Coreutils which in practice have been quite reliable and not that badly in need of rewriting. Maybe the zealots should turn their attention to ffmpeg (a bug minefield of long renown) instead.

    Alternatively (or in addition), some effort should go into sandboxing ffmpeg so its bugs can be contained.


  • AI reports are ignored because they are so frequently crap that they are almost not worth investigating. If these ffmpeg reports are from Project Zero though, they are presumably real. Shipping code with vulnerabilities is always a terrible idea. If Google can find them, attackers can also find them.

    I do have to wonder how many of these vulnerabilities are actually in the assembly language parts of the codecs. I had guessed they were more likely to be at the higher levels.