Nvidia GPUs give smut viewed incognito a second coming Canadian student hacker Evan Andersen says Nvidia graphics cards retain content users would rather not be preserved, such as the material appearing in web pages viewed in the supposedly-private "incognito mode" offered by Google's Chrome browser. The flaws were reported to Nvidia and Google in 2014; the former did not respond …
This post has been deleted by its author
Is there anyone anywhere who doesn't watch the occasional bit of porn?
Contrary to popular belief, yes such people do exist. They have quite an active community/cult on a certain subreddit about the evils of porn addiction.
I would also imagine that asexuals (the orientation, not the method of reproduction) don't indulge in porn. It would, after all, be a little more than a waste of time for someone who is neither capable of being physically attracted to anyone nor at all interested in sex.
So basically the guy has admitted to everyone he watches pr0n...
Yes, and he can stand proud for having seen it twice and realizing what that meant. NVIDIA and Google are the ones who should be ashamed of their limp performance, though. Google puts out an application that it says will clear all evidence of itself, is informed this is not in fact the case and then blows it off as someone else's problem. And what does NVIDIA say concerning a flaw in their memory management that makes it trivial to do a side-channel (I so wanted to say "back door" here, but it really isn't) attack to take screen shots of whatever their cards put up on the screen? .... Anyone? No? It appears to be at least misleading in the first case and negligent in the second.
But Google created the content AND presented it on an OS with GPU compositing, meaning they KNEW their stuff would show up on the GPU's RAM. As as saying goes, "You made the mess. You clean it up." There's not much Google can do with active Incognito pages in GPU RAM since it must be in an accessible state for the GPU to put it on the screen. But once the page closes, Google should assume the memory won't be cleaned up on its own, so it should zero or otherwise blank the page before releasing it.
"Google should assume the memory won't be cleaned up on its own, so it should zero or otherwise blank the page before releasing it."
With main memory, the physical ram is constantly allocated and reallocated. Pages are swapped to disk and back. The physical ram is a fluid object. This means that when a page of physical ram is allocated to a process, it may be a reallocation of a page used a second ago by a running process. If the VM subsystem didn't clear it before page allocation, the second process would see what is currently in it - SSH keys, for instance. The first app has no idea that the page will be reallocated and therefore cannot clean it. In any case, "cleaning" it would mean deleting data necessary for the process to continue running.
This is why memory should always be zero'd before allocation - you literally have no idea what just used it or how, and the only thing that can zero it effectively is the OS's allocation subsystem.
Now I'm kinda wondering if there ever was a video driver from any manufacturer who went to the trouble of actually cleaning memory - to be honest, I remember seeing this sort of "flash something at video resolution change" quite a few times in the past; I always considered it shoddy and inelegant programming (wait, is there any other kind left?) but didn't quite realize the privacy implications...
"If it's a privacy problem, it should be up to the os or driver to clear it"
No, it should be up to the app that's doing something privacy sensitive to clear it before closing it. Zeroing a large buffer has a performance hit, and it's not needed most of the time. I would also expect an app to ensure a newly requested buffer was initialised with the desired content before displaying it to screen.
No, its clearly a hardware issue. The MMU being used seems to not be doing its job. A proper MMU should, at the very least, clear the contents of any memory that is to be assigned for use. Nowadays, it should be clearing memory as soon as its been freed. At the very least, it should be part of routine house-keeping on the part of the MMU.
Although a part of me suspects that there isn't even an MMU present at all, which is quite troubling...
Usually, a driver has routines for 'change video mode' and 'clear screen' (== zero framebuffer); the most obvious thing to do is clearing after mode change, but it depends on the application writers to do both in sequence. Zeroing a large buffer might take a while, and if the next thing you do is loading a full-screen bitmap anyway, clearing the screen first might be considered a waste of time. You could even figure out the new framebuffer first, load the bitmap and then switch modes, in which case you obviously don't want to clear after switching. Or it's simply a bug in the application.
IMO in this case the bug lies squarely with Chrome: if you go out of private mode, you erase ALL of the stuff you did until then.
Zeroing a large buffer might take a while
But doesn't an O/S kernel zero out regular memory before handing it to an application? And aren't there specific CPU instructions for doing this to make it as efficient as possible?
GPUs are often touted as having large memory bandwidth, so surely they can use a bit of that to zero out a newly allocated region? Surely a new framebuffer allocation isn't going to be in the "hot" path of drawing an image on the screen but in the initalisation code path?
"But doesn't an O/S kernel zero out regular memory before handing it to an application?"
Why should it? The memory you get from an allocation should be considered to be "undefined", and therefore it should be the applications' responsibility to handle it accordingly, using as you said common memory-fill techniques if necessary.
"GPUs are often touted as having large memory bandwidth, so surely they can use a bit of that to zero out a newly allocated region?"
Again, that's if they WANT that. If you're allocating the framebuffer to say play a video, then zeroing is redundant. You let the video take care of that.
I'm agreeing with the point that if an application is touting a low-trace operating mode, the onus is on the application to ensure low-trace operation.
Why should it? The memory you get from an allocation should be considered to be "undefined", and therefore it should be the applications' responsibility to handle it accordingly, using as you said common memory-fill techniques if necessary.
Unless of course the application wants to see what was in the memory previously...
It's basic defensive programming. You always assume the things you can't control were written by idiots*. So, whose fault was it that the memory wasn't cleared - the browser, the OS or the driver? All of them. They didn't know another part of the stack had cleared it, so they should have assumed it was their job. The only one not to blame is Diablo, as they don't care what was there previously, although it would have been neater to blank the frame buffer.
* - It's also best to assume the code you wrote was written by an idiot as well
Basic defensive SECURITY programming says Don't Trust ANYONE. That goes backwards AND forwards. In other words, don't make assumptions of inputs AND don't release anything you don't want seen since anything you release COULD be seen. So like I said, Chrome should wipe any Incognito pages before releasing their framebuffers on the assumption that they don't want the contents to be visible to anything else.
There's also the matter of the KISS principle. Assume the least work was done on your request, and do yourself the least amount to accomplish your goal since you may be subject to delays or repetition that result in small delays adding up. Why should Diablo blank their framebuffer if they're just going to immediately overwrite it anyway?
"
Why should Diablo blank their framebuffer if they're just going to immediately overwrite it anyway?
"
Immediately? The contents remained without overwrite long enough for the user to recognise the image it contained. Which is one heck of a long time in CPU cycles - consider that the entire buffer can be completely overwritten within 1/30th second at the very least (otherwise it would be incapable of displaying standard NTSC videos).
As reported it seems like the video replayed. If it's Flash, won't fix deprecated things seems to apply, but if it's incognito mode and HTML5, seems pretty significant to issue, commit and execute the GPU cache and data (otherwise frustrated by OpenCL, which I thought we liked.)
Contrawise, if there were a link that broke out bugzilla or git so:
-Reproduced with 4K HSVC 3D Gay pr0n; 2 hour video plays during aggravated Just Cause 5 Load (parameters at link, etc.) Came twice anyhow like always with loading screens.
-Reproduced with Hatsune Miku while falling at 7000' in an updraft 3:40.2. Came twice. Peckish.
-Reproduced with locals. Came four times. 5:13:11 according to FitBit. More worried than anything I can't remember seeing them.
-Cannot reproduce; it's video not actual fecundity. [Don't want to know.]
"Why should it?"
Security. An application could malloc() a big chunk of memory and there's a good chance of extracting cryptographic secrets or other things it should not have from memory recently free()ed from another process, or from a process that terminated.
It's a compile-time option for the linux kernel. The default is to zero, but you can disable that to slightly improve performance in very resource-tight embedded environments.
The OS most certainly does. Otherwise you'd be in for a world of hurt in any multi-user environment.
malloc generally doesn't however, so you can expect to see old data from the _current_ process. However, if the kernel ever hands you a page (eg via sbrk/mmap on *ix) without clearing it from the previous owner , it's a (potentially severe) security issue, not normal or intended behavior.
For some details on this in Windows, see http://blogs.msdn.com/b/tims/archive/2010/10/29/pdc10-mysteries-of-windows-memory-management-revealed-part-two.aspx and mm\zeropage.c in the Windows kernel source.
On *ix it's frequently implemented somewhat differently, with zero-fill-on-demand (i.e. pages aren't actually zeroed out until they are accessed under new ownership for the first time).
This should be a C2 security failure, reusable resources should be cleaned before reuse.
Imaging if the GPU was doing the whole video session, it could have the credentials including passwords etc in it's memory, these could then be picked out by another app on the same box.
This is a major security snafu.
"IMO in this case the bug lies squarely with Chrome: if you go out of private mode, you erase ALL of the stuff you did until then."
Chrome probably did. If the OS (driver land) lies to you about having wiped and discarded frame buffers, there's not much an app can do. Drivers should never give a buffer to a new app without wiping it first. Apps should not have to write code to work around the possibility of the OS not doing its job. I put the blame entirely on the GPU driver.
Why is it a bug in Diablo? They initialize the memory with their first frame of rendering. What happened to the framebuffer before them is, frankly, none of their business. It should fall on Google to ensure that when a Incognito page is closed, it's blanked BEFORE it's released. In security terms, this is a memory leak on THEIR part.
There is blame on all sides here, I think.
Chrome holds the majority of the blame: They are specifically stating that incognito mode keeps your session private. It should wipe anything which could be used to discover information about the session as soon as it is no longer needed. This would include the frame buffer.
Diablo holds some blame, as it should not be displaying the frame buffer until it has initialised it. It is not their fault that the data was leaked, but it is their fault that it was visible through their application.
NVidia should (although they may, I don't know) offer easy-to-use privacy hooks and APIs to ensure data is not leaked, triggering a zeroing of the frame buffer on release.
The OS (and applications) should use those hooks and APIs to ensure data is not leaked (e.g. on logoff or user switching).
"This is more of a bug in Diablo - not initializing memory correctly before enabling display."
It isn't. On any multi-user, multi-tasking system, the OS* should always zero or otherwise scramble memory before allocating to another app. This is otherwise a well-trodden attack vector. The OS should not trust the apps running on it, and apps should not trust each other. On a well designed OS, the apps should not even be aware that other apps are running and each app should be able to consider its own memory space private and secure. We're not quite there yet, but it's a good aspiration ;-)
(* By OS, I actually mean "OS downwards", depending on what is appropriate - the OS itself, or the video driver, or the specific card driver, or the card itself)
"Zeroing a large framebuffer could take some time"
A video card is designed so that the framebuffer can be filled with data quickly enough to satisfy a decent framerate - hopefully well in excess of 25 frames per second. So zeroing the framebuffer should take a lot less than 1/25th of a second, which is a tiny price to pay on app startup.
"On a well designed OS, the apps should not even be aware that other apps are running and each app should be able to consider its own memory space private and secure. We're not quite there yet, but it's a good aspiration ;-)"
Can't. There are times when an app NEEDS to know another app or module is running. Example, what good is a web browser without an Internet connection, which means knowing the socket driver is available, which may or may not be in Userland (depends on the OS, but microkernels by design would put everything non-essential into Userland). And there are such things as "ethical" process snoopers like anti-malware and anti-cheat programs.
"There are times when an app NEEDS to know another app or module is running. Example, what good is a web browser without an Internet connection [...snip...]"
Indeed, but this is handled in an OS by IPC, not by asking the OS to hand you blocks of another process's private memory space to sniff. Let's not get off-topic here ;-)
When designing an OS, one starts by creating an environment whereby each process thinks it is a single task running exclusively on the hardware, and then solve the issue you speak of by providing communication APIs. If you do it in reverse, you end up with very weak sandboxing and highly dangerous inter-process privacy issues. This is why memory allocation by the OS - whether main memory or graphics memory - is meant to provide "clean" blocks. Imagine if it didn't, and you could trigger the OS to allocate to you a memory block which contains SSH private keys used by another process?
"Example, what good is a web browser without an Internet connection"
Most browsers are just as good at rendering locally sourced content as they are content from the web, so still useful in some circumstances. If the OS is well designed, wouldn't the browser just pass it the URL and the OS sort out where to get it from, and whether it could?
Probably mobile apps need to be more aware of other stuff going on as they will need to hibernate gracefully when, for example, a call comes in.
"Probably mobile apps need to be more aware of other stuff going on as they will need to hibernate gracefully when, for example, a call comes in."
Yep - but they don't do this by being aware of or monitoring or sharing memory of other processes running on the device. They do it by registering a signal handler which is called by the OS when a call comes in.
and the consuming applications should clear up after themselves and/or initialise their allocated buffers.
I cant see any reason why the NVIDIA drivers should go round randomly clearing frame buffers just in case.
I would propose this issue is not limited to google solutions, indeed the video framebuffer may well be from another player applicaion. Chrome can't go round flushing buffers of other services either.
Risk, yes,
fixable by one vendor, no
...
No, on any sane OS the memory is cleared (typically zeroed) between one app releasing it and another app allocating it.
However, malloc() is not an OS function, it's a C library function, and it will reuse free()'d blocks of memory from the same application without clearing them, and only get more memory from the OS if there are no free()'d blocks available. So if you are using malloc() you have to assume that the block is uninitialized.
There's a big difference between "app X can see data from the same app" and "app X can see data from other apps, that may be run by other users or contain more sensitive information". In the "same app" case we need the performance boost from not zeroing the memory, in the "different app" case we need the security benefits of zeroing.
The OS & drivers SHOULD have applied the same principles to graphics memory, but they obviously didn't. This is clearly an OS or drivers bug.
"I cant see any reason why the NVIDIA drivers should go round randomly clearing frame buffers just in case."
Because if they don't, someone can write a low privilege application that just goes around allocating frame buffers and saving the "uninitialised" contents as a bitmap and posting the bitmaps off to the NSA. For the OS (or driver) not to wipe memory before passing it to a new process is a COLLOSAL security failure that has been LAUGHABLE since the 1960s.
All modern operating systems handle this problem for RAM by maintaining lists of dirty pages and having a "zero fill" task that wipes them clean before transferring them to the free list. It happens whenever there is free time (waiting for I/O perhaps) or, as a last resort, at the point of allocation. Since the last resort is hardly ever reached, the cost to the interactive end-user is effectively zero.
A GPU driver could use a similar scheme and (as already mentioned) certainly has the bandwidth to make it affordable.
"A GPU driver could use a similar scheme and (as already mentioned) certainly has the bandwidth to make it affordable."
But not the TIME. GPUs are normally built for high performance, so there are frequently zero-time context switches (a freed buffer has to immediately go to another application, with no chance to wait because, like I said, performance is demanded). Now you're in a security-vs-speed dilemma, and people why buy performance GPUs will demand the latter.
This has been a known exploit since the earliest days of personal computing. It was quite common to quickly reboot a machine and discover troves of information left by the last program running (I used to do this quite a bit in the latter days of using my Commodore 128). I recall very few programs have the know-how to interrupt the warm boot sequence to erase their code to block this (I think Lenslok-protected games actually cared).
Yeah, it's also for buying anniversary presents without your other half seeing what you bought them in the browser history.
No worries here. My other half doesn't know what browser history is....and besides, we have separate computers. For.....you know.....reasons.
(No, not porn. It's because when we briefly shared a computer - long, long ago - we were constantly fighting over it. Plus I prefer Linux and she prefers Windows so the thing ended up getting rebooted every time one of us sat down. It was easier just to get separate computers.)
as the old paradigm of disk and volatile memory is blurred with SSDs and more and more NVRAM. In fact I would go as far as to suspect that the current crop of "coders" (i.e people who use DreamWeaver for 5 minutes and call themselves "web programmers") haven't the faintest clue about the differences.
The future looks interesting ....
It's definitely diablos bug. They are displaying random video memory to the user. The fact that it managed to get some image bytes from a private chrome tab is kinda irrelevant. They could display literally anything because they are incorrectly initializing their rendering process.
If you wrote a program to scan every accessible byte of RAM i would expect you'd find all the strings associated with your smut pages too. You'd also find strings associated with every other application you ran in the recent past (assuming nothing overwrote that memory).
Most modern operating systems do not guarantee memory is zeroed before the OS hands it to you. That's why you have calloc and malloc. It's a pointless performance penalty in the vast majority of cases.
> Most modern operating systems do not guarantee memory is zeroed before the OS hands it to you.
The O/S certainly should clear memory that has been owned by a different process. Otherwise, as has been said above, there are at least privacy issues. It absolutely has to clear memory previously owned by a process with a different UID.
IIRC there was one such (leaking other's memory) bug with memory mapped via mmap().
For a program leaving contents in the framebuffer is IMO bad style as well.
"The O/S certainly should clear memory that has been owned by a different process. Otherwise, as has been said above, there are at least privacy issues. It absolutely has to clear memory previously owned by a process with a different UID."
But what if the program in question is a recovery tool that NEEDS to see that memory? One size can't fit all here, and the principle of DTA dictates that ultimate responsibility falls to the program that made the data (the origin point, if you will, the point of first responsibility). If you don't trust another program to see their data, it should be wiped before you release it. And before you say the OS should do this (maybe not wipe on the alloc but on the free instead), remember that bulk memory operations mean an unavoidable performance hit, and if the OS is designed for high performance, such a hit may not be desired.
> But what if the program in question is a recovery tool that NEEDS to see that memory?
If any program (let's restrict that to non-root UID) can see another's memory then privacy and security is gone.
It's a design decision. Except possibly for systems where the is essentially no user, and only for VERY compelling reason I'd accept leaked memory contents.
By he way, if a program unexpectedly terminates (say, by SIGKILL) it cannot by any means do any cleanup.
"If any program (let's restrict that to non-root UID) can see another's memory then privacy and security is gone."
Then we're essentially doomed. Anti-malware, anti-cheat, basically any defensive program worth its salt MUST be able to see into other processes to make sure they're not malicious, and if THEY have to do this to be able to function, any other program can pretend to be this, too. We've gone into a Quis custodiet ipsos custodes? situation, and there's no easy answers to that.
The kernel can always peek 8^)
One can also check an executable before it becomes a process (pretty sure scanners do this) and by watching the _actions_ of a process: You want to open SMTP? Sorry Dave! (again, pretty sure those "protection" programs are doing this). Whatever else they do to completely drain your resources, I don't know.
In other news a certain system where security was an afterthought has been found "fucking hopeless anyway".
"One can also check an executable before it becomes a process (pretty sure scanners do this) and by watching the _actions_ of a process: You want to open SMTP?"
But what if the malware waits until it becomes a process AND disguises its malware act as a legitimate act (Yes, I have to open SMTP--I'm an e-mail client!)? Then you need more sophisticated sniffing that can also work outside an encryption envelope, meaning it has to be able to see the process while running.
Openfb had this problem only 12 and 15 years ago, so it's understandable that OpenCL, which doesn't allocate specific memory (only something free on the scoreboard) and which can have quite a long callback in being asked to clear the buffer used would have a hitch or so coughing up a callback for a clear displaylist. Or, your OS might use Persistent Mapped Buffers (OpenGL) and not be compiled to worry horribly about data load determinacy (v. callbacks w. new pointer optional) and reusing them. Quite normal if you're not running signboards in a government building! (Flying dockets! Tripletake defense-state!)
Either you want a fuzzer or active padding daemon competing (maybe with some determinism) with your power scheme to prevent leaks on displays of the same machine...reproduce it with lvm and you've earned an educational use case to ponder...or you really want to save power and noise.
If I play Rocket League, the last photo I've viewed flashes up on the screen immediately before the Rocket League splash screen. Can't swear to it, but think I'm using the Picassa photo viewer.
Very strange the first time it happened. Only comes up for less than a second and thought I was seeing things.
This bug is common in graphics drivers... not familiar enough with the inner workings of OpenGL to know but I'd assume the driver could zero framebuffer and texture memory when deallocated/no longer used. Maybe this is hard or non-performant.
Either way it needs fixing as this represents a serious security hole for things like web browsers' "incognito" mode or in fact anyone working on sensitive information displayed by GPU. Which, with many recent window system implementations using texture memory for window pixmap store, could mean every application you use.
Sure, main memory suffers the same issue, but secure operating systems already zero memory on allocation, and highly sensitive applications will mlock memory and zero it when finished with cryptographic keys etc.
"This bug is common in graphics drivers... not familiar enough with the inner workings of OpenGL to know but I'd assume the driver could zero framebuffer and texture memory when deallocated/no longer used. Maybe this is hard or non-performant."
The problem is that memory wipes take time, and GPUs are typically built for high performance, meaning it's a trade-off. Speed frequently clashes with security, unfortunately. And in a paranoid system, one should assume their mess won't be cleaned up for them.
PS. Why should the memory be wiped on the alloc? Shouldn't it be wiped on the free instead?
Then it should be an option on the free() call, unless it's a free called by the program's termination (in which case it can an automatic wipe; performance becomes less of an issue in the graceful termination phase). That way, the program can judge if the memory needs to be wiped (for example, because sensitive memory is involved--they'll want to clean it regardless and doing it this way minimizes the chances of a read by elevated code). As for abnormal termination (essentially "nuking" an app), then perhaps only then should the OS intervene and wipe the program's memory space as it's performing an intervention. Any other method should leave it the program's responsibility.
I remember demonstrating to a government research place in the 1980s that on their VAX computer one could grab a few megabytes of disk space and dump out the contents. A lucky dip, really. I could see it was interesting stuff; but the government people were appalled.
It still is a problem with disks, of course. All the arguments above about whose fault it is apply equally. But little has been done. Yes, there is SATA secure erase if you really want to clean up a disk. But countless second hand machines are full of titbits, in every sense of the word.
Equally, it seems nothing will be done about graphics memory.
Linux laptop -
Currently on nvidia drivers, but if I were to stop and restart xdm, when I log in again, I can see in the map space between my two screens (one is 1600x900 the other 1920x1080) - the bottom edge of the larger screen has tiles and textures from the previous state of the X display. This is far from new. The GPU memory has never been cleared on either set of drivers. And I'm not going to comment on the issues with the Intel drivers - even on single screen display setup they leak textures and image data from *the current* session....
I'm not sure why *anyone* has ever had the idea that data on the GPU memory was cleared out before reuse. I recall from a *very* long time ago that the routine to wipe the page allocation table was the only thing done on a recycle in the rageII drivers to clean out memory on the card, it was considered sufficient since the memory would be reused *and reloaded* with data before re-reading it. I somehow doubt that that basic concept has been changed anywhere.
<shudder. 30 years. I've been doing this crap for 30 years.>
But I thought incognito mode/private browsing was so you could order your wife presents online without her knowing about it?
Who here actually believes that any of these incognito type settings from ANY of the browsers actually hides anything from anyone apart from the wife?
Goolge even give themselves a get out clause when you start Chrome up
"Going incognito doesn’t hide your browsing from your employer"
They promise nothing
"Going incognito doesn’t hide your browsing from your employer"
That copout is due to hypervisor capabilities in enterprise settings. Basically, Incognito can do squat against an agent that can snoop at all programs actively running. Basically, that scenario is like getting caught with a salacious book wide open. You can't do much against that kind of eye.
Nevertheless, Chrome should be obligated to perform due diligence when handling incognito windows. It should, as standard security procedure, retain the information for no more than is absolutely necessary to function, meaning any information it no longer needs should be immediately wiped clean to minimize administrative/hypervisor/root-class malware spying.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
