Boffins say they can turn typing sounds into text with 95% accuracy

Researchers in the UK claim to have translated the sound of laptop keystrokes into their corresponding letters with 95 percent accuracy in some cases. That 95 percent figure was achieved with nothing but a nearby iPhone. Remote methods are just as dangerous: over Zoom, the accuracy of recorded keystrokes only dropped to 93 …

[OhForF']

Practical attack?

What is the accuracy if they get good audio quality but can't train their model on the specific user and keyboard and location beforehand?

If that although falls down to 40% or lower like it does for "skilled users relying on touch typing" the attack is not all that practical.

When you can install a keylogger or similar to train your model there are easier ways to get the password ;)

[Anonymous Coward]

Yes, very practical.

Character N-grams have known a priori distributions, among other clues. For example, a listener could probably manually pick out the periods and space characters of a passage of natural language, even without a computer.

An automated system could do even better.

You have one parameterized system working to categorize the clicks, with some pretty good initial guess using, e.g. PCA.

You also also inferring a categorized-click-to-character map which minimizes the difference between observed click-N-grams frequencies and a priori character-N-gram frequencies. You could approach the problem with simulated annealing, with the click-to-character map (and the click-categorizer parameters) as the solution space and let the evaluation function be a function of the N-gram distribution differences.

Note the method is very tolerant of noise - e.g., even if some clicks are not being completely distinguished from each other you'll have some idea about which characters-clicks are close to each other. That will help in making password guesses. A lack of known click-to-character key training data is not going to be a limiting factor.

In my judgement, this is a real security issue, mostly because cell phones are easily hacked. A noisy keyboard and a single passage of say one thousand words worth of natural language typing noises would be enough to start making educated guesses about passwords.

Also, testing random key pairs on my own keyboard right now pretty much every pair can be be distinguished.

[that one in the corner]

Re: Yes, very practical.

No, no, no, that won't do at all.

You aren't playing the Modern Game here, with all this discussion of basic knowledge of character use in language and (probably) old-fashioned signal processing to separate the individual clicks. You can't go around suggesting that this is something that could have been done for decades.

> without relying on a language model. Instead, they used deep learning and self-attention transformer layers to capture the sounds of typing and translate it into data

> Those were then analyzed by a deep learning model, which fed them into convolution and attention networks to guess which particular key, or sequence of keys, was pressed.

There, you see: the Proper Modern Way is to just feed raw data into a "deep learning" system and various convoluted mechanisms and just Trust The Magic Machine to work it all out for you.

THAT is how you get grant money!

[Sorry that handle is already taken.]

Re: Yes, very practical.

Hah!

But I think less cynically, it's the ML model that discriminates the sound signatures for each different type of keystroke in a non-trivial and non-obvious way

[jmch]

Re: Yes, very practical.

" the ML model that discriminates the sound signatures for each different type of keystroke in a non-trivial and non-obvious way"

If the model is just using sound signatures on their own, there needs to be a separate model for each type of keyboard. As the OP suggested, keyspaces can easily be used to identify individual words and/or phrases, and once each individual key sound can be individually identified it is relatively easy to identify each key using statistical analysis of how common each letter / word is, and whether they make sense in combination.

Maybe the only way to mitigate the attack is to have a keyboard design that is both as noiseless as possible, and designed in a way that all the keys sound as much alike as each other.

With regard to 40% recognition, that is probably enough to hack passwords. If the system can identify "Ctrl-Alt-Del" and "Enter", it knows anything in between is a password. If it can correctly identify 40% of the characters AND narrow the other characters to a few possible keys, it becomes amenable to brute-force attacks

[wallyhall]

Re: Yes, very practical.

For some of the folk I work around, they'd get a lot of W,A,S and D examples in the dataset! (Even from working hours ... :S)

[that one in the corner]

Re: Yes, very practical.

> a non-trivial and non-obvious way

I'll certainly grant the non-obvious part, as it is almost axiomatic that neural nets tend to spot oddities in the data - and then are incapable of explaining what they're actually doing, so even if that would fall under "obvious once you know the trick" you will never be allowed to peep behind the curtain.

[Sorry that handle is already taken.]

Re: Practical attack?

I didn't get the impression that pre-training was required; almost everything can (probably) be inferred through statistical analysis and knowledge of the language being attacked. After all, when typing something you expect to be understood by others, keystrokes aren't random since language is full of patterns - certain letters, word fragments, words, combinations of words and phrases all have a characteristic frequency of usage.

For example, in English (and American), if a particular sequence of three keystrokes appears far more frequently than any other, it is probably THE, so now you can make a guess about other words that contain the keystrokes you think might be associated with T, H and E, and so on. You can also build a map of the frequency of each keystroke sound and compare it to the frequency that each letter appears in the language. A keystroke heard twice in succession shortens the list of letters it's likely to be. Many statistical analysis techniques can be combined to improve the overall accuracy of the "guess".

[that one in the corner]

Re: Practical attack?

The paper explicitly states that they are not using this sort of n-gram approach - there is a section discussing the fact, look for the phrase "Hidden Markov Models".

Of course, there is the chance that the model itself is learning to look for this sort of pattern, somewhere within its black box of nadans.

However, as they (and the Reg's article) point out, there is concern about using this attack against passwords, which - hopefully - aren't subject to that form of analysis in the first place (even the longer pass-phrases used in some setups are short enough to confound basic "etaoin shrldu" attacks).

[Little Mouse]

Re: Practical attack?

"What is the accuracy if they get good audio quality but can't train their model on the specific user and keyboard and location beforehand?"

But that's exactly what's being described in the article. There is no prior-training required. For each deployment, the model just needs to listen long enough to figure it all out for itself.

Edit: "Sorry that handle..." pipped me to the post, and spelled it out (heh) far better than me.

[yogidude]

Mitigation

Play a soundtrack of someone typing in the background. P-I-S-S-O-F-F

[MatthewSt]

2016 called...

... https://www.pindrop.com/blog/recording-keystroke-sounds-over-skype-to-steal-user-data

Slightly suspicious that the Skype percentage hasn't changed!

[FeRDNYC]

Re: 2016 called...

I suspect that's because the Skype figure comes directly from that exact 2016 paper, which is cited in this new one.

[The commentard formerly known as Mister_C]

Prior art...

Peter Wright described this method in his book "Spycatcher" back in the 80s. Put a fault on an embassy's phone line and they'll be glad that 'GPO' engineers come to fix it pronto, not realising that they've invited spooks in to place a bug on the phone nearest to a typewriter of interest (usually a cipher machine). Of course, that was analogue phones listening in on mechanical typewriters but the concept remains the same

[Anonymous Coward]

Re: Prior art...

Ah, the good old telephone Infinity Bug. You don't hear much about things like that these days. Still available today (allegedly, hem hem, that circuit in the first page of web hits - just ignore it) but get in quick before they turn off the proper landlines.

[NeilPost]

Re: Prior art...

I can conceptualise a manual typewriter could make distinguishable noise from the mechanical action and perhaps with a lot of learning and audio sampling an Apple Scissor keyboard - the article is *very specific* on a 2021 16” MacBook Pro.

… but not an unknown PC Keyboard with a dull/dead chick let mechanism.

Needs very controlled environment and hardware.

Sounds more 1Apr story.

[Natalie Gritpants Jr]

Just drink enough coffee that you get the shakes

[that one in the corner]

Given that coffee-powered key bashing is SOP for many, and the advice given is to "just change the style of your typing", maybe we'd be better off with a few minutes of tai-chi and zen before typing in the password for the morning log-on.

Or on a morning after the night before, log in and then down the alka-seltzer: the hiss will also help mask the clicks.

[Anonymous Coward]

Shift or not is only one bit of information per character

- a thin thread on which to hang if you have a long way to fall. Using special characters would be a little better, like putting an old mattress on the spot where you were about to fall.

It would be harder but most definitely possible to infer character and common punctuation keys without "complete" key+noise training data, using the a priori frequency of adjacent character, esp the space character, and prepositions, etc., to bootstrap. Those bootstrapping frequencies themselves could be implicitly picked up by an LLM-analogue text based learning system, no need to explicitly store them.

Anybody working on open source, whose clicks are also being recorded, could be in trouble. That days commit is a rich source of special characters. Open source Crypto-Bros beware.

One possible and easy mitigating strategy would be to use a usb keyboard splitter and a different keyboard just for entering passwords, although that doesn't protect private text other than passwords.

Your secret keyboard clicks and clacks may already have been recorded and uploaded and be awaiting their turn in the queue for processing. If your only worry was that those included passwords, change them and introduce mitigating strategies. However, if there were life threatening secrets told, you are out of luck.

[FeRDNYC]

Re: Shift or not is only one bit of information per character

Even better, just use key-exchange authentication instead of passwords. Having the passphrase that unlocks your private key is useless, unless you also have the key itself.

[Wobblin' Pete]

I want one....

I know this article was supposed to be highlighting the security risks, but am I the only one who read it thinking why have i been dragging round all these cables, spare batteries, chargers and so forth for the various 'portable' keyboards all these years, when I just needed to bash the keys that little bit louder?

just need an old xylophone to use as a mouse mat now....

Mines the one with pockets full of old duracells

[Anonymous Coward]

It's not a security risk, it's a feature!

Patent coming.

[that one in the corner]

Welcome back to the Typing Pool

> playing fake keystroke sounds to mask the real ones

Why bother with fake when we can just bring back a faithful institution?

But how will you get your text to the Pool without being intercepted? If you dictate then that can obviously be recorded just as easily! The answer is obvious: just go back to that other old institution, perching the secretary[1] on one's knee and whispering into the shell-like.

Side benefits include getting "important" people to give up their PCs for the prestige of being seen to be dealing with Secure Data: the savings in IT costs from removing those over-specced and under-used machines from service, along with all the strange problems they seem to have (see many an On-Call), will pay for the Pool.

[1] note: no sexism here; secretaries can be any gender these days, as can the owner of the knee. We're being selective about which old institutions are revivable in the name of secure data.

[jmch]

Re: Welcome back to the Typing Pool

"...playing fake keystroke sounds to mask the real ones..."

...or software-defined keyboard that displays the letters on different keys from time to time. Of course forget about touch-typing then!!!!

[Paul Hovnanian]

Membrane keyboards

What click?

[Anonymous Coward]

Re: Membrane keyboards

The model then learns to detect the various sounds of your finger nails smooshing into the keyboard, your muttering that this key really feels like a dead fish...

[FeRDNYC]

Never thought my declining accuracy would work in my favor...

But as long as I'm careful to edit multiple keystrokes by holding down the Backspace key instead of pressing it multiple times, the system is probably pretty screwed for deciphering what the final state of my input actually ends up looking like. (Even better if I use the mouse to navigate around the input field while editing, which I do tend to do.)

[Arezzo]

A wet boffin never flies at night.

[Mishak]

Adding keyboard sounds to Zoom audio

I'm hoping that only happens when the user is actually typing and isn't on mute!

I really hate it when people are hacking away in a meeting and forget to mute...

[Jim Whitaker]

Re: Adding keyboard sounds to Zoom audio

Just as irritating when they are using a pencil.

[jamesdagger]

tH3y'lL n£V=r C47ch mE!

[Jim Whitaker]

Old news

Back in another life in about 1973, the IBM golf ball typewriter (aka Selectric) was known to be a target (I could not say whether a successful target or not) of this sort of attack. Ditto teleprinters.

[John Brown (no body)]

Re: Old news

<clackity clack>

Might be time to ditch the IBM Model M keyboards too :-)

</don't look back>

[Will Godfrey]

Hmmm

if 9a^H^H(a .=^H^H>=*)^H^H80

need I go on?

[Binraider]

I can hear where the boffins are coming from on this one.

Typing cd\games\doom2 has a particular cadence to it that you can hear. A combination of assuming the typists choice of language e.g. is it a document, excel, or command line could go a long way towards figuring out the sound in question.

If you know the make/model of keyboard then you have quite a lot to go on in your detective role.

Mark one ears in sat in front of this mushy, awful laptop can still make out distinct sounds of typing different keys. The most obvious are the space bar and return, but certainly distinct for others.

Clever stuff. Applications are obvious.

[Eclectic Man]

Bach, Beethoven or Mozart?

Well, instead of playing annoying background sound of typing, how about some J S Bach, L van Beethoven or W A Mozart? Their piano music is superb and complicated and

Oh, just realised that I was listening to the music instead of typing.

As you were.

[that one in the corner]

Re: Bach, Beethoven or Mozart?

Spotify reports an unexpected upsurge of interest in Leroy Anderson's music, one piece in particular.

[FeRDNYC]

Re: Bach, Beethoven or Mozart?

I think my favorite part of that is watching the typewriter slide itself across that tiny desk while he's "playing" it, to the point where he has to reposition the thing more than once during the piece. That just seems like one of those things that absolutely could've been worked out in advance, and compensated for... and it boggles the mind how it was not.

[Anonymous Coward]

But we already have some protection in place anyway ...

Certainly a good password hygiene policy that requires a regular change helps.

As does 2FA.

In fact if you are "doing passwords" correctly, this isn't really much of a threat.

Secret keypairs for stuff like AWS are a different matter. But then who types them ?

[amanfromMars 1]

Beware, Take Care, IT is a Crazy FCUKing Jungle out there .... Devoid of Vanquished Prisoners

And, no matter how much data/metadata they may or may not be able to sniff/steal relatively effortlessly, will they .... the powers that be thinking they are in leading command and control ..... always be reacting to that which they have no prior knowledge of know and which renders them at a distinct, and wholly worthy, catastrophic disadvantage to renegade and rogue freelancing entrepreneurial forces and sources.

Advanced IntelAIgently Designed Entities which they, the former powers that were thought to be, will have to accommodate and make absolutely fabulous, fantastic deals with, in order that they are not summarily dismissed and completely removed from Future Greater IntelAIgent Games Plays by such alien means and/or memes.

And now that you know that, quite what your reactions might will be, will certainly clearly tell that which you have every good reason to quite rightly fear, all that it is necessary for them to know to successfully move on further with myriad other plans, totally untroubled and unhindered without such as would be classified and recognised as dead wood input/diseased output.

Such a fate/destiny/future has one damned and cursed if you do do something, and damned and cursed if you do nothing, and sure to one and all suffering the slings and arrows of outrageously well earned misfortune via a resulting whole series of corrupt and perverse failed systems of administration stuck fast and deep and dark between a huge rock and a hard place in an almighty virtual space ..... teeming with mortal enemies and SMARTR AIgents.

Who you gonna call then whenever prized western goods are on a slow boat east to China and Sino-Soviet/BRICS forces and sources ?

[FeRDNYC]

Re: Beware, Take Care, IT is a Crazy FCUKing Jungle out there .... Devoid of Vanquished Prisoners

Sir, this is a Wendy's....

[keithpeter]

masking...

...try this LOUD

https://freesound.org/people/keithpeter/sounds/123344/

[Ken Moorhouse]

Brings a whole new dimension to...

...Phone Tapping.

[Luiz Abdala]

TEMPEST

This has been the subject of TEMPEST for over 20 years.

Straight from that wiki: "TEMPEST is a U.S. National Security Agency specification and a NATO certification referring to spying on information systems through leaking emanations, including unintentional radio or electrical signals, sounds, and vibrations".

Yeah, a leaky VGA cable or the sound of a keyboard, all the same. Altough reading the electromagnetic emanations of a VGA signal is a much more esoteric way of snooping around, IMHO.

Keyboard icon for reasons.

[Ken Moorhouse]

This reminds me of the time some customer of mine had a problem with their password

The original password was typed in on a 'non-standard' keyboard. Then, later, trying to login using a standard keyboard. 'Non-standard' in this case being maybe a laptop user with Num Lock inadvertantly set to 'on', or a foreign keyboard, or more simply, Caps Lock set to 'on'.

Those were the old days.

Designers of login screens seem to have woken up to this flaw in their designs more recently.

[FeRDNYC]

Re: This reminds me of the time some customer of mine had a problem with their password

Designers of login screens seem to have woken up to this flaw in their designs more recently.

I mean... to an extent. Yeah, any good login interface will now warn you if CAPS LOCK is turned on while you're typing your password... but that's about it. If anything, I think the infosec world is waking up to the fact that the flaw in login systems is the entire concept of successfully typing out a certain string of characters as a means of verifying a user's identity, not so much the fiddly details of exactly how they type in those characters. Better they just... not do that. Like, at all.

Passwords are a poor means of establishing credentials, full stop. Fortunately, advances in biometrics, key-exchange authentication, federated logins, and other "post-password" technologies are gradually making the humble string-of-characters textual password obsolete. Thanks to built-in browser/OS password-management features, the practice of manually typing in said password (just like grandpa used to do!) is even nearer to extinction. Can't happen soon enough.

[Nursing A Semi]

So now

You can find yourself bombarded with adverts for cars after typing that its time for a change within range of an Amazon device? Just need to figure out how to get those adverts to display on my A500 with a HTML only browser.