How Amazon red-teamed Alexa+ to keep your kids from ordering 50 pizzas

If Amazon's Alexa+ works as intended, it could show how an AI assistant helps with everyday tasks like making dinner reservations or arranging an oven repair. Or things could go terribly wrong: it might turn on the oven and turn dinner plans into a house fire. This is why the e-commerce giant brought in security engineers, …

[MrMerrymaker]

hate to praise Amazon

So I won't but

Go, red team!

Apply logic.

AI Is so over hyped and doesn't really exist as I'm sure this place knows

So a company actually looking to make a home use of it more than just hype = I ain't bowing to any overlords just yet but I won't insult Amazon over this

Not when there's so many other things.

But yeah go red team.

[DMcDonnell]

A 100% way

A 100% way to keep the kids from ordering pizza; We don't have an Alexa. Never shall.

[ecofeco]

Re: A 100% way

Elegant solutions are always elegant.

[MachDiamond]

Re: A 100% way

"A 100% way to keep the kids from ordering pizza; We don't have an Alexa. Never shall."

They could order the pizzas, but the shop would be a bit wary to get an order for that many out of the blue and would ask for a credit card number they would charge before making the order. With Alexa, the card is on file so little Johnny doesn't need it in hand.

I'm against convenience at every turn. The reason it's a big priority for companies is to make it easier to part me from my money. Having to think 3 seconds and do a tiny bit of work is often plenty for me to say, "naw, don't need it" and also recall I still have lasagna in the fridge that needs eating up.

[CowHorseFrog]

Re: A 100% way

If they have hardcode these limits doesnt that prove the point that AI is dumb and cant be trusted ?

[doublelayer]

Re: A 100% way

Not exactly. I mean AI is dumb and can't be trusted, but this doesn't prove it. What this proves is that the interface is in a situation, to use the security term a threat landscape, where AI is unable to patch the gap. When you're worried about a child sending a command, then filtering out massive orders is important, but so is any other weird command a child might give. A child ordering fifty pizzas can be eliminated by a simple rule, but what about a child ordering a single pizza because they don't want to eat whatever was planned and they are able to correctly understand their pizza consumption potential? That's also a problem, and as long as the system will accept an unauthenticated purchase request, it is possible.

You could try to fix this with yet more AI: do voice age detection to try to eliminate certain requests from childlike voices. In a world with functioning AI that identifies users, that might be how you'd do it. Doing it here will likely lead to even more annoyances, such as the age guessing function to get it wrong and refuse an adult, or if the child is smart, the child using a voice generator to create an adult voice to make the request. Most likely, the method to solve this one is the boring old standby, for any purchase request, get a confirmation from someone with spending authority and provide that person with enough information that they know what's going to happen if they approve. Layers of AI can help to hide the gap, but unless those mechanisms exist, that gap is likely to still be there.

[LybsterRoy]

Re: A 100% way

Downvote because of any assumption that a tech problem can be solved by applying more tech.

[doublelayer]

Re: A 100% way

Out of curiosity, do you mean the voice age verification thing which I spent three sentences explaining how it would a) annoy everyone and b) not fix the problem? If it was, can I suggest a rereading until you understand the core point, which was specifically that technological patches will never fix the problem, only hide it from the initial glance? It seems strange that you're critiquing the comment by assuming I said the opposite of what I said.

[CowHorseFrog]

Re: A 100% way

I agree with what you are saying...

My point is AI obviously cant tell good from bad and thats the problem. It is guided by samples that it sees, it cant reason which is why new or slightly different situations that a human would identify as bad, might be randly marked as good by AI.

Its the tesla driving into a wall problem.

Ignoring the mistake of having no lidar sensors, the probelm is the engineers didnt "think" ahead and add reason that crashing into what looks solid becase of its colour and location is a bad thing. Again same problem, the engineers are trying to feed all combinations of situations that they think and they "miss" some nad bad things happen.

[Anonymous Coward]

When the 100% o2 atmosphere caught fire on Apollo 1, killing three astronauts, NASA termed it a failure of imagination.

No Red Team can conjure up what a three year old thinks. Just because it can be automated, does not mean it should be. (I'm looking at you Tesla.)

However, some people think that bugging their own home is a good idea, so it's swings and roundabouts really. (Not really, it's a horrifically bad idea, but just trying to placate the woefully ignorant.)

[Alan Brown]

"NASA termed it a failure of imagination."

Huh? Half the engineering team was PLEADING not to use high pressure oxygen. They already knew velcro burns furiously in such an environment when given a spark but were overridden by manglers

[Gene Cash]

Interesting

So I wonder how they bar ordering 50 pizzas but still let you order pizza... OR have a party. Yes, I've had parties where we ordered 30+ pizzas, so it's an actual possibility.

Right now, my credit union fraud detection shits itself when (for example) I change my Patreon subscriptions, or my salary changes because of tax/overtime/bonus/insurance/etc reasons. It's gotten really frequent and annoying.

[DS999]

Re: Interesting

It could say "hold on, if you want to order 50 pzzas I'm going to need more confirmation" and maybe it has to give you your credit card number used to pay for the Amazon subscription or something else your kid (hopefully) doesn't have handy.

Sort of like when fraud detection kicks in on your credit card and you need to do something to authorize the charge it has put on hold. I would think it would be better to have overly active fraud detection than not enough - ESPECIALLY if you are talking about your actual bank account (which I assume you are since you mention a credit union) where fraudulent transactions will create 10x more potential issues than on a credit card.

Personally I don't use a debit card for ANYTHING, and avoid direct payment out of my bank account as much as possible.

[MachDiamond]

Re: Interesting

"Sort of like when fraud detection kicks in on your credit card and you need to do something to authorize the charge it has put on hold. "

That can be good and bad. If you are traveling and get an alert, do you have to call a toll-free number that only works in your home country (or need to call since the online faff around isn't working)?

I helped a friend move from south to north in the US where he was driving his car and I was driving the van with the things he didn't want to go anywhere near the movers. He called the credit card company to let them know, filed the address change, etc so there would be no grief about charging a S-ton of petrol on the card. Wanna guess how well that worked? Fortunately, he had a couple of other cards to fall back on and a wodge of cash, "just in case".

I use my debit card for routine bills and cash for pretty much everything else. Especially cash when traveling. I'll fill up with petrol in-town before leaving with the card and pay cash for the rest of the trip. Hotels will be booked in advance and prepaid to get the best rate. I still have the card with me and will use it if I have to, but the goal is to not financially advertise the trip and to be insulated from fraud protection trip wires that could leave me stranded miles from home. It's happened before. The credit cards are used with planning when I use them and as emergency backup.

[DS999]

Re: Interesting

That's why you take more than one card with you when traveling. I always make sure to take both a Visa and a Mastercard, just in case one of them has a processing network outage. Then you also want your bank card so you can hit an ATM for cash - that's less important these days but used to be critical.

Back in '99 I was traveling in Spain and Morocco with a girl I'd met a month before in England, and my ATM card which was old and not as thick as current ones had developed a crack and stopped working and I had no access to cash (other than likely really bad deals on foreign exchange cash advances on my credit card) So she covered the cash stuff like cabs and nights out and I covered the stuff where credit cards worked like hotel rooms and trains. Worked out OK but if I'd been traveling alone I would have had to find out just how unfavorable those foreign cash advance fees and exchange rates would have been.

[MachDiamond]

Re: Interesting

"That's why you take more than one card with you when traveling. I always make sure to take both a Visa and a Mastercard, just in case one of them has a processing network outage. Then you also want your bank card so you can hit an ATM for cash - that's less important these days but used to be critical."

If you have multiple cards, yes, having more than one can be a good back up. Not everybody has multiple cards. Cash is a universal solvent. If power is out or a network is out, chances are good that you can still pay for things with cash. I bring cash with me as a network outage can mean any card is useless. I book and pay for hotels in advance to get the best deal as well as not having any issues while there. I'm not sure what they would do if I were checking out and they couldn't process the transaction. Checking in might be a different matter. The trips I like to take are off the beaten track and away from the US interstate highways as much as possible. That often means the choice of petrol stations or motels is narrowed down to one with it sometimes being that they are both the same business.

Two unmarried people traveling together also aren't connected financially so one person's bank shutting off funds through a trigger-happy anti-fraud algorithm still leaves some back up. I traveled with a girlfriend where she had a brand new card and it took a few days before the activation took hold and it didn't make it any easier that we were visiting friends in the UK having traveled from the US. She was treating me to the trip so a bit embarrassing that she had to ask me to put the car hire and some other stuff on my card initially. Fortunately, I had room to do that and hadn't let the balance ride to set aside money for the trip. All's well that ends well. We had a great time and got to see a bunch of friends in Manchester and Whitby. I also got to do some business in Bucks and see a customer I had never met in person before.

[doublelayer]

Re: Interesting

"I would think it would be better to have overly active fraud detection than not enough"

In most cases, you're right, but overly active fraud detection can have some significant problems too. I've had online payment systems go totally haywire when the payment wasn't even declined yet; my card simply wanted additional verification of the charge, then approved once it got it. In the meantime, the payment system gave up on me and canceled the order, so I had paid them without completing my order. I ordered a second time which worked, and the situation got cleaned up without too much chaos, but that could cause problems for some groups of people, for example if the transaction was large, they didn't have a lot of cash on hand, and the payments were on a statement before one was removed. It's probably easier to err on the side of caution when I haven't had to experience the effects of frequently incorrect caution. If it was happening to me a lot, I too might want them to dial it down a bit.

I'm reminded of a piece of antivirus software that decided that the output of a certain compiler was malware. I couldn't run any code I compiled because every time I ran it, the file would either be quarantined when it was created or deleted when I tried to execute it. I screwed with the compiler to fix the antivirus's inaccurate judgement, but it probably would have been easy to convince me to remove the antivirus software when I had been fighting with it for a while (work computer, so for better or worse, I couldn't remove the antivirus).

[MachDiamond]

Re: Interesting

"I couldn't run any code I compiled because every time I ran it, the file would either be quarantined when it was created or deleted when I tried to execute it. "

The OS couldn't find a security certificate to verify that the software came from an "approved" vendor.

I've managed to shut all of that stuff off, but I'm several revs behind on OS's (other than linux) so it makes me wonder if I will go one update too far and find I can't switch that stuff off any more without opening a developer account and surrendering even more of my personal information.

[doublelayer]

Re: Interesting

No, that wasn't it. It was a specific compiler whose included libraries were triggering something in the AV. Unsigned binaries coming out of GCC were just fine by it. Unfortunately, binaries from GCC wouldn't do what we needed; only that compiler was supported.

[NoneSuch]

Re: Interesting

My local bank deactivated my card for "suspicious activity." I'd tried to buy four movie tickets to see "Paddington", so understandable, right? I'd frequented that cinema for almost a decade and luckily I had cash on me. Had to call the bank the next day to get the lock lifted. Three weeks later, the same bank allowed a $2,500+ charge on my card for a SmartTV in a city three thousand miles away. No calls, emails, texts or warnings; until the wife saw the statement.

[LybsterRoy]

Re: Interesting

I feel for you. Just to prove you don't need technology to screw up. Many years (digital cameras were new so you can guess how long back) I bought my wife an Olympus digi can - it didn't have zoom so a while later I bought a screw on telephoto lens (c£100) for her birthday. It never arrived. I chased the company to be told the credit card hadn't been approved. Phoned the bank and was told yes they'd stopped payment because I wasn't in the habit of buying that type of thing or from that company. I wouldn't have mined if someone had told me. Either the supplier of the bank would have been good but neither.

[Jay 2]

Re: Interesting

Oh yes. I once had a fairly standard grocery delivery charge declined, even though something similar happened every few weeks. Meanwhile a one off several grand payment for a holiday I made sailed through...

[MachDiamond]

Re: Interesting

"Yes, I've had parties where we ordered 30+ pizzas, so it's an actual possibility."

If you do that with no planning.... I'm the sort that would call at least a day in advance to say I'm having a party and will be ordering a stack of pizzas or just order them at that time to be ready so I've got that out of the way. I'd hate to call and have them tell me there was another big order and are totally booked until 10pm or have run out of dough. I know lots of people aren't like that and will just wing it. I want to enjoy a party I put on so not having much to do means I get more enjoyment out of it. In my younger days it also meant time to chat up the ladies rather than having to deal with food/drink.

[doublelayer]

Re: Interesting

Doing that changes how long you have to retry the situation, not whether it works in the first place. If the system rejected the order out of hand, it wouldn't work a day in advance either. I assume there is a mechanism for verifying that a user is permitted to conduct the action for the ones that have been filtered, so I doubt it's as problematic as the financial fraud detection examples it's being compared to.

[MachDiamond]

Re: Interesting

"If the system rejected the order out of hand, it wouldn't work a day in advance either."

The vendor might reject the order too is what I was getting at. 50 pizzas ordered with no notice online sounds too much like a frat boy prank. If the payment service bucks, that's good if it's fraud, but a PIA if it's a real order. If you need to order the pizza for an event and are being blocked at every turn, that's a problem if you've promised there will be pizza. Do you have the time to travel to the store (if they are currently open) and place the order? If the credit card company rejects the in-person charge, you should be very concerned. On 9/11, a friend of mine living in LA was landed in Chicago while on a business trip. He's a bright guy so before anything else, he marched over to the car hire desks and booked a luxury one-way rental. Time was of the essence as others might start catching on that air travel was going to remain at a screeching halt for some time. If he had to go back and forth with the credit card company to verify the expensive charge (provided cell service wasn't overloaded), he may not have been on his way within the hour. I expect that the car hire companies saw a perfect opportunity to put up their rates too if they moved fast enough.

[agurney]

Re: Interesting

"So I wonder how they bar ordering 50 pizzas but still let you order pizza..."

.. some smart youngster would order multiple pizzas multiple times, but avoid 10x5 in case duplicates were discarded.

[Mentat74]

Yeah but...

What if someone tries to order 2 tons of creamed corn ?

[Anonymous Coward]

Re: Yeah but...

I tried that at a friend's house once. Alexa didn't listen to me, but the look on his face was priceless!

[J. Cook]

Re: Yeah but...

Yeah, someone tried that with me as well, but I had already changed the wake word on it.

Personally, I'm going to be ditching the things here shortly because of the changes to the T&Cs for things, especially now that the madlads at Home Assistant have cooked up a voice assistant that works entirely on-prem, provided a sufficiently powerful box to run it on...

[trevorde]

True transcript

Visited some friends who had an Alexa:

[Me] Alexa! Tell me a fart joke.

[Alexa] The fart skill is not enabled by default. Would you like to enable it?

[Me] YES! YES! YES!

[Alexa] [tells fart jokes]

After a few glasses of wine:

[Me] Alexa! What is a schlong?

[Alexa] Schlong is a river in Italy.

[Me] Err, I they she got that one wrong...

[ecofeco]

What am I missing here?

If you have a family and you have enough credit to be able to afford 50 pizzas (as an example) maybe you should put better security on your damn cards.

Just a thought.

[doublelayer]

Re: What am I missing here?

The problem is that it's the security on the device that makes the transaction, not the card to pay for it. If I can secure my card, but that device can charge to it, which it needs to have if I'm using it to make other transactions, then the security I added to the card isn't going to help.

While we're at it, I have cards with the default level of security and there don't seem to be a lot of options to change that level. There's no interface that lets me put a cap on number of pizzas I could order at once. There's an opaque fraud detection algorithm which has fortunately never gone off by mistake but admittedly I don't spend much money and a slightly less opaque location tracking algorithm, neither of which can be turned off anyway. The next security level I have is "card locked except for recognized monthly bills". Those are the only two levels I'm aware of that let the card work at all, and since only one of them lets me buy anything I haven't purchased on a schedule for several months, on that one I must remain. I'm not sure what you're recommending is a feasible option for people who have any payment cards.

[MachDiamond]

Re: What am I missing here?

"has fortunately never gone off by mistake but admittedly I don't spend much money and a slightly less opaque location tracking algorithm"

I've had years go by between being able to take a proper holiday so any algorithm looking at my spending patterns is going to sound off when I do. I might have purchased tickets from the US to Canada to Scotland on a particular card that's tied to an airline rewards program but put a car hire or train tickets on a different card. In one day I might have charges in the US, Canada and Scotland that are all me making legitimate purchases. But, I never go on holiday according to the bank's records.

[IGotOut]

Four sides

"the product engineer thinks about making the intended thing work, and the security engineer thinks about all the ways that you can game that system"

The CEO "how much can I ignore the security team so I can sell more shit and make number go up?"

The intelligent home user "Why the fuck would I want this spyware in my house?"

[Ken Shabby]

Re: Four sides

I hear they make great doorstops