back to article AI-powered IT security seems cool – until you clock miscreants wielding it too

We're hearing more about AI or machine learning being used in security, monitoring, and intrusion-detection systems. But what happens when AI turns bad? Two interesting themes emerged from separate recent studies: the growth of artificial intelligence coupled with concerns about their potential impact on security. A survey of …

  1. Ole Juul

    knowing when to stop

    I see a problem with appropriate application of the technology in some cases. Actual output of systems can move in directions that are not desirable and this has been shown in its use in police work. Unravelling such a situation is not easy, if even possible.

    1. GnuTzu

      Re: knowing when to stop -- Self Referential

      When AI is tasked with detecting insider threats, it might well have to learn how not to detect itself as an insider threat before it can become one. Gives me the willies.

  2. Pascal Monett Silver badge

    Now wait a minute

    "AI is able to learn sensitive information, such as personal preferences, from a vast amount of seemingly insensitive data"

    Yeah, but up to now, all that data is made available to the statistical analysis machine that is not AI. The machine does not go look for it itself.

    Now, I am sure that it could be possible to surf Facebook and Twitter and glean some info about a specific target, but unless you have your malicious pseudo-AI sitting inside the company you want to attack, or you have compromised its network to extract that data, I don't see how it could capture the vast amount of data it needs to analyse.

  3. big_D Silver badge

    William Gibson

    Isn't this what William Gibson wrote about in Neuromancer? AI security and deck jockeys using AI generated ICE to break into systems?

    It is funny, I would have thought more tech aware leaders would be more wary of AI, they should have a better understanding of what can go wrong with it, for a start.

    1. Rich 11 Silver badge

      Re: William Gibson

      Maybe that awareness is countered to some degree by the knowledge that a lot of the talk about AI is overblown, and that in this context machine learning can fail to produce an effective attack just as easily as it can fail to produce an effective defence.

    2. Sir Runcible Spoon Silver badge

      Re: William Gibson

      I'm pretty sure that the article asking 'does knowledge of AI make you less afraid of it' is a case of taking something obvious and stating it in reverse.

      The actual truth of the matter is simple..

      The less you know of something, the more afraid of it you are.

      (also: Familiarity breeds contempt)

      1. amanfromMars 1 Silver badge

        In Your Dreams, Sir Runcible Spoon

        I'm pretty sure that the article asking 'does knowledge of AI make you less afraid of it' is a case of taking something obvious and stating it in reverse.

        The actual truth of the matter is simple..

        The less you know of something, the more afraid of it you are.

        (also: Familiarity breeds contempt) .... Sir Runcible Spoon

        Hmmm? In this particular and peculiar case, Sir Runcible Spoon, the more you know of the something which cloaks itself in and wears the clothes of AI/Machine Learning/Virtual Machinery, the more one should be terrified of it, given the fact, clothed in many a disparaging fiction from every kind of Doubting Thomas which would be telling you otherwise, of what it is capable of doing to disrupt and destroy humanity as it is portrayed and controlled by mass media machines.

        And IT is only just warming up to the task which humans have not even begun to grasp is their fate/future destiny.

        And also, in this unusual case, Familiarity Breeds Contentment and Deep Satisfaction.

        1. Sir Runcible Spoon Silver badge

          Re: In Your Dreams, Sir Runcible Spoon

          I am suitably terrified, even though I only studied the barest spattering of what was available at the time of my state sponsored education, although at a bit of a loss as to how learning more about it would lead me to feelings of contentment and satisfaction (I lie, but to admit the truth would be like admitting I believe in Unicorns).

    3. steviebuk Silver badge

      Re: William Gibson

      All they care about is saving money. If using AI means they can do away with real people and wages then they'll do it. It's why sales teams have gotten away for years selling shit to the gullible in business'. Because the person paying normally falls for their sales pitch and sees it as a "saving". Ignoring the fact pretty much all of them put their prices up after a year or charge stupid amounts of service or change request charges.

      1. Rich 11 Silver badge

        Re: William Gibson

        This is a situation recounted secondhand, so feel free to dismiss it. One of my mates was booked at short notice to provide technical advice on a cloud proposal being sold to a CTO and his cronies. He turned up to listen to the sales pitch and asked a number of pertinent questions, but had the feeling that his answers were being treated a little too lightly by the management representatives present. It was like they'd already made up their minds. Oh well, he was getting paid for the gig regardless.

        On the way out he noticed that there were several glossy pamphlets scattered between the tables in the reception area outside, and he realised he'd seen some of the managers skimming them in the meeting room while waiting for the sales team to arrive. The pamphlets were extoling the financial benefits of cloud solutions. He went over to the receptionists and started chatting with them, and asked if they knew where the pamphlets had come from. It turned out that they'd all appeared the previous morning, but the receptionists couldn't remember anyone leaving them there (and certainly no-one had asked permission). My mate reached the obvious conclusion: the sales people were real sneaky bastards.

  4. Giovani Tapini

    AI's pitting themselves against each other

    Usually goes quite wrong...

    It is possible to imagine scenarios where successful defence ramps up attack to epic proportions. Just like automated trading systems can cause insane and unbalanced share price changes.

    Closer to the real world...

    The challenge I fear is that finding these AI's is likely to require more cooperation between providers in sharing some level of traffic data between themselves to identify and thwart the miscreants. Trying to defend at your own endpoint will become more like throwing a rock into a pond.

    AI for the low and slow probing is probably more effective at looking like "real" traffic too, also making it harder to detect and mitigate against. Security at application level potentially becomes a far more important level of defence (i.e. not assuming the "techies" can mitigate for me). I don't know many shops that look at application behaviour as an insight into trouble brewing...

    1. amanfromMars 1 Silver badge

      Re: AI's pitting themselves against each other

      It is possible to imagine scenarios where successful defence ramps up attack to epic proportions. Just like automated trading systems can cause insane and unbalanced share price changes. .... Giovani Tapini

      That is the present 'real' world, GT, but it is never ever going to be successful whenever it guarantees the self-destruction of systems admins with enemies both fake and phantom which they and IT are defenceless against.

      And it is something which is certainly realised by both corrupt and subverted sysadmins and that terrorises them quite rightly methinks.

      Real Hope and Virtual Change to Believe In?

  5. Nick Ryan Silver badge


    ...and 100% of the IT literate people who know what they are on about know that AI is just marketing BS and at best what we'll get is human defined limited machine learning metrics, if that. So pretty much what any good monitoring software has been doing for years.

  6. ThatOne Silver badge

    This is getting tiresome

    AI is a marketing buzzword. There are wild-eyed salespersons running around screaming "The end is coming! Did you buy our AI solution yet, or are you doomed? Doomed I say!"

    As for savvy people using it, well, it has its uses so some people will use it, although one would need to check what ISACA means by "AI". In marketing newspeak any program able to make simple "if...then" decisions is "AI", so obviously that moves the goalpost a little...

    (Besides: Do savvy people use AI more often, or are they just savvy because they actually happen to use it?...)

  7. Version 1.0 Silver badge

    So what?

    When one cloud "attacks" another cloud it's called a thunderstorm - there's not real surprise about this, it's the way that the world works - if we build tools then people will use them.

    1. P. Lee Silver badge

      Re: So what?

      >When one cloud "attacks" another cloud it's called a thunderstorm

      And both sides are trying to tell you that, for some unknowable reason, their cloud is better than the other one, and therefore worthy of your cash.

      Personally, I recommend not throwing things into a cloud. Its just fog which prevents you from seeing properly and has been put so far away that it looks pretty rather than grey and wet.

      If you actually keep track of what you are doing with your data rather than sticking it all in a foggy place and hoping for magic unicorns to defend you, you'll do better.

  8. Mystery Machine

    Jumbled article

    Jumbled article - stats and FUD for both

    - The use of AI in the domain of information security to improve the capabilities of attackers and defenders

    - The implications of the adoption of AI within modern/digital businesses and it's/their potential manipulation by attackers (using non-specific means) to disrupt business

    Keep it focused / tight or else you're just writing a bullshit article loosely linked to a buzzword.

  9. fluffybunnyuk

    *sigh* more AI bollocks. Machine learning et al isn't bad or good.It is what it is.

    Its the fuckwits leveraging it that are {bad, badder,baddest or good} usually.

    Its a good thing nadine dorries isn't on this forum to explain it, or i feel sure we'd be hearing how robocop ED-209s will be guarding the irish border next year.

  10. Anonymous Coward
    Anonymous Coward

    Oh, but as soon as both sides become AI enabled, it becomes advantageous to start applying evolutionary learning techniques, the hacker AI's will get better at hacking and the anti-hacker AI's will get better at stopping them in an evolutionary arms race explosion until the hacking-AI is hooked up to the Kill-All-Humans-AGI and ugh we all die >.<

  11. nijam

    It may simply be because they have a different understanding of "risk". IT types will see "risk" as meaning "security risk" whereas PHBs will simply be thinking of the "business risk" of it simply not doing what they want (which is, in any case, something they don't really know either).

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2020