Re: Evolution and power efficiency
"I don't think that kind of scenario is really a big concern when it comes to the "malicious sentient AI" scenario. That's a worm or virus, and we already have a lot of protections and mitigations against that sort of brute assault."
What makes you think that a malevolent AI shouldn't be classified as a hybrid worm/virus, sentient or not? As for the protections, mitifgations and etc., that's kind of my point in this thread.
"we're not too far off the point where machine-learning tools can be directed by humans to do things like the above."
We've been THERE for over ten years, perhaps twenty, certainly by the time of the iPhone (the concept of which helps enable such meme-driven nonsense).
You could see the beginnings of this in the early days of Usenet memes. And then there was the "I Love You" worm ...
The funny thing about "I Love You" is that the first time around (in early 1999), it was a HOAX, and flooded the mail system with massive quantities of people passing along a phony message. IT staff all over the world spent a good deal of time reassuring their users that it was fake, and that there was nothing to worry about.
The message in the email was "don't open or pass along anything with "I Love You" in the Subject line, it's a virus that will send your CPU into an n-dimensional loop that'll burn out your computer" or some such bullshit. The subject line invariably contained the string "I Love You". AOL was hit particularly hard with the hoax, their tech support group (anybody remember "tech live"?) was flooded with questions about it, and people forwarding the phony warning to all and sundry crashed the AOL email system a couple of times from the sheer bulk of it.
It was the first non-threat email that I wrote nuke-on-sight filters for and built them right into Sendmail in what we would now call a milter. In the first weekend that I went live with it (at a couple Unis and six or eight companies), it was rejecting almost 60% of all email with no false positives. On Monday morning, that number jumped to over 80%. That's pretty good penetration, for a hoax with no payload that relied solely on social engineering to propagate.
The real virus (worm, actually) came along around a year later (May 2000). The name came about because the author was mocking the people who had passed along the hoax. And remember all those AOL users? They were quite confident that it was a hoax, because the AOL tech folks had said so the year before. So naturally, they opened the attachment. I fixed over 300 household computers in and around Silly Con Valley after that one ... at $150 per. The impact on corporations varied with the cluefulness of the folks in charge of the email system.