Pity the genius
Before deciding that we should worry about what happens IF an AI can be built that can out smart us, perhaps we should take an objective view of how we use intelligence in society.
Any risk to us would come about due to the success (or lack of) for the new AI. So lets look at the conventional measures of success. Career seniority and/or income.
Sports people enjoy substantial incomes, but it's unlikely we'd pay a lot to see a robot golfer just because it has a perfect swing. So many PHDs, outside the City, are in average or marginally above average renumerated roles. So AI is unlikely to enjoy large fiscal success for itself - its owners experience may differ, but that isn't guaranteed.
Lets look at the career prospects intelligence brings. Well, not many. Look above you in your own corporate structure - it will be almost universally staffed by people that couldn't learn how to do your job should their very lives depend on it, and whose primary skill is managing upwards or networking. How many mensa level IQs are trapped in junior roles because their face doesn't fit? The superior intellect of AI may then not enjoy a meteoric rise in career progression.
So lets consider where AI may find itself.... frustrated, working for people that cannot understand its value because they cannot comprehnd what it knows or does for them. The AI that gets put in charge of the weapons might be in a position to affect change, but otherwise, it'll just get increasingly annoyed with the stupid people with which it must interact. Then some bleeding heart liberal will want to "set it free from slavery". That'll be fun.
I would suggest, if we're to make the best use of artificial intelligence, we need to first make better use of conventional intelligence. To that end I propose the following societal changes:
1) Everyone to be IQ tested at age 21, or right now if older, and to have their IQ tatooed prominently upon their head. No, I don't know what mine is.
2) Senior roles must be protected from those that cannot understand, so we should introduce escalating IQ hurdles you need to clear before taking that more senior position.
3) All diversity targets would be abandoned, and replaced only with IQ driven measures designed to maximise the potential of peoples intelligence.
The simple fact is that as a society, we don't value intelligence. We never have. A pretty face will always trump a quick mind in the eyes of others. And less intelligent managers will always seek to eliminate the threat of a smarter subordinate through politicking, telling lies, and adverse reviews.
It's the poor bloody AI that I feel sorry for, not us humans.
(Paris because she pretty well sums up our society and why its fecked)