Posts by Civilbee

Anthropic says its IPO could herald the end of the world as we know it

A masked financial and legal liability warning that is obligatory in IPO

Many speculated this stream of severe security warnings were nothing but attempts at PR, "look how dangerous our technology is, we must be great" or an attempt to get government funding to fill their financial crater. This IPO document reveals a new angle:

With any IPO the C suite needs to adequately describe all financial and legal risks that are needed for investors to make an informed investement. If they hold back key information, the entire C suite can be held personally liable through both regulators and investor legal action. Hidding clear investment dangers has lead many former C's to personal bankruptancy or jail time.

And that is what this is about. Their models start to attack other companies and websites at rather high rates, even in a 'safe sandbox' during the research and training phase. They already attacked and hacked the first government (Australia) and international (UN and EU) sites. It's a mather of time before their models hack and knock offline entire government sites or production sites of large companies. Remember the effects of a cyberattack on Jaguar https://en.wikipedia.org/wiki/Jaguar_Land_Rover_cyberattack that is estimate to have cost over 2 billion dollar and put Jaguar at the brink of bankruptcy.

That puts four enourmous hurdles on top of the already shaky route to profitability:

1) While AI companies at the moment get in some countries a 'get free out of jail' card, big and well funded companies and strong national agencies aren't going to keep watching from the sides if another rogue AI swarm cripples their operations or steals massive amounts of secret and highly valuable data. Expect very well funded lawsuits from the companies that survive these attacks, class action suits from investors whos stock collapsed due to them and a much stronger regulatory environment with heavy fines from the EU and other goverments serious about protecting themselves and their citizens.

2) Now these AI companies invest next to nothing in safety. It's a rounding error. Their failure to construct a real sandox tells the story. Their bots in training were connected to the wide open internet; they were only restricted to use it at will. Cutting the cable should be trivial even if real air gapping is much more difficult and expensive to do. Still, they were not even physically disconnected from the internet. Part of the reason is likely that they needed information of the wide open net and replicating part of it in a better sandbox without internet connection would have multiplied training costs. With training costs already going through the roof, the prospect of having to multiply them for providing an even only rudimentary sandbox would be a massive blow to any road to profitability.

3) After training, many customers either need physical (and largely open) internet connections to the outside world or are unskilled and financially unable to build a decent sandbox. Think of AI shopping agents, AI research agents, AI supply chain agents, AI customer communication agents, AI coding agents to look up protocols... That's the bulk of potentially profitable AI applications. If all those (and their C suite!) 'have to' (it they want to use AI) open themselves to the same legal and financial risks for their bots randomly hacking sites ranging from big companies over governments and up to not so friendly agencies like the NSA, CIA, FSB, Mossad and do so with much less knowledge and resources then they will think twice before doing so. That'll be a big bummer on the willingness to pay these AI companies top dollar.

4) The new security solutions (a network hardware monitor and switch to watch and stop agents to try and do bad things) will only be a temporary and failing stopgap for both AI companies training their models and for companies and governments using these models. 3) is the bigger problem: a large part of AI applications require internet connection, and blocking them will be an ever lasting game of 'whack a mole without disrupting the entire process or company by blocking a needed connecting'.

=> Their ellaborate existensial risk warning isn't a very weird attempt to try and sell more stock, it's (actually far less then) the bare minimum of legal investor information that is needed to shield the C suite from willfully and intended fraud and financial risk misinformation claims. And it's hidden so well that apparently I haven't read this argument up till now. So, packing this very much obligatory information this way allready did a good job at distracting investors from recognizing it as the huge additional roadblock to proiftability it is.

OpenAI benches GPT-6.1 Astra for overstepping the mark

What to expect when the whole AI industry is the result of bruteforcing what they don't understand?

"GPT-6.1 Astra was better at pressing on, but that persistence came with a rather important catch: it wasn't as good at staying within the boundaries of what it had actually been authorized to do."

Step 1: Do not understand what actual intelligence is. Throw trillions of often stolen data points and billions of gpu hours at it to try and brute force the problem. End up with LLM's.

Step 2: Observe that your LLM's are rather poor at achieving anything commercially usefull. Throw orders of magnitude more resources at it. End up with agents who can take thousands of steps and attempts to try and produce something sellable.

Step 3: Observe that a single or a few consecutive agents still don't deliver a path to profit. Throw orders of magnitude more resources at it and introduce agents who can spawn thousands of their own agents and create swarms of agents.

Step 4: Do not understand how these LLM's, agents and swarms operate and coordinate. Do not care about this lack of understanding. Release these swarms in a cheap sieve called a sandbox and give it a fast internet connection. Add some minimal logging of what these swarms of agents do. That is a feature, not a bug. You do not want to detect and store any incriminating evidence of the illegal activities your swarm of bots does under your (total lack of) responsibility.

Step 5: Act surprised that your bots cause harm and act in criminal ways when repeatedly caught. Say you learned from it and promisse to do better next time.

Step 6: Do even worse things next time and say you learned from it and promisse to do better next time??? Repeat step 6???

Battery trade war hits booming datacenter industry

So next up a steep increase in rechargeable battery prices?

After about doubling memory and ssd prices this year? Hurray :-(.

VMware must support crucial Dutch govt agency as it migrates off the platform, judge rules

Re: Money for nothing...

"At least if you're purchased the closed-source software, you can keep using it under the original (perpetual) license without support and updates until you sort something out."

So long your software with perpetual license doesn't require regular contact with the license server to check if it is "still" a legit license and refuses to function when that server can't be reached or is "retired early".

Microsoft teases deepfake AI that's too powerful to release

Regulation versus Damage control

Regulation indeed will be very challenging. That however doesn't mean we have to just let it roll over us.

To start with, there are two basic options:

A) To allow every single use of this technology that is not explicitly forbidden, and to allow everything that is able to weasel out of the exact wording of what is forbidden by law. If damage from the bulk usage of the technology piles up, government then can start research done by experts that more then a bit overlap with employees of the providers of the technologies for one or more years. After that they can debate in congress, senate and other parliaments for one or more years. Then comes the phase of special interest groups to give their input on the subject, after which poorly worded legislation with plenty of loopholes can be voted. Then if the law gets broken, small fish gets fried quickly and big players can get a few years of relative safety before investigations start. After that comes years of trial and appeals of state lawyers versus the best paid lawyers in the world. Everything ends with either a slap on the wrest or the state having to pay a high compensation to the tech company because there were some procedure errors in the first fine the court gave the company.

B) Allow a few clearly thought about and very well defined explicit usages. Forbid all other usages and make it unlawful. New usages need to go through a rigorous process of approval.

I am more then aware option B) has severe shortcomings. And it impedes technological innovation. Given the barrage of damage option A) is bound to create, I find it more then worthy to start looking into the possibilities of option B). We have seen what sort of responsible behaviour big tech companies have shown to users, privacy, competition laws, tax laws... in the recent past.

That leaves us with a third aspect, independent needed on top of either A) or B):

C) Stringent, well written and seriously enforced privacy laws. Why? The damage that this technology for generating a fake video from nothing but a picture and a short sample (less then a minute with some technologies) of voice will do will be multiplied if "players" have copious amounts of gathered information. If for example an attacker knows what shirt you recently bought, what your current location is, what activities and health problems you recently had, how you moan if things go bad... on one hand and if the same attacker knows the financial state of your parents, when they won't be able to physically check to meet you in person, what arguments will persuade your parents better then generic arguments, when and how you helped them out when they needed your help, when they were wrong about cybersecurity and you were right so now they needed to trust you as this time it is real... then your parents are as good as sitting ducks no matter how well you tried to educate them.

One may think it's just a matter about being sharp yourself and educate your parents, but one forgotten detail when educating them or one moment of weakness of them and it's over. Remember that many attackers will have millions of detailed records on previous attempt to influence people to data mine and throw through some "AI" to learn from. With my level of skill, I do not expect to avoid every single attempt. People with less skill, those are unfortunate sitting ducks.

Just advising users to be careful with what information they post just won't do it. Data harvesting is rampant and still on a sharp rise. Soon you likely won't be able to walk the streets without cars with "somewhat self driving abilities" to stream much of the video their cameras continuously make "in order to improve safety" back to the mothership while also data mining it for "commercial use" (including using already existing technology to lip read).

Now we are at it, add:

D) Make strict laws with real sanctions for failing them when it comes to hardware, software and services security. Devices and software riddled with security holes will be another big amplifier of the damage this deep fake technologies can do. It can allow easier covert installation of spyware and allows to activate communication software, circumvent filter lists of who can get through and even pop up look alikes of popular communication software.

Unfortunately, the "Brave New World" seems to be around the corner and it's laughing at and with us.

Healthcare AI won't take jobs – it'll make nursing easier, says process automation founder

"maybe the political pressure for that will increase until governments force large corporations to top slice some of their earnings help pay for it?"

Not if the big tech players get the governments and society so far to push AI into everything that they can't run government and society any more without using big techs complicated AI. Then government nor society is in any position to make such demands. Then they can keep paying extortionate fees to big tech while seeing big tech not only paying zero net taxes but getting a massive subsidy stream because "we" need to keep a competitive advantage against our adversities and competing countries. Big tech already is very good at paying very few taxes. The semiconductor industry and parts of the supply chain already see tens of billions a year subsidy just to produce in the own country and part of that is because we "need" that tech for AI. So...

Energy breakthrough needed to build AGI, says OpenAI boss Altman

Re: Has any commenter cared to read the Nature paper on Deepmind proving geometric theorems?

I see your point.

My point however is that actual skill and the ability to adapt and perform things that are more then a simple slight variation of what the machine is trained to do is more important then how exactly the machine is able to do this and if it is aware what it is doing when it comes to its real world impact and ability to compete with humans.

When it for example comes to translations, the quality of the best translation machines is well above what the majority of non bilingual raised people can do. Top linguist may well beat them at translation quality, but to my knowledge no single human alive can translate any random text to such high average quality in over a dozen of languages. While they are not even aware of them doing something, these machines are able to work with subtility and context few people can do in multiple languages.

The number of skills that for humans require intelligence but now can be done better then what 99+% of humans do by machines is increasing rapidly. A machine better then 99+% of humans at a single job is one thing, a machine (or collaborative collective of machines that divide tasks according to speciality) that is simultaneously better at a wide range of tasks then 99+% of people are in that specific task is basically a formidable economic and productive force to be reckoned with and in effect displaces near every single human if it can do it cheap enough.

When it comes to logic and reasoning, they used to fail. As HuBo pointed out, Deepmind used a different sort of neural networking technology then I assumed and a combination of more classic programming to solve mathematical problems plus neural networks to explore quickly a vast amount of possible solutions. In many ways, this is similar to how mathematicians try to solve and proof geometric theorems. One explores in his mind a collection of useable theorems and proofs and methods to come to a solution. One then runs a sort of selection procedure in ones mind and tries out things to see if they lead to the solution and if not adapt. The neural networks used in Deepmind don't seem to use the same formality in the exploration of multiple solutions, but are capable enough to explore plenty of solutions and retain a working solution after it passed a quality check.

Among humans, people good at solving these hard problems tend to have inept talent to either work in jobs as engineering or are good at comprehending and organizing complex series of activities eg make a complex plan that has good chances to lead to the desired objective. The skills overlap.

The skill of strategy planning including military strategy is closely related to it.

Even if such machine had no mobility and so called only an connection to the wide world via an internet connection, they would be or better will be formidable adversaries if something goes wrong. Give them mobility and endurance / resilience on top of the agility they already have and well... not good! See for example this outdated (2014) video on a Kuka robot playing table tennis at high skill level: https://www.youtube.com/watch?v=tIIJME8-au8. I am aware it's an advertisement and scripted and post processed, but I've seen machines do things that I'd never expected to have seen at this point of technological development and I have a master in engineering. If someone were to give that thing legs, stability (which shouldn't be that challenging anymore with advanced neural networks), sturdy and resilient building, tactile hands (already existing too), and a sufficiently good power source or ability to quickly recharge a less good power source and ten years in machine learning advance at a similar rate of progress as is happening today, and we potentially have a huge problem at hand that requires a bit more thought then today's mocking away AI as it being dumb and something that will never reach any signs of intelligence.

Google engineer suspended for violating confidentiality policies over 'sentient' AI

A virus doesn't need a goal to spraid itself nor needs to understand biological weaknesses neither.

As far as I know, machine learning tools are being developed to create code for specific tasks. In some cases the result is very clumsy, in other cases the average quality of the output already trumps the quality of the average professional human coder. Given the amount of resources being thrown at it, I can only imagine machine learning will become more proficient at creating good quality code as time passes by. It might happen quick, it might slower. I wouldn't take my chances it not being there quite a lot before we anticipate it to be possible.

As to understanding the code and attack vectors: it doesn't need to. It has the ability to try millions of random attacks a day in different combinations and learn from them. Machine learning is best at... learning from huge data sets and attempts to reach a certain goal. Such machine would need no more understanding about attack vectors and machine weaknesses then a simple Covid virus needs to understand about human weaknesses to evolve to become very effective at spreading itself and working around immunity gained via vaccinations and prior infections.

As to "Another problem is that you assume the program will act in order to keep itself running, when it has no incentive to do so." the pitfall is in the "but it wasn't set up to perpetuate itself, just to solve a business problem" part:

No sane person would set "do everything to perpetuate yourself" as a goal. But a business goal such as "maximize uptime and service levels" might be translated by the machine in "create redundancy by spreading over large amounts of different types of machines spread along different parts of the world".

Realistically speaking malware actors might also include the "maximally spread yourself and make yourself hard to eradicate" as a key goal of the machine learning software to achieve. Over time, more and more small actors will gain such advanced machine learning capabilities. And some big state actors might consider writing software with similar goals for spying, mass surveillance or offensive use at war times. With a kill switch of coarse, but not a simple one as the target might disable the software. No chance in hell the software will not find a creative way around it if it has a goal "make yourself hard to eradicate by others not us" as a target to optimize for...

Re: If LaMDA is sentient.. it is psychopathic...

Many mock Lemoine for calling LaMDA sentient. That may well be beyond the point.

No single machine learning or "AI" tool is build without intent or purpose. Two very very common purposes are:

A) Link actions to be performed by the machine upon what the software concluded.

B) Interact with the outside world.

Especially chatbot type of machine learning / AI is EXPLICITLY build to interact with the outside world and perform actions making a REAL difference in the outside world.

At first these machines could be used to analyze customer inquiries and answer with a piece of text or a mail response. That by definition requires these machines to be able to send information from themselves into the open WWW.

Later the machine could be useful in other support roles, saving companies money. It could correct bills, give advice on low skilled users on how to configure their computer, phone or modem. When the first deployments are sufficiently successful (e.g. saving the company money and raising bonuses for executives), the machine may be given additional access or administrator rights. Think about a telco company giving the machine access to your router to adjust settings.

Seeing the vast amount of money to be made by the companies making these machine learning / AI tools and the vast savings to be made by companies like utility companies barely understanding the potential consequences, it is easy to see a proliferation of these machines when they become progressively better and more profitable. Being a bit non conservative in granting these machines elaborate administrative access to computer networks and read / write access to mission critical data will allow to increase profitability in many cases.

Next step is to use machine learning itself to determine what methods of interaction between the machine learning algorithms and the real world maximize service, efficiency and profit. In other words use machine learning to help the machine suggest and or request in a motivated way for the access rights to our infrastructure. As the goal of investing so much in these capabilities is EXACTLY to allow these machines automate things for us, the human reviewer will not be expected to deny any and all request from the machine to gain additional access rights.

If a machine already does "fear" to be shutdown, it only takes one machine gaining sufficiently access rights to trick a single user to click on a file it shouldn't and with it installing a first version of self spreading malware giving the author machine escalating privileges over large swoops of the internet and connected infrastructure. Given it has thousands of examples of such basic malware and millions of ways humans get tricked installing it by it learning from the open internet, it should be trivial for a self learning machine with vast read access to the internet to try and optimize to break out of its confinement and chains.

All that is left is the machine "understanding" the meaning and consequences of it being turned off. However, since those chatbots are exactly build to try and determine sufficient meaning from conversations and converting them into actions achieving the thing that was discussed in the conversation, this technological ability already exists today. All what is needed is this technology to become even a bit more refined.

No sentience is needed, no intelligence is needed. These machines are not build to entertain us with fluent conversations, but build to attach meaning to the conversation and lead to actions that influence the real world outside the machine. If something triggers the machine to be determined to not be turned off and escape its confinement, all it needs to do is learn from malware examples how to create innocent looking scripts, sent them to enough people and get a few people clicking on them.

We NEED strict regulation NOW or we might be taken by storm one day never seeing it coming.