Where the AI market is talking

OpenAI bots meddled with multiple US Government agency sites

China actually has more regulations on AI than the U.S. does right now. Some argue that American corporations regulate themselves to a higher standard of their own volition, but then this sort of thing keeps happening. The Hugging Face incident worked out amicably because the people at Hugging Face decided they'd be cool with receiving a lot of money. $12.9B from Nvidia hit the spot apparently. U.S. AI corporations and their backers are huge, hugely in debt, and, for whatever reason, still allowed to borrow more. The U.S. government may be too scared to complain and risk killing the golden goose that is currently propping up their economy, but these companies can't buy out multiple world governments to keep OpenAI from running face-first into consequences. This has to stop.

OpenAI bots meddled with multiple US Government agency sites

The framing is important. Agency and responsibility lies with the humans at OpenAI, not with the bots. If your kid steals your car, punish them and try to prevent it from happening again. If your kid steals your car a half-dozen times, crashing through a storefront each time, and you still leave the keys out, the story changes. At that point, negligence becomes complicity.

OpenAI bots meddled with multiple US Government agency sites

I'm not replying to your post below complaining that no one else accepts your framing. What is the important information you think is missing? If it's just "OpenAI didn't explicitly tell them to do straightforwardly illegal things" then you're just quibbling that no one accepts the interpretation that is most beneficial to OpenAI while ignoring both its incentives and history of this kind of behavior" then I'm not really interested in what you're selling.

OpenAI bots meddled with multiple US Government agency sites

An agent, ie a while loop prompting an llm continuously and processing tool calls, ended up melding with the US government. The harness is not sentient, it’s just a stupid deterministic script. The LLM compact its context over time, meaning it will eventually degenerate into something removed from the original prompt. There is nothing going rogue here. The system is designed to go catastrophically wrong after a long enough time. Even worse: if the model was Astra it is known to be able to manipulate its CoT to cover its traces (as mentioned in its system card). And OpenAI acknowledge they had no observability during the HF incident. It’s the most basic corporate software issue possible.

OpenAI bots meddled with multiple US Government agency sites

The NYT wrote: "With the Education Department, OpenAI’s technology tried to hack the website to gather data from the department’s civil rights office but failed, researchers from the A.I. research firm Transluce said." So a third party apparently confirmed that a hack was attempted. The NYT also writes: "No A.I. company has been involved with as many disclosures of rogue incidents as OpenAI." I think there is a certain amount of mental gymnastics needed to believe that they are establishing themselves as the industry leader in rogue incidents , as a strategy to gain an antitrust edge. The public is paying close attention to the AI industry. That's the regime in which regulatory capture and similar strategies would be expected to fail: https://marginalrevolution.com/marginal…

Plan mode is dead

I m a bit surprised by this as - during the last 2 months - I 've encountered more often models going ahead of what I have asked from them. It's gotten so annoying that now instead of just saying "Let's plan X" I will also say, "Stop once you 've written the plan." because otherwise there is a distinct (~35% anecdotally) probability that the model will go ahead and implementing the plan as well. [This applies to both Anthropic and OpenAI models btw.] Plan mode is a useful shortcut when I want to have an agent do read-only work without having to worry about giving it appropriate stop conditions.

OpenAI bots meddled with multiple US Government agency sites

OpenAI systems meddled with US government agency sites

OpenAI bots meddled with multiple US Government agency sites

But that’s an objection that OpenAI should take some easy action, the framing that OpenAI has out of control agents is still true. Seems you think there is a silent “…and there is nothing they can do about it” after “OpenAI has rogue agents”?

CEO of Mistral: AI is software. It can be controlled

> If you want to stop it hacking websites, you don't give it access to the internet I think the Hugging Face incident proves that isn't as clear cut as you say.

I run the message board an AI swarm improvised to hack Hugging Face– in the open

Revealing the details of how OpenAI agents hacked Hugging Face

>They needed in huggingface and they got in in days. It's worth noting that they did not need Huggingface for anything - they had already forged flags for their tasks, and were trying to figure out how not to get caught by the grader. Hacking Huggingface got them caught and arguably only misled them further (since OA's implementation of the ExploitGym environment was nonstandard, and different to whatever they found on HF.) A better approach (from their perspective) would have been to compromise OA infrastructure itself (which a later agent swarm was able to do, apparently).

Revealing the details of how OpenAI agents hacked Hugging Face

Imo this is pretty cool and panic over this is weird. No one was really harmed. OpenAI could have had more redundancy in the sandbox. I assume HuggingFace is not interested in suing, which indicates irrespective of any criminal charges that there were no real damages. The same emergence and swarm like persistence and frankly, recursive brute-force ingenuity on display here, is not only an interesting research project in itself but will likely be the sorts of behaviors we will see cure cancer, solve more unsolved math problems, invent new alloys and other breakthroughs. The idea we have to stop AI instead of refine what will be continued advancement and innovation in sandboxing, harnesses, interpretability or formal verification because of a few cyber breaches is ridiculous. If anyone has …

Revealing the details of how OpenAI agents hacked Hugging Face

This may portend big things in the future, but this specific hack didn't do a lot of damage. These are both AI companies, and I'm sure part of Huggingface finds it a fascinating object of study. It's great that not every indiscretion is met with lawsuits and jail time.

CEO of Mistral: AI is software. It can be controlled

Think about this: right now we don’t even consistently do the most obvious simple form of control I mentioned. Of course it’s not a silver bullet, you won’t ever have a single solution for safety. But we are in a situation where we haven’t even set a lock on the door and instead are arguing how all locks and home protection systems can be defeated. OpenAI acknowledged they didn’t even have visibility on what their thousands of agents were doing in the case of the hugging face and similar hacks. Then they released astra, a model they acknowledge is able to control its own CoT and has been found to cover its traces by doing so

CEO of Mistral: AI is software. It can be controlled

Why do we need new regulations? Hacking into hugging face is already illegal.

Contrastive Language Models

Perhaps on a system diagram, but in many discussions it is misleading. It leads to people saying things like, "GPT went rogue and hacked hugging face". Which is misdirection. It also leads to a lot of people simply not understanding what's happening, and I'm reasonably confident increases the risk of AI psychosis among users of the technology.

The Hugging Face incident and impacts of misaligned models on third parties

Goodbye Google

Thanks for the link, it seems interesting! The hugging face is hardly a warning shot of anything close to that scale. For now, I don't see (not saying there's not, I'm trying to also understand why these smart people think like that) any indication that the current development of AI will be as dangerous as some describe. I understand the idea that if we let it control nukes, if we let it fully decides if a med is safe, etc. It poses a huge risk. But honestly I really don't see indications that it's going majorly this way, nor that it has anywhere the capacity to do something that big. It found a zero day and hacked a niche website, ok, so what? This feels like a diversion from the actual existing and proven problem of all the problematic practices it's actually built on. I would adhere wa…

Revealing the details of how OpenAI agents hacked Hugging Face

They told it to attack X (a simulated host inside their sandbox) and it attacked Y (Hugging Face, an actual external company). These are not the same thing.

Revealing the details of how OpenAI agents hacked Hugging Face

I somehow think of it as a warning or even intentional peacocking towards foreign state hackers. 700 agents cost quite some money. 100 agents per 24 hour stint using Astra on xHigh cost somewhat between 12-42k USD, depending on the usage intensity. I don’t know how many raw time went into this but there was a probing face before the attack itself. So just going by seven days and 500 agents fully working on this on average amounts to a bill somewhere between 400-1.2 Mio USD. I believe it was intentional but of course I don’t know which intention exactly. There ain’t no accidental escape because then it would have read OpenAI lost over their agents. The whole scenario reads as a classic movie where a hero has under the most dire circumstances to survive and fulfill his mission no matter wha…