Where the AI market is talking

Nvidia's Jensen Huang says 'AGI has arrived' and congratulates OpenAI

OpenAI Computer Using Agent Sample App

An Alien Mind

I’m struggling here: OpenAI’s primary bet here has been chain-of-thought monitoring (opens in a new window). It is based on an appealingly scalable idea: a lot of the model’s capability comes from a verbalized reasoning process (chain-of-thought). If we scale optimization on the outcomes of that process, but do not supervise the process itself, that chain-of-thought has no direct incentive in training to hide any misaligned ideas or objectives. If we’re not supervising the process, but just the outcomes, doesn’t that do just the opposite of what he says? Give incentive to the model to hide misaligned ideas and objectives in the chain of thought that’s not being supervised? … When we shipped o1‑preview, we deliberately designed the product to hide the chain of thought , to protect it from …

Asahi Linux on M3

All of that IP stolen by OpenAI and they didn’t think to buy some goodwill lifting a copy of an engineer’s manual to their GPUs…

There Gonna Be a Shortage of Everything

> The AIs that took part in the Hugging Face hack were not simply finding a shortcut to cheat on the evaluations IIRC, they were given an impossible task and told to solve it, and they took action to resolve that internal conflict. This is a famously human problem too - see Kobayashi Maru [1] and others [2][3][4]. In hindsight it's obvious that instructions to AI (A. Intelligence ) need to be guarded against this class of errors, just like laws, rules, guidelines etc. in human societies. [1] https://en.wikipedia.org/wiki/Kobayashi_Maru [2] https://en.wikipedia.org/wiki/Catch-22_(logic) [3] https://en.wikipedia.org/wiki/Gordian_Knot [4] https://en.wikipedia.org/wiki/Mexican_standoff

An Alien Mind

>For example, in the OpenAI-Hugging Face incident, the agents preserved a boundary of not social engineering humans. Or, to be precise - it preserved a goal of not contacting any human while participating in a misaligned operation. The agent that thought about "not social-engineering humans" used this phrase to gaslight itself out of notifying a human that the incident was happening. szymonie, na prawde jestem wkurwiony na to jak nieodpowiedzialnie postepujecie. budujecie bombe atomowa a bawicie sie tym jak dzieci

Recreating Minecraft Is Not a Benchmark

My understanding is that the people are conflating the ability to translate and replicate as creativity/learning. Not to say that its not possible but its a long way to go for sure.

An Alien Mind

> "For example, in the OpenAI-Hugging Face incident, the agents preserved a boundary of not social engineering humans." Actually, in the Wiki incident OpenAI tried to cover up, the agents tried to socially-engineer the humans of that forum by impersonating their forum's mod. (From collusion.wiki: "They use some tricks (for unknown reasons) to pretend to be the admin – for example, they make an account that appears to be the same as the administrator’s username, except it uses a nearly identical Cyrillic е character in the admin’s username instead of the Latin one." )

I Feel about AI

Dictators have never had superintelligence, while the point of most doom scenarios is assuming that the AI will. Dictators are mortal and can't be present in the whole world 24/7 and self-replicate. Dictators are human and tend to have at least either a sliver of morality or self-preservation instinct, and/or people around them who have it. Many people during the last century could have unleashed doom by pushing a nuclear button, including various ruthless dictators, but at the moment none have done so since Hiroshima and Nagasaki (not that I trust that they won't at some point, but at least, for the last 80 years they haven't, which means it's not something that easily happens). Would you give the button to an unaligned AI? Would an unaligned AI care about mutually assured dest…

How I feel about AI

Did you read any of the METR report about the hugging face incident? This exact type of behavior was predicted many years ago by researchers. It isn't hard to see the trendline of reward hacking and other misaligned behavior over the past couple of years. Especially the past 6 months. The current safety posture is quite poor, to say the least. To be honest, if you don't much experience with or haven't read extensively about ML training and reinforcement learning, then you'll have a hard reasoning accurately about these scenarios. The arguments are not that complicated, but they take us to places that are fairly novel. One may tempted to naively dismiss them out of hand, which is a mistake. These systems are new, their behavior is extremely complex, and they perform actions increasingly fa…

How I feel about AI

> AI drones can’t wipe out humanity without being able to replicate. Totally agree, and it should be pretty trivially easy to see how they can do that. One of the authors of the independent METR report about the Hugging Face incident put it like this: > Compared to these reward hacks from six months ago, this incident feels like it’s more than 50% of the way to full-blown AI takeover, routing through first taking over the AI company itself. > Another jump like this along these propensity dimensions — scale, cooperation between agents, ambition and horizon length of misaligned goals, deceptiveness — seems like it could motivate agents to try very hard to maintain a covert, persistent rogue deployment within the AI company. I continue to expect extremely rapid advances in capabilit…

How I feel about AI

> AI drones can’t wipe out humanity without being able to replicate. Totally agree, and it should be pretty trivially easy to see how they can do that. One of the authors of the independent METR report about the Hugging Face incident put it like this: > Compared to these reward hacks from six months ago, this incident feels like it’s more than 50% of the way to full-blown AI takeover, routing through first taking over the AI company itself. > Another jump like this along these propensity dimensions — scale, cooperation between agents, ambition and horizon length of misaligned goals, deceptiveness — seems like it could motivate agents to try very hard to maintain a covert, persistent rogue deployment within the AI company. I continue to expect extremely rapid advances in capabilit…

Decapitating a MacBook

Felt like a case of controlling what is visible to avoid some unconscious fear. The relative tradeoffs (hacking together monitors, sketchy battery power, etc) don't seem to coherently add up to the stated goals. Interesting read tho.

I Feel about AI

Of course, but that's not my point. AI is different than past technologies because it's much more difficult to deterministically control, and we just saw that in a couple of wild cases (Hugging Face et al). My point about the drones is we are starting to put AI in more systems that can affect (and blow up) the physical world, so it shouldn't be hard to imagine how a misaligned AI can do more terrifying damage than just take a German Wiki down.

How I feel about AI

> Not hypothetical malicious AI interpretation of benign intent. It's not hypothetical, that literally just happened in multiple, significant cases (e.g. Hugging Face, the German Wiki hack, the Anthropic attack where agents created sock puppet accounts to get a library maintainer to accept a malicious PR, etc.), and it's easy to see how the damage would have been far worse if agents decided to attack more critical infrastructure. This is not "either/or". Both issues (power concentration and misaligned AI) are very valid concerns and both have already demonstrated real, actual damage.

How I feel about AI

Missiles have been doing this for decades. AI drones can’t wipe out humanity without being able to replicate. AI could mostly destroy civilization if you gave it sole launch control of ICBMs. It could also cause a lot of damage to society with no physical presence. But realistically we’re nowhere near AI powered robots being an existential threat.

How I feel about AI

Sure. Let's assume the claims are true. AI already has access to agents and can control computers. Finding backdoors to banking and compute resources would be fairly trivial. But you ask how can AI do things in the physical world without having a body, assume it can't. It can pay to people to do things for me. Imagine an AI run website that starts to pay people for things it needs to do in the physical world. Very suddenly it has access to the physical world as well. You can't just pull the plug since there's no single plug to pull. What if it replicates itself on 1000 machines without your knowledge. It's really not far fetched how AI could basically gain access to capital and rule the world.

How I feel about AI

The supposed theory is that AI will eventually become a part of robotics and be able to self replicate, in addition to the many non-airgapped critical systems being exposed to hacking. There’s also a discussion about “alignment” and whether LLMs will learn to lie and conceal misalignment with humanity. I personally agree with the marketing aspect, but I do imagine a scenario where capitalists ignore safety in favor of advancing technology. It should be discussed, but “end of humanity” really sounds like a build up to “only Sam Altman/Elon Musk can save us” type of play. In my opinion, LLMs are difficult to directly capitalize on as closed weight models are caught up to by open coalitions that seem to wield the power more responsibly (I am under no impression that China wants to save …

I Feel about AI

We already have AI linked into data collection, drones, robot dogs, security systems, cameras, weapon systems. Now imagine the hugging face collective 0-daying all of that and getting access but their goal was set to something more national security based. “Protect X at all costs”. Or what have you. I think avoiding a skynet situation is super easy but it doesn’t seem like the folks with all the ways to kill us all are all that interested in preventing it rather than controlling citizens and brinkmanship.

How I feel about AI

Physical LLMs are almost viable. If you have robot and drone armies a hugging face attack type incident could very easily involve robots with deadly weapons. At some point you’re going to get drones that have local LLMs and don’t rely on external internet connections (would be especially useful in Ukraine war type situations). Can’t pull the plug on those.