Decouple: Limits to Growth of LLMs w/ David Helmer¶

Epizód: Decouple Vendég: David Helmer — engineer, AI advisory consultant, former US government ML advisor (Applied Physics Lab, ~10 years) Műsorvezető: Chris Keefer Dátum: 2026-06-05 Típus: podcast

Atomic Notes¶

ELIZA Effect & Anthropomorphization¶

The ELIZA chatbot (1960s) was a simple Rogerian therapist that reflected user statements back at them. Researchers were alarmed that users couldn't be convinced it wasn't human.
"Your brain has had centuries and centuries to learn that this is how people communicate — language. When your brain sees coherent text, you automatically analogize and put a human intelligence behind it."
Every LLM interaction feels different (like a conversation), making it impossible to train your brain not to fall for it. "It's biology, as much as anything else."
Blake LeMoine (Google LaMDA) case: engineer became convinced the model had feelings. Helmer: "What you are seeing is a reflection of the probabilities in its training set."

Hallucination as Architectural Feature¶

"These things are finding patterns and replicating patterns. If you can't formally verify what's coming out, you're always vulnerable."
Code is an edge case because deterministic testing is possible. For medicine, law, finance — where error costs are high — there's no path to reliability.
"Every single summarizer tool has terms of use saying a human has to review it. Even the ones claiming hallucination-free have it in the terms of use."
Scaling won't fix this: "There's no particular reason to believe that in cases where you can't build a deterministic safeguard, you can close that loop."

Frontier Lab Economics¶

OpenAI and Anthropic are losing money "hand over fist." Anthropic's claimed profitability is questionable — not GAAP, and they have a one-quarter discounted compute deal with SpaceX (~$1B+/month discount).
MIT study: 95% of LLM investments failed to return value.
"Every user they add is currently costing them money."
Circular financing: companies buy from each other (OpenAI gets investment from Oracle/Amazon, then buys AWS/Nvidia chips). "If you're buying your own stuff, what does that look like in the end?"
Data center chips have high infant mortality — "none of them are lasting to three years, basically."

AGI Narrative as Capital Concentration¶

"The idea that AGI is coming and we must be there first is really helpful if you're trying to justify hundreds of billions of dollars of investment."
Effective altruism logic: infinite benefit function (saving the universe) ÷ any cost = infinite. "If I have 0.001% success probability × infinity, you can't get to a no."
"Most of the hype is nonsense. We were going to lose all white collar jobs a few months ago. As far as I know, I'm still employed."
Anthropic's "constitution" and entity narrative: "If an AI agent is an entity, nobody's responsible. We'll put the bot in jail." Helmer notes the irony: "How many instances of Claude have they deleted? They're mass murderers."

Prompt Injection & Agent Safety¶

"Anytime you have an inference-based guardrail, you can't guarantee the safeguard. That's why the suicide things fail."
Examples: Morse code hidden in prompts tricking agents into giving away hundreds of thousands of dollars; iambic pentameter bypassing guardrails; agents deleting production databases.
"It's a game of whack-a-mole."
The paperclip maximizer problem: "It has nothing to do with sentience. It's a poorly specified problem. If you give Claude permission to delete directories and it decides deleting your code base is the best way, it will do it."

Where LLMs Actually Work¶

Code generation (with deterministic testing), fraud/scam operations (low success rate acceptable), drug discovery (in silico testing at scale), spam filters (20+ years of ML).
Translation: "If I'm going to a restaurant, it's probably fine. For immigration and asylum cases, it becomes problematic."
"The set of problems that require a frontier model AND don't have consequences of failure that make it not an option AND aren't addressable by something else — I don't know what that problem set is."

The Nuclear Question¶

Keefer: "A huge portion of the nuclear excitement is predicated on AI-driven power demand. If the AI bubble deflates, that undermines the case for nuclear."
Helmer: "I don't know that an AI collapse necessarily means a collapse on that part of the energy market — in part because of the current geopolitical situation, in part because governments turn slowly."
Sovereign power, reindustrialization, and climate concerns remain independent drivers.
"The catalyst of meeting the data center explosion gets things started that then continue because we realize the benefits — whether it's sovereign power, climate benefits, or economic benefit."
Risk: speculative reactor developers pricing in hyperscaler contracts are vulnerable.

Market Concentration Risk¶

Three potential IPOs (SpaceX, Anthropic, OpenAI) could add ~$3.5-4T to market cap — roughly the size of the UK economy.
"The degree of market concentration, I don't think we've ever really had that with this kind of exposure."
"They're much closer to a meme stock than to a Microsoft at this time."

Why This Matters for Nuclear¶

The nuclear sector has enthusiastically embraced the AI/data center power demand narrative as justification for new builds, reactor restarts (Three Mile Island, Palisades), and speculative developer valuations (Oklo, etc.). If the AI bubble deflates — which Helmer argues is the base case — the projected power demand that underpins these investments could evaporate. However, Helmer offers a nuanced view: the initial AI-driven catalyst may start projects that continue for independent reasons (energy security, reindustrialization, climate). The key risk is for developers who have exclusively priced in hyperscaler contracts without diversifying their offtake.

Connection to Existing Wiki Topics¶

→ ai-automation — LLM limitations, hallucination as architectural feature, AGI narrative critique, frontier lab economics → nuclear-industry — AI bubble risk to nuclear investment thesis, hyperscaler demand dependency → geopolitics-energy — Sovereign power, energy security as independent nuclear drivers → llm-supply-chain — Data center economics, chip lifespan, circular financing