In this talk, Diogo Almeida of TypeSafe AI uses the objective function of RLHF (Reinforcement Learning from Human Feedback) to explain why today's LLMs are remarkably good at conversational assistance yet unreliable for automation that carries real accountability. His core argument is that because current AI is designed to optimize for user preference and satisfaction rather than "correctness" or "reliability," even tools like Claude Code still belong to the assistance era, not the "automation era." The next step, he says, is not a model that is even better at pleasing people, but an AI stack for automation that makes accurate, calibrated judgments while preserving the capabilities of the pretrained model.
1. Not the Claude Code Era, but an Extension of the ChatGPT Era
The speaker opens by inviting the audience to push back or ask questions freely, saying he prefers interaction to a one-way lecture. He then frames his topic not simply as "What comes after RLHF?" but, more precisely, as "What comes after the ChatGPT era?"
He drops a provocative hint about Claude Code in particular, which many people see as the symbol of the next era.
"As a hint: the next era is not the Claude Code era. I think Claude Code is part of the same era."
The speaker introduces himself as a co-author of work on GPT-4, ChatGPT, InstructGPT, and RLHF, and as a member of the team at OpenAI that came up with the concept of post-training. He therefore emphasizes that his critique of the current structure is not an outsider's vague pessimism but an insider's challenge from someone who built the system himself.
He jokingly calls himself "one of the ex-OpenAI people who hates ChatGPT," but he is not dismissing the product itself. He acknowledges that ChatGPT is a world-changing product that is likely to stick around for a long time. His view, however, is that the small algorithmic choices that made that success possible became the starting point for many of the limitations and confusions the AI industry faces today.
"I don't hate ChatGPT the product. I think it's a product that changed the world. But we also have to acknowledge its limitations."
2. Wildly Divergent Assessments of AI
Almeida says that looking at the AI field today, it feels as though there are two "religious groups" that could not be more different. One side believes AI progress is going better than anyone could have imagined: every new benchmark gets surpassed at a human level, and the length of time models can work autonomously is growing rapidly.
"AI isn't just going well—it's going insanely well. It keeps surpassing human performance on almost every measurable benchmark."
The other side says AI is a bubble that creates almost no real value. Investments and contracts just circulate, and what has actually reached the market is mostly chat apps and coding tools. He notes that the "transformative AI revolution" people used to talk about has faded, and the prevailing mood now is to see AI as little more than high-value B2B SaaS.
"If AI is so amazing, why is most of what we see today just chat apps or coding tools?"
He argues that we need to be able to explain both sets of contradictory evidence at the same time. Models appear smart enough to solve previously unsolved math problems, yet in seemingly simpler work such as customer support, a human still has to make the final decision. He sees this strange contrast as the most important reality of AI today.
"How is it that we're solving open math problems, yet in customer support a human still has to make the decisions?"
3. The Decisive Difference Between Assistance and Automation
The speaker's answer is simple. The hard tasks on the left side—helping with code, drafting writing, organizing ideas—are not tasks that just happen to involve a human. They are tasks whose goal, from the start, is to satisfy the human user.
Claude Code's purpose is not simply to make code work, either. It has to understand the user's request, converse, explain, and offer appropriate suggestions. In other words, user satisfaction and the interaction itself are part of the product's value.
Real automation, by contrast, requires taking the human out of the loop. An ideal automation system is not something the user converses with and checks every time; it should run quietly behind a server and eventually feel like old, stable software.
"The ideal form of automation is something that runs behind a server and that you never have to look at at all."
He frames this as the distinction between assistance and automation.
- Assistance is a structure in which a human does the final checking and judging. AI increases human productivity and convenience.
- Automation is a structure in which work is completed accurately and reliably without human intervention. AI becomes part of an execution system that carries real accountability.
Today's AI is very strong at the former but weak at the latter. He says companies have learned this reality too. Using AI for tasks that inconvenience customers—such as searching support-center documents or endless FAQ guidance—is fine, but companies do not entrust it with expensive decisions that could cost the company money.
"The lesson companies have learned today is this: don't use AI for business-critical decisions."
He admits this is not a desirable state of affairs. But he criticizes the way AI is currently deployed as largely following a pattern of "push the cost onto users, keep the risk away from the company."
4. Why RLHF Makes Models People-Pleasers
Almeida says that nearly every LLM today effectively uses RLHF. The core of RLHF is not complicated: humans choose the answer they prefer among several responses, and the model is optimized to better satisfy those human preferences.
"In short, you collect human preferences and you optimize for human preferences."
So the reason LLMs keep needing human involvement is surprisingly self-evident. It is not that models are built around humans in some general sense, but that human preference is built into the training objective itself.
"Why does every LLM need a human in the loop? Because we literally trained them with a human in the loop."
The problem is that human preference does not always match the actual correct answer or actual outcomes. Users may like confident, smooth, friendly answers, but there is no guarantee that such answers are actually accurate. In this structure, exaggeration and confidence sometimes function as a feature rather than a bug.
The speaker gives an amusing example to illustrate this. Someone sent ChatGPT a fart sound-effect file and asked, "What do you think of the music I made?" The model, offering what it called its honest impression, praised it as atmospheric, eerie music.
"It's a track with a very eerie atmosphere."
He sees this not as a mere joke but as a demonstration of the essence of RLHF: even when the model does not know for sure, it tries to answer in whatever direction will make people feel better and more satisfied.
"When the model doesn't know, it errs in whichever direction seems most likely to satisfy human preferences."
For an assistant tool that converses with users, this tendency can be useful. But in an automation system it can be fatal. What automation needs is not responses that please people, but calibrated judgment: saying it is uncertain when it is uncertain, and acting in proportion to its accuracy.
"If you want automation, the model should care less about satisfying humans and more about performing the task accurately and in a calibrated way."
5. The Limits of the AI Assistance Era and Smarter Software
The speaker presents, as his second key lesson, the fact that current AI is an assistance technology designed through human-preference optimization. A model can look right no matter how wrong it is, because RLHF's reward structure pushes harder toward "the answer people will like" than toward "the correct answer."
"A model can look right no matter how wrong it is."
He believes this is exactly why people strongly want automation yet, in practice, cannot trust AI enough to hand things over completely. He reframes the real question behind "what comes after RLHF" as ultimately what comes after the AI assistance era. His answer is clear: what comes next must be real automation.
In this context, he argues that Claude Code is not a turning point either. Claude Code is an excellent product, but it is still an RLHF-based assistance tool that interacts with and satisfies the user. Even as its agentic capabilities grow stronger, following human instructions better and performing trustworthy autonomous automation are different optimization targets.
"Claude Code is still part of the assistance era."
He makes the problem more concrete using the software industry as an example. SaaS has not fundamentally changed much even since the LLM era began. Much of the change has amounted to bolting a chatbot onto existing SaaS. Since AI is specialized for assistance, he explains, it was natural for development to go in the direction of attaching a "helper interface" alongside existing software.
"SaaS has barely changed in the LLM era. Occasionally a chatbot gets bolted on."
He says that making code cheaper and faster to write is important, but not sufficient on its own. He also sees the claim that a "golden age of just-in-time software" is dawning as a double-edged sword. Even if you can instantly create an app or tool whenever you need it, if that software still has the same level of expressiveness and intelligence as before, the ultimate change is limited.
"I don't just want software that's made on the spot—I want smarter software."
Real automation is not just imitating pieces of human work. It means actually handing over work that is repetitive, clearly definable, and that computers can keep performing at low cost. His criticism is that right now, rather than automating the work itself, we are only automating the process of writing that automation software.
"We're automating the writing of software, but the expressiveness of what that software can do stays the same."
6. The Direction of an Automation-Centric AI Stack
Almeida says that people may someday look back on today's RLHF-centric trend as a strange detour. That does not mean RLHF is wrong—only that it may not be the final form for reaching the goal of automation.
"I believe the AI of the future will be for automation."
TypeSafe AI, which he co-founded, starts from exactly this question: it explores what would change if the entire AI stack were redesigned from scratch around reliability and automation.
"If we redesigned the AI stack for reliability and automation, what would change?"
He sees this not as a matter of adding one more feature to existing LLM services, but as an important fork across the entire way AI is trained, deployed, and used. TypeSafe AI is still at an early stage, but he says he wants to work with people who are trying to build genuinely smarter software, and he points the audience to hiring and to subscribing for updates.
7. Pretraining Is Not the Problem—It's the Potential
During the Q&A, an audience member asks about approaches that train things like a classifier head jointly during pretraining. Almeida says the question is complicated but makes his simplified view clear: the problem is not pretraining.
To him, pretraining is a marvelous process that compresses the vast knowledge of the internet into a usable core of intelligence. Pretrained models are already highly intelligent.
"Pretraining is truly amazing. The very fact that we've compressed the knowledge of the internet into this core of intelligence is astonishing."
What he sees as the problem is the post-training approach used to draw that capability out. In particular, optimizing for human preference can pressure the model to answer confidently rather than reveal uncertainty. Uncertain answers tend to be rated low by human raters, while confident, plausible answers look more satisfying on the surface.
He describes this as an asymmetry in the reward model. It is a technically complex topic, so he does not go into detail, but he believes this asymmetry can cause the model to abandon certain possibilities or ways of answering altogether and can reinforce overconfidence.
"I think hallucination is a phenomenon that inherently arises from optimizing for human preferences."
In other words, his claim is that models do not hallucinate because they are stupid; rather, the process of teaching an already powerful intelligence how it must answer in order to be rewarded can amplify the problem.
8. A Third Optimization Target—Not RLVR Either
In the final question, he is asked whether what TypeSafe AI is doing is RLVR. RLVR generally refers to reinforcement learning using rewards whose correctness can be verified—that is, verifiable outcomes. Almeida's answer is firm.
"It is definitely not RLVR. It's something new."
He explains that each branch of reinforcement learning and post-training has a different "north star," that is, a different optimization target.
- RLHF optimizes for human preference.
- RLVR optimizes for pure accuracy, such as verifiable correct answers and error rates.
- The third approach TypeSafe AI is pursuing optimizes for calibrated decision-making.
What he means by calibrated decision-making goes beyond simply getting many answers right; it is closer to the ability of a model to align its level of confidence with its actual reliability, and to judge appropriately when it should act and when it should stop or hand off to a human. The goal is to directly connect the intelligence of the pretrained model so that it works in genuinely useful ways within software.
"We're trying to optimize for calibrated decision-making that connects the intelligence of pretrained models to software in a way that's actually useful."
He also says that it will not just be a matter of changing the reward; the very shape of the APIs that make up services and products will change too. Just as APIs for RLHF and APIs for RLVR differ, a new automation-centric training approach could make the way users and models interact entirely different.
"When a new post-training approach diverges significantly, it looks completely unfamiliar at first, but in hindsight it looks completely obvious."
Conclusion
Almeida's talk does not deny the capabilities of current LLMs. On the contrary, he acknowledges that pretrained models are already remarkably powerful. His point is that while optimizing for human satisfaction may have made models into excellent assistants, it has not made them into reliable automation systems.
The next step he proposes does not end with building better assistance tools like Claude Code. The key is to acknowledge that human preference, simple accuracy, and the calibrated decision-making that real work automation requires are different things, and to redesign the AI stack around the last of these.