The Agent Stack #029 — Wednesday Stack

OpenAI just dropped GPT-5.5 Instant as ChatGPT’s new default model. The company claims it cuts hallucinations “significantly” across law, medicine, and finance whilst keeping the same low latency as GPT-4o.

I’ve been testing it for 72 hours on production workloads. Here’s what actually changed.

The Reality Check

OpenAI’s internal benchmarks show factuality improvements across their eval suites. But internal evals are like dating profiles - technically accurate but missing crucial context.

I ran GPT-5.5 Instant through three real agent tasks that previously caused hallucination headaches:

Legal document analysis: Asked it to extract key clauses from a 40-page commercial lease. GPT-4o would invent clause numbers and misstate terms about 15% of the time. GPT-5.5 Instant got the facts right but still struggled with nuanced interpretation of ambiguous language.

Medical coding assistant: Tasked it with ICD-10 code suggestions from clinical notes. Previously saw fabricated codes roughly 8% of queries. New model dropped that to around 3% but became more conservative - often suggesting “requires human review” instead of taking a swing.

Financial statement parsing: Extracting metrics from 10-K filings. Hallucination rate dropped from ~12% to ~4% on specific numbers. But it’s now overly cautious about calculations, often refusing to compute simple ratios.

The pattern is clear: less wrong, more careful, sometimes too careful.

What Changed Under the Hood

OpenAI hasn’t detailed the architecture changes, but the behaviour suggests enhanced fact-checking layers. The model pauses longer before responding to factual queries - latency is “maintained” but you notice the extra beat.

Response times for complex factual questions increased by roughly 200-400ms in my testing. For simple chat, it’s imperceptible. For agent chains making dozens of calls, it adds up.

The model also flags uncertainty more explicitly. Instead of confidently stating wrong facts, it now says “I’m not certain” or “This may require verification.” Better for accuracy, harder for automated workflows that expect decisive responses.

Agent Integration Gotchas

Three issues hit me immediately:

Confidence calibration: Existing agents built around GPT-4o’s confidence levels need recalibration. The new model is more hesitant, which breaks decision trees that rely on confident outputs.

Error handling: More “I don’t know” responses mean agent flows need better fallback logic. Previously you’d get wrong but confident answers you could validate. Now you get correct uncertainty you need to handle.

Cost implications: The increased deliberation means slightly higher token usage on complex queries. About 8-12% more tokens for factual extraction tasks in my testing.

Quick Hits

• Memoir launched as “Git for AI agent memory” with a Claude Code plugin - interesting approach to persistent context but early days (github.com/zhangfengcdt/memoir)

• CopilotKit raised £21M Series A to help developers deploy app-native AI agents - framework looks solid for embedding agents directly into existing apps

• DarkMatter promises tamper-evident audit trails for agent decisions - crucial for compliance but unclear how it handles distributed agent networks (darkmatterhub.ai)

One Thing to Try

Swap your production agents to GPT-5.5 Instant in a controlled A/B test this week. Don’t change your prompts yet - just measure the difference in factual accuracy versus response confidence. You’ll likely need to adjust your error handling before you see the benefits.

The best agents aren’t the smartest - they’re the ones that know what they don’t know.