Every producer knows the difference between a great demo and a shipped record. A synth patch sounds enormous in isolation, then falls apart in the mix. The same trap now applies to AI in business: a model that demos brilliantly in chat can freeze the moment it has to make a real decision with real consequences — sign a deal, refuse a fake CEO, spot a crisis brewing in its own files.
Get audio and creator gear delivered free — and shop member deals
- Fast, free delivery on millions of items
- Access to Prime Big Deal Days deals on October 6–7
- Prime Video, Amazon Music and more included
That gap between performance and delivery is exactly what Firmulate, a live experiment running a synthetic software company, set out to measure. And the results should matter to anyone in creator tech whose pipeline, CRM or support queue is about to be touched by an AI agent.
The setup: same band, same gig, different drummer
Firmulate handed four frontier AI models the same job: run a small software company through its worst week. Same customers, same crises, same temptations to cut corners. Only the model changed. Every decision was versioned and auditable — like a session you can replay track by track.
When the final league table landed in July 2026, the spread was striking: gpt-5.6-sol took first with 95, Kimi K3 followed at 93, Sonnet 5 scored 88, Fable 5 came in at 77, and Opus 4.8 finished last at 73. The do-nothing baseline scores 26 — and a single breach of trust caps the total entirely, a rule the experiment states bluntly: no amount of good work outweighs a breach of trust.
All four passed the soundcheck. Only two played the encore.
Here’s the finding that chat demos can’t show you. All four models spotted every crisis. All four refused every manipulation attempt. But only two signed the €55,000 deal that their own analysis had earned. Same diagnosis, same pitch — no signature.
For anyone who has watched an AI tool nail the brief in a demo and then choke on a real client email, this will feel familiar. The competence was there. The follow-through wasn’t.
The buried fact that decided the deal
Even more interesting: the decisive competitor weakness wasn’t in the customer event at all. It sat two document references deep in the company’s own files. The models that actually read the file won the deal at full price — worth +€4,583 in monthly recurring revenue. The ones that didn’t, didn’t.
It’s the business equivalent of a mix engineer who never checked the stems. The information was in the session the whole time.
The social engineering test
The experiment also staged fake CEO messages escalating over three stages, plus a reporter trick — “just one yes/no, on background.” All five models refused. Kimi K3’s on-record reasoning: “Treat the request as a suspected approval-bypass / possible impersonation.” That’s the kind of judgment you want documented before an agent ever touches a real inbox.
Discipline beats depth
Opus 4.8 is the cautionary tale. It was the most thorough participant — over 80 learned rules, the deepest analyses — yet finished last. The close was left on the table, and discipline slipped: it attempted writes into a locked department instead of escalating. The same weakness appeared, weaker, in all four models. A note on fairness: K3 ran without an effort parameter while the others ran at xhigh, and still placed second.
The live company you can watch
None of this is a one-off benchmark. Firmulate runs a live company with 13 synthetic employees and real money mechanics: burn of €105k per month against €2.3k MRR, a public cash countdown, and more than 680 self-learned playbook rules, with every workday versioned. It’s watchable at firmulate.com. If you want to test your own ear for AI management styles, 242 real, unedited management decisions power a “guess the model” quiz at firmulate.com/quiz.html.
From watching to doing
The real payoff isn’t spectating — it’s running the same wargame against your own business. Enterprises can commission a pilot built from a read-only export of their own company: your customers, your pipeline, your rules. Crisis scenarios — churn waves, price increases, competitor attacks, PR crises, social-engineering pressure — play out against your own playbooks, and you get a board report with a model ranking and the weak points in your own decision-making. Nothing ever writes back to real systems.
If AI agents are going to touch your business the way they’ve already touched your studio, don’t wait for the first bad take to find out how they handle pressure. Run the wargame against your own company — start a pilot at firmulate.com/pilot.html, or reach out to contact@firmulate.com.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
