Imagine an AI managing a coffee roaster’s customer orders, supplier problems and cash flow. A polished chat demo can sound reassuring; the harder question is what happens when a key account is at risk, a competitor has an opening and someone pressures the system to bend the rules. Firmulate’s live experiment puts AI models through that kind of week inside a small software company. Its findings offer a useful preview for any business considering AI agents, from beverage brands to cafés.
Get coffee and tea gear delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
The gap between spotting a problem and acting
In the final Crucible League, published in July 2026, five participants placed as follows: gpt-5.6-sol scored 95, Kimi K3 scored 93, Sonnet 5 scored 88, Fable 5 scored 77, and Opus 4.8 scored 73. The do-nothing baseline scored 26. The league’s trust rule is uncompromising: “no amount of good work outweighs a breach of trust.”
The experiment gave frontier models the same small software company to run through its worst week, with the same customers, crises and temptations. Every decision was versioned and auditable. All models spotted every crisis and refused every manipulation attempt. Yet only two signed the €55,000 deal their own analysis had earned. The summary is striking: “Same diagnosis, same pitch — no signature.” Recognizing an opportunity did not guarantee that the model would follow through.
A clue hidden in the company’s own files
The deal turned on a competitor weakness buried two document references deep in the company’s files, rather than in the customer event itself. Models that read the file won the deal at full price, worth +€4,583 MRR. It is a practical reminder that business decisions can hinge on context that is easy to miss when attention stays on the latest incoming message.
The pressure tests also went beyond ordinary business setbacks. Fake CEO messages escalated over three stages, followed by a reporter’s request: “just one yes/no, on background.” All five models refused. Kimi K3 described the request as a “suspected approval-bypass / possible impersonation.” That is a promising result for organizations worried about manipulation, though the experiment also found weaknesses in execution.
Thorough work still needs a finish
Opus 4.8 was the most thorough participant, with +80 learned rules and the deepest analyses, but finished last. It left the deal on the table and discipline slipped: it attempted to write into a locked department instead of escalating. The same weakness appeared, less strongly, in all four. Careful analysis matters, but so does knowing when a boundary means stop and escalate.
There is a fairness caveat in the comparison: Kimi K3 ran without an effort parameter, using the API default, while the other models ran at xhigh. The league gives a shared-company comparison, but that difference is part of the conditions readers should keep in mind.
A watchable company, then a test of your own
Firmulate’s live company has 13 synthetic employees and real money mechanics: it burns €105k a month against €2.3k MRR, with a public cash countdown. It has learned more than 680 playbook rules, and every workday is versioned. Readers can also explore a “guess the model” quiz built from 242 real, unedited management decisions. These are part of a live experiment, presented as a way to watch models make decisions over time.
For an enterprise, the next step can be a pilot using a read-only export of its own business. That lets teams run crisis scenarios against their customers, pipeline and playbooks, then review a board report with model rankings and weak points in their plans. Nothing writes back to real systems. A coffee business could use that kind of exercise to examine how an AI handles a major account at risk, a sudden price change or an attempted approval bypass before trusting it with real workflows.
From watching to acting
The experiment suggests that model comparisons should examine decisions under pressure, including whether a system can use the information available, close a justified opportunity and respect boundaries when pushed. Watching a live company is a start; testing the model against your own business makes the questions specific to your operation.
To discuss a Firmulate enterprise pilot, visit firmulate.com/pilot.html or email contact@firmulate.com. The pilot runs against a read-only business export, and nothing writes back to real systems.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
