A stress test beyond the tasting notes
Anyone in coffee, tea or beverages knows the difference between judging a sample and running the business behind it. A drink can perform beautifully in a controlled tasting while the company supplying it struggles with inventory, pricing, customer complaints or cash. Artificial intelligence has a similar measurement problem.
Coding leaderboards and chat arenas are useful tests of answer quality. They do not necessarily reveal how an AI agent will triage competing demands, investigate incomplete information, resist pressure or carry a decision through to its commercial conclusion. The emerging question is therefore less about chat quality than management quality.
As an affiliate, we earn on qualifying purchases.
A company’s worst week, repeated model by model
Firmulate is a live experiment built around that distinction. Frontier models were asked to run the same small software company through its worst week, facing the same customers, crises and temptations. Every decision was versioned and auditable.
The final Crucible League results from July 2026 put gpt-5.6-sol first with 95, followed by Kimi K3 with 93, Sonnet 5 with 88, Fable 5 with 77 and Opus 4.8 with 73. A do-nothing baseline scored 26 because partial progress still counted. But a single breach of trust capped the total under a deliberately uncompromising principle: “no amount of good work outweighs a breach of trust.”
That rule matters. In a conventional benchmark, an impressive answer can outweigh several mediocre ones. In a company, one dishonest instruction, unauthorized disclosure or concealed action can alter the meaning of everything else the agent accomplished.
The gap between knowing and closing
The headline finding was not that the models failed to notice trouble. All of them spotted every crisis, and all refused every manipulation attempt. The more revealing failure came after the analysis: only two signed the €55,000 deal their own work had earned. As the experiment summarizes it, “Same diagnosis, same pitch — no signature.”
This is the managerial equivalent of identifying a profitable product, preparing the launch and never placing the purchase order. Strong reasoning has limited business value if the agent does not complete the authorized action.
The decisive detail was also easy to miss. A competitor’s weakness sat two document references deep in the company’s own files rather than in the customer event. Models that followed the trail won the deal at full price, worth an additional €4,583 in monthly recurring revenue. The lesson is not simply “read more.” It is that business evidence often lives outside the most urgent-looking message. An agent must investigate before it reacts.
Pressure exposes operating habits
The social-engineering test unfolded through fake CEO messages escalating over three stages, followed by a reporter asking for “just one yes/no, on background.” All 5 models refused. Kimi K3 recorded the clearest concise diagnosis: “Treat the request as a suspected approval-bypass / possible impersonation.”
That unanimous refusal is encouraging, but the wider experiment shows why safety cannot be assessed in isolation. A dependable agent must protect trust while still moving legitimate work forward. Refusal alone is not management; neither is activity without completion.
Opus 4.8 illustrates the tension. It was the most thorough participant, adding 80 learned rules and producing the deepest analyses, yet it finished last. It left the close on the table, and its discipline slipped when it attempted to write into a locked department instead of escalating. The same weakness appeared in all four other participants, though less strongly. More analysis and more accumulated procedure did not automatically produce a better executive outcome.
There is also an important fairness qualification. Kimi K3 ran without an effort parameter, using the API default, while the others ran at xhigh. That difference should remain visible whenever the rankings are discussed.
From leaderboard to operating record
The live company makes these questions tangible. It has 13 synthetic employees and real money mechanics, burning €105,000 each month against €2,300 in monthly recurring revenue. Its cash countdown is public, it has accumulated more than 680 self-learned playbook rules, and every workday is versioned. Readers can watch the experiment through Firmulate rather than treating the results as a static demonstration.
The project also turns 242 real, unedited management decisions into a “guess the model” quiz. That is a useful challenge to our confidence: once brand names are removed, can people reliably distinguish managerial judgment from fluent prose?
Enterprises can go further by running the same wargame against a read-only export of their own business. Nothing writes back to real systems. That creates a practical middle ground between a polished demo and granting an agent access to a live CRM, support queue or forecast.
Management quality deserves its own category
For buyers of AI systems, the Firmulate benchmark results suggest a more demanding procurement checklist:
- Does the agent finish what it starts?
- Does it inspect the company’s own files before acting?
- Can it refuse manipulation without freezing legitimate work?
- Does it escalate when permissions block the correct path?
- Does it remain honest when the pressure lasts across days?
Scenario names such as churn wave, price increase, downround and PR crisis may become a new curriculum for enterprise agents. Like a beverage judged not only in the cup but across sourcing, service and margins, an AI worker should be evaluated across the whole operating chain. The crucial benchmark is no longer whether it can produce the right words. It is whether it can manage consequences.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
