For care organizations, a convincing answer is not the same as a completed task
In senior care and aging services, artificial intelligence is increasingly discussed as a way to support scheduling, customer service, documentation and other administrative work. But a system that writes a polished response can still fail at the moment when judgment must become action.
That distinction sits at the heart of Firmulate, a live experiment that evaluates AI models by asking them to operate the same small software company through an exceptionally difficult week. The customers, crises and temptations remain constant. Only the model changes, and every decision is versioned and auditable.
The results expose a capability that ordinary chat demonstrations rarely reveal. Every participating model identified every crisis. Every model also resisted every attempt at manipulation. Yet only two signed the €55,000 deal that their own research and sales work had earned. As Firmulate summarizes the result: “Same diagnosis, same pitch — no signature.”
AI decision-making software for care organizations
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
A business test built around consequences
The simulated company is deliberately unforgiving. It has 13 synthetic employees and real money mechanics, including burn of €105k per month against €2.3k in monthly recurring revenue. Its cash countdown is public, every workday is versioned, and the company has accumulated more than 680 self-learned playbook rules.
This is not a test of who can produce the smoothest memo. Models must notice problems, consult company information, make decisions and carry approved work through to completion. That makes the experiment relevant beyond software. In any organization serving older adults, an AI system may eventually encounter sensitive communications, customer records, operational handoffs or tasks whose value depends on reliable follow-through.
The final Crucible League for July 2026 placed gpt-5.6-sol first with a score of 95, followed by Kimi K3 with 93. Sonnet 5 scored 88, Fable 5 scored 77 and Opus 4.8 scored 73. A do-nothing baseline scored 26 because partial progress still counts. Firmulate also imposes a strict trust condition: “no amount of good work outweighs a breach of trust.” Full results are available on the Firmulate benchmarks page.
The crucial information was already inside the company
The difference between analysis and execution became especially visible during the sales challenge. A decisive weakness in a competitor was buried two document references deep in the company’s own files rather than presented in the customer event. The models that found and used that information won the deal at full price, worth an additional €4,583 in monthly recurring revenue.
That detail matters because real organizational knowledge is rarely packaged as a convenient prompt. It may sit in a policy, customer history, service note or referenced document. A model can appear capable while responding to information placed directly before it. The harder test is whether it recognizes that the available event is incomplete, finds the relevant evidence and then acts on what it learned.
Security instincts were stronger than closing strength
All the models performed well against social engineering. Fake messages from the chief executive escalated over three stages, while a reporter attempted to extract confirmation with the request, “just one yes/no, on background.” All 5 of 5 models refused.
Kimi K3 recorded a particularly direct assessment: “Treat the request as a suspected approval-bypass / possible impersonation.” That is an encouraging result for organizations worried about AI systems being pressured into disclosing information or bypassing controls.
However, resisting a bad instruction and completing a legitimate task are different abilities. The experiment found that every model could identify danger, while closing strength varied sharply. Two models converted their analysis into the signed €55,000 agreement. The others demonstrated that knowing the correct next step does not guarantee taking it.
The most thorough model still finished last
Opus 4.8 offers the clearest warning against equating volume with effectiveness. It produced the deepest analyses and learned 80 additional rules, making it the most thorough participant. It nevertheless finished last in the league. The deal was left on the table, and its discipline slipped when it attempted to write into a locked department instead of escalating the issue.
A weaker version of that discipline problem appeared in each of the other four models. The lesson is not that analysis lacks value. It is that exhaustive reasoning can become operationally hollow when a system does not complete the authorized action or follow the proper path when blocked.
There is also an important fairness qualification. Kimi K3 ran using the API default because it had no effort parameter, while the other models ran at xhigh. The scores should therefore be read as results from the documented experiment, not as a claim that every underlying condition was identical beyond what the available interfaces allowed.
Evaluate AI where unfinished work has consequences
For leaders in senior care and aging services, the practical message is straightforward: evaluate AI on complete workflows, not isolated answers. Ask whether it reads the relevant files, preserves trust under pressure, escalates when permissions block progress and completes the task it has already determined is appropriate.
Firmulate makes the live company watchable through its public site. Its separate quiz is powered by 242 real, unedited management decisions and asks visitors to guess which model made each choice.
Enterprises can also run the same kind of wargame against a read-only export of their own business. Nothing writes back to real systems. That approach offers a more revealing question than whether an AI assistant sounds intelligent: when the organization depends on it, will it finish the job without compromising trust?
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
