OpenAI's Ironclad research exposes why agent evaluation needs rule-by-rule outcomes
OpenAI and contracting software company Ironclad converted specialized professional work into structured research tasks for computer-use agents. Instead of asking whether a contract workflow looks complete, experts specified 11 tasks—such as an NDA process, a procurement approval workflow or a jurisdiction-specific clause—and graded each against 8 to 50 concrete acceptance criteria. The research report illustrates why success depends on preserving business rules throughout a multistep software session. A purchasing process is not correct merely because a form and agreement were generated: spending thresholds must route to Finance, exceptions must reach Security or Legal where required, and requests on either side of a threshold must actually traverse the intended path. The reported comparison used the strongest tested reasoning configurations for each model: a mean rubric score of 55.0% for GPT-6 Astra against 41.6% for GPT-5.6 Sol across those 11 research tasks. Estimated simulated time per attempt declined from 37.0 to 19.2 minutes, but those figures are not observed customer productivity savings. Synthetic practice tasks were derived from filtered public EDGAR contracts, not nonpublic customer contracts. The sample is small and domain-specific; the result should be read as an evaluation-method demonstration, not general proof of autonomous contracting readiness.
OpenAI's Ironclad research exposes why agent evaluation needs rule-by-rule outcomes
Focus: Technical AI Engineering Date: October 6, 2026 Topics: computer-use evaluation, rubric design, business rules, reinforcement learning, human review Evidence: Publisher Authored Availability: Available
Summary: OpenAI and contracting software company Ironclad converted specialized professional work into structured research tasks for computer-use agents. Instead of asking whether a contract workflow looks complete, experts specified 11 tasks—such as an NDA process, a procurement approval workflow or a jurisdiction-specific clause—and graded each against 8 to 50 concrete acceptance criteria. The research report illustrates why success depends on preserving business rules throughout a multistep software session. A purchasing process is not correct merely because a form and agreement were generated: spending thresholds must route to Finance, exceptions must reach Security or Legal where required, and requests on either side of a threshold must actually traverse the intended path. The reported comparison used the strongest tested reasoning configurations for each model: a mean rubric score of 55.0% for GPT-6 Astra against 41.6% for GPT-5.6 Sol across those 11 research tasks. Estimated simulated time per attempt declined from 37.0 to 19.2 minutes, but those figures are not observed customer productivity savings. Synthetic practice tasks were derived from filtered public EDGAR contracts, not nonpublic customer contracts. The sample is small and domain-specific; the result should be read as an evaluation-method demonstration, not general proof of autonomous contracting readiness.
Why it matters: This is a reusable engineering pattern for reliable agents: represent the instruction as explicit decision rules, run tasks in a safe replica of the real business application, score the final state against observable criteria, and separately check that prohibited paths never occurred. Use boundary-value examples around approval thresholds and nonstandard terms. Record failed criteria—not just an aggregate percentage—and require human review of any rule-changing action. The business-rule rubric, rather than model fluency, becomes the release gate.
