IdeaBox Monthly Benchmark Report

Run date: 2026-10-15T08:41:23.497842

Results by model

claude-haiku-5.5 (claude-haiku-5-5)

Benchmark	Score	Rate
GAIA L1	30/53	56.6%
τ-bench retail	61/115 (1 errored)	53.5%
τ-bench airline	20/50 (1 errored)	40.8%

gpt-oss-120b (openai.gpt-oss-120b-1:0)

Benchmark	Score	Rate
GAIA L1	15/53 (2 errored — 2 permanently unsupported by this model, won't be retried)	29.4%
τ-bench retail	15/115	13.0%
τ-bench airline	12/50	24.0%

Cross-model comparison

Benchmark	claude-haiku-5.5	gpt-oss-120b
GAIA L1	56.6%	29.4%
τ-bench airline	40.8%	24.0%
τ-bench retail	53.5%	13.0%

Month-over-month trend

Date	Model	gaia_rate	tau_bench_airline_rate	tau_bench_retail_rate
2026-10-06	claude-haiku-4.5	0.0%	—	—
2026-10-06	gpt-oss-120b	0.0%	—	—
2026-10-08	claude-haiku-4.5	56.6%	—	—
2026-10-08	gpt-oss-120b	29.4%	31.9%	8.7%
2026-10-08	claude-haiku-5.5	47.2%	46.9%	48.6%
2026-10-12	claude-haiku-5.5	56.6%	41.7%	—
2026-10-12	gpt-oss-120b	29.4%	36.0%	—
2026-10-14	claude-haiku-5.5	56.6%	40.8%	53.5%
2026-10-14	gpt-oss-120b	29.4%	24.0%	13.0%

Top failure modes this run

claude-haiku-5.5 — top failure modes

retail: placeholder/fabricated value in tool call: 50 occurrences
retail: hit max turns without finishing: 24 occurrences
airline: hit max turns without finishing: 19 occurrences

gpt-oss-120b — top failure modes

retail: placeholder/fabricated value in tool call: 32 occurrences
airline: hit max turns without finishing: 3 occurrences
airline: placeholder/fabricated value in tool call: 1 occurrences

Known issues this run

Permanently unsupported tasks

gpt-oss-120b: 2 task(s) permanently unsupported (e.g. image input on a text-only model) — not retried further

Job interruptions this run (3 total — 3/3 were runner/pod shutdowns or network errors, not application bugs)

RUNNER_SHUTDOWN: 3