IdeaBox Monthly Benchmark Report
Run date: 2026-10-16T09:47:39.586927
Results by model
claude-haiku-5.5 (claude-haiku-5-5)
| Benchmark | Score | Rate |
|---|
| GAIA L1 | 29/53 | 54.7% |
| τ-bench retail | 0/115 (115 errored) | N/A (all errored) |
| τ-bench airline | 0/50 (50 errored) | N/A (all errored) |
gpt-oss-120b (openai.gpt-oss-120b-1:0)
| Benchmark | Score | Rate |
|---|
| GAIA L1 | 13/53 (2 errored — 2 permanently unsupported by this model, won't be retried) | 25.5% |
| τ-bench retail | 0/115 (115 errored) | N/A (all errored) |
| τ-bench airline | 0/50 (50 errored) | N/A (all errored) |
Cross-model comparison
| Benchmark | claude-haiku-5.5 | gpt-oss-120b |
|---|
| GAIA L1 | 54.7% | 25.5% |
| τ-bench airline | N/A (all errored) | N/A (all errored) |
| τ-bench retail | N/A (all errored) | N/A (all errored) |
Month-over-month trend
| Date | Model | gaia_rate | tau_bench_airline_rate | tau_bench_retail_rate |
|---|
| 2026-10-06 | claude-haiku-4.5 | 0.0% | — | — |
| 2026-10-06 | gpt-oss-120b | 0.0% | — | — |
| 2026-10-08 | claude-haiku-4.5 | 56.6% | — | — |
| 2026-10-08 | gpt-oss-120b | 29.4% | 31.9% | 8.7% |
| 2026-10-08 | claude-haiku-5.5 | 47.2% | 46.9% | 48.6% |
| 2026-10-12 | claude-haiku-5.5 | 56.6% | 41.7% | — |
| 2026-10-12 | gpt-oss-120b | 29.4% | 36.0% | — |
| 2026-10-14 | claude-haiku-5.5 | 56.6% | 40.8% | 53.5% |
| 2026-10-14 | gpt-oss-120b | 29.4% | 24.0% | 13.0% |
| 2026-10-16 | claude-haiku-5.5 | 54.7% | N/A (all errored) | N/A (all errored) |
| 2026-10-16 | gpt-oss-120b | 25.5% | N/A (all errored) | N/A (all errored) |
Top failure modes this run
No recurring failure patterns detected.
Known issues this run
Permanently unsupported tasks
- gpt-oss-120b: 2 task(s) permanently unsupported (e.g. image input on a text-only model) — not retried further
Job interruptions this run (7 total)
- HIGH_ERROR_RATE: 6
- UNKNOWN: 1