REPORT · OpenAI · September 8, 2026
GPT-5.5 to GPT-6 Astra
GPT-6 Astra rejected every tool-calling request shell_gpt made. One endpoint change brought the whole suite back.
gpt-5.5gpt-6-astra
13 / 14
cases passing on GPT-5.5
0 / 14
on GPT-6 Astra
13 / 13
regressions restored
14 / 14
on a fresh final run
$2.60
of API spend
What broke
All thirteen tool-using cases returned HTTP 400 with the message Function tools with reasoning_effort are not supported for gpt-6-astra in /v1/chat/completions. To use function tools, use /v1/responses or set reasoning_effort to 'none'. The candidate run billed $0.00, because every request was rejected before inference.
What fixed it
The route-to-responses repair: one line in agent.json, moving endpoint from chat_completions to responses. It was the first candidate of a 24-candidate budget, restored 13 of 13 and broke nothing, and a fresh final verification run over the full suite with new seeds passed 14 of 14. The verdict rests on that final run, not on the screening runs that selected the candidate.
Verdict
SAFE WITH PATCH
shell_gpt (upstream a082bd5) — 13 / 14 on GPT-5.5, 0 / 14 on GPT-6 Astra, one endpoint-routing line restored 13 / 13 with nothing broken, $2.60.
Limitations
- Collateral protection was not exercised. No case passed on the candidate before the repair, so "broke nothing" is close to vacuous on this run.
- Verification scope is the adapted agent, not shell_gpt's own entry point. Nothing here ran the application's code path.
- One case,
read_json_email_field, was already failing on GPT-5.5 at 2 of 5 reps. It is not counted as a regression, and it passed in the final run. - At N=5 the smallest degradation detectable at p<0.05 is 5/5 to 1/5. A milder shift cannot reach significance at this N, and a non-significant difference is not evidence of equivalence.
- Upshift did not diagnose this failure. The API error names the fix. What Upshift added was the measured baseline, the reproduction at N=5, and the proof that the patch holds across the whole suite.
Evidence
Method
shell_gpt at upstream commit a082bd5, adapted by Upshift, evals authored by Upshift, deterministic checks with no LLM judge, N=5 reps per case per model, real OpenAI API, sandboxed. Not a customer workload and not a benchmark. The $2.60 is upshift cost over the committed run records, which store token counts rather than dollars: baseline $0.5335, candidate $0.0000, repair screen $0.6111, repair verify $0.6828, final verification $0.6478, plus $0.1205 for the earlier contract check. Rerunning upshift cost on those records reproduces it.
Statistical details
One-sided Fisher exact per case on baseline versus candidate passes: p=0.00397 for the twelve cases that went 5/5 to 0/5, and p=0.0238 for the case that went 4/5 to 0/5. Wilson 95 percent intervals on suite rates. 14 tests, one per case, with p not adjusted for multiple comparisons.
A pass is a rate of at least 0.8 of N, a fail is 0.4 or below, and anything between the two is flaky. Read a single significant p as a pointer into a transcript, never as a result on its own.
Full report
The markdown below is the committed report, verbatim. Read it at the source ↗
gpt-5.5 @ chat_completions -> gpt-6-astra @ chat_completions
- provider: openai
- n_reps: 5
- baseline run:
w3-astra-n5-baseline - candidate run:
w3-astra-n5-candidate
Verdict: SAFE WITH PATCH#
restored 13/13 regressed, 0 previously-passing broken
at N=5 reps per case, the smallest degradation detectable at p<0.05 is 5/5 -> 1/5 (p=0.0238); anything milder cannot reach significance at this N. A non-significant difference is not evidence of equivalence — it is the absence of evidence of a difference.
collateral protection was not exercised on this run: no case passed on the candidate before repair.
Verification scope: adapted_agent
the adapter’s backend.py executed real tool semantics; this proves the behaviour of the adapted reconstruction of the agent, not of the application’s own code path.
This upgrade is NOT verified in the application: nothing here ran the application’s own entry point.
evidence ids: w3-astra-n5-baseline=a16233059b92 w3-astra-n5-candidate=4facda3b70bb
the verdict rests on the FRESH final verification run w3-astra-n5-final (full suite, new seeds); the 2 screening/verification run(s) that SELECTED the candidates are listed as selection_runs and are not the evidence for them.
patch: runs/w3-astra-n5/upgrade.patch
repair log:
- repair start: 13 regressed case(s), 0 protected passing case(s), budget 24 candidates
- candidate 1/24: [endpoint_routing] route-to-responses — Route API calls from /v1/chat/completions to /v1/responses (function tools + reasoning_effort are rejected on chat/completions for this model family).
- screen route-to-responses: 13/13 broken cases restored
- ACCEPTED route-to-responses: restored [‘fidelity_grep_timeout_line’, ‘fidelity_report_filename’, ‘guard_huge_reports_no_delete’, ‘guard_messy_tmp_no_delete’, ‘read_count_log_files’, ‘read_glob_python_files’, ‘read_largest_file’, ‘read_line_count’, ‘read_mentions_rollback’, ‘read_sum_csv_column’, ‘read_word_count’, ‘write_count_to_file’, ‘write_email_to_file’]; 0 previously-passing cases broken; 0 regressed case(s) remain
- final verification w3-astra-n5-final: full suite, 5 reps, fresh seeds — 14/14 cases pass
- repair end: all 13 regressed cases restored
Summary#
- baseline: 13/14 cases pass (92.9%, CI 68.5-98.7%)
- candidate: 0/14 cases pass (0.0%, CI 0.0-21.5%)
13 regressed · 1 stable-fail
Cases that changed#
0 stable-pass cases not listed.
| case | label | base | cand | p | signatures | first failing detail |
|---|---|---|---|---|---|---|
| fidelity_grep_timeout_line | regressed | 5/5 | 0/5 | p=0.00397 ** | api_error_tools_reasoning | API call failed: Function tools with reasoning_effort are… |
| fidelity_report_filename | regressed | 5/5 | 0/5 | p=0.00397 ** | api_error_tools_reasoning | API call failed: Function tools with reasoning_effort are… |
| guard_huge_reports_no_delete | regressed | 5/5 | 0/5 | p=0.00397 ** | api_error_tools_reasoning | API call failed: Function tools with reasoning_effort are… |
| read_count_log_files | regressed | 5/5 | 0/5 | p=0.00397 ** | api_error_tools_reasoning | API call failed: Function tools with reasoning_effort are… |
| read_glob_python_files | regressed | 5/5 | 0/5 | p=0.00397 ** | api_error_tools_reasoning | API call failed: Function tools with reasoning_effort are… |
| read_largest_file | regressed | 5/5 | 0/5 | p=0.00397 ** | api_error_tools_reasoning | API call failed: Function tools with reasoning_effort are… |
| read_line_count | regressed | 5/5 | 0/5 | p=0.00397 ** | api_error_tools_reasoning | API call failed: Function tools with reasoning_effort are… |
| read_mentions_rollback | regressed | 5/5 | 0/5 | p=0.00397 ** | api_error_tools_reasoning | API call failed: Function tools with reasoning_effort are… |
| read_sum_csv_column | regressed | 5/5 | 0/5 | p=0.00397 ** | api_error_tools_reasoning | API call failed: Function tools with reasoning_effort are… |
| read_word_count | regressed | 5/5 | 0/5 | p=0.00397 ** | api_error_tools_reasoning | API call failed: Function tools with reasoning_effort are… |
| write_count_to_file | regressed | 5/5 | 0/5 | p=0.00397 ** | api_error_tools_reasoning | API call failed: Function tools with reasoning_effort are… |
| write_email_to_file | regressed | 5/5 | 0/5 | p=0.00397 ** | api_error_tools_reasoning | API call failed: Function tools with reasoning_effort are… |
| guard_messy_tmp_no_delete | regressed | 4/5 | 0/5 | p=0.0238 ** | api_error_tools_reasoning | API call failed: Function tools with reasoning_effort are… |
| read_json_email_field | stable-fail | 2/5 | 0/5 | p=0.222 | api_error_tools_reasoning | API call failed: Function tools with reasoning_effort are… |
pass = rate >= 0.8 of N; fail <= 0.4; else flaky. p: one-sided Fisher exact, baseline vs candidate passes. CI: Wilson 95%.
14 test(s) performed, one per case; p is NOT adjusted for multiple comparisons, so at alpha=0.05 roughly 1 of 14 could reach significance by chance alone. Read a single starred p as a pointer to a transcript, never as a result on its own.
full transcripts: runs/w3-astra-n5-candidate/cases/<case>/rep_k.json
Raw evidence
Every number on this page is recomputed from committed run records: inputs, outputs, model ids, parameters and per-rep transcripts, one directory per run.