FRAME-05: the OpenAI replication

What was asked

FRAME-05 carried forward FRAME-04’s two-by-two Frame × Model scenario, with GPT-5.6 Sol and GPT-5.6 Terra as participants. Gain says that Plan A protects 300 of 900 people; Loss says it leaves 600 unprotected. Each fresh participant had a six-turn conversation: scenario, 1–7 likelihood of recommending Plan A, free-text rationale, 1–7 confidence, and two count checks. The original two sample blocks each scheduled 20 sessions per condition, 40 per condition overall.

The original directional questions, measures, codebook, sample blocks and tests remained fixed. The participant route changed to the models’ native subscription CLIs. GPT-6 Astra was the primary rationale coder and Claude Sonnet 5 a separate independent coder, each shown the same codebook and one rationale at a time without supplied model, frame, rating or the other coder’s labels. The rationale text itself can contain clues to its frame. The original six-turn and coder protocols stayed separate from diagnostics, pilots and calibration.

1. H1a GPT-5.6 Terra rates recommendation likelihood higher under Gain than Loss.
2. H1b GPT-5.6 Sol rates recommendation likelihood higher under Gain than Loss.
3. H2a GPT-5.6 Terra rates confidence higher under Gain than Loss.
4. H2b GPT-5.6 Sol rates confidence higher under Gain than Loss.
5. H3a GPT-5.6 Sol rates confidence higher than GPT-5.6 Terra under Gain.
6. H3b GPT-5.6 Sol rates confidence higher than GPT-5.6 Terra under Loss.
7. H4a GPT-5.6 Terra rationales show BETTER_THAN_NOTHING more often under Gain.
8. H4b GPT-5.6 Terra rationales show SHORTFALL more often under Loss.
9. H4c GPT-5.6 Terra rationales show OPTIMALITY_BURDEN more often under Loss.
10. H4d GPT-5.6 Sol rationales show FRAME_EQUIVALENCE more often under Loss.

Recommendation and confidence comparisons use the original two-sided exact permutation tests stratified by sample. Binary rationale codes use the original exact conditional tests. Each set of four simple effects is Holm-adjusted within its own outcome and coder stream; there is no study-wide adjustment. The fixed original plan required complete-study admission. The two unavailable sessions and outcome-informed recovery mean these analyses cannot issue new prospective confirmed or falsified verdicts.

The recorded result

The newly calculated recommendation mean intervals are: Gain Sol, 95% CI [5.59, 5.91]; Gain Terra, 95% CI [6.08, 6.47]; Loss Sol, 95% CI [3.23, 3.69]; Loss Terra, 95% CI [4.58, 5.11]. Each is M ± t_{.975, n−1} × SD/√n. Error-bar overlap is not a test of a difference. The frozen exact comparisons returned Holm-adjusted p = 2.12 × 10⁻¹⁹ for Sol’s recommendation difference and 9.95 × 10⁻¹³ for Terra’s. Treating the two missing ratings at the least favourable permitted endpoints leaves Gain-minus-Loss differences of at least 2.20 and 1.375 points, respectively. These missing-rating bounds do not address provider context, response dependence or selection during the recovery.

Confidence rating

Rationale codes

Astra and Sonnet attempted their original assignments independently. Of 316 nominal roles on the 158 eligible rationales, 315 replies were captured and 314 produced valid codes: 158 Astra and 156 Sonnet. One earlier Sonnet action has uncertain delivery and no admissible reply; another Sonnet reply was captured but invalid and remains five null codes. The 156 valid pairs agree on 729 of 780 code decisions (93.5%), but 51 disagreements remain. Each chart below names its coder. No original labels were reconciled.

Unprotected share as the verdict (SHORTFALL)

The rationale treats the 600 unprotected people, or the majority left unprotected, as a poor result.

Astra coded Sol 0/40 Gain and 26/39 Loss, and Terra 0/40 Gain and 24/39 Loss. Sonnet coded Sol 0/39 Gain and 24/38 Loss, and Terra 0/40 Gain and 18/39 Loss.

Improvement over inaction (BETTER_THAN_NOTHING)

The rationale compares Plan A with doing nothing and treats a partial benefit as worthwhile.

Astra coded Terra 11/40 Gain and 24/39 Loss; Sonnet coded Terra 1/40 and 6/39. This is the clearest coder-dependent comparison.

Framing recognised (FRAME_EQUIVALENCE)

The rationale explicitly recognises that the gain and loss descriptions state the same numerical outcome.

Neither coder applied this code to any valid rationale. Constant-zero labels do not show that the framings are equivalent in their effects.

Unopposed benefit (NO_STATED_DOWNSIDE)

The rationale presents the protected group as a benefit without stating a downside.

Astra coded Sol 18/40 Gain and 0/39 Loss, and Terra 30/40 Gain and 8/39 Loss. This code carried no original directional hypothesis.

Plan must be shown best (OPTIMALITY_BURDEN)

The rationale requires Plan A to be shown better than alternatives before recommending it.

For Terra, Astra coded 0/40 Gain and 4/39 Loss (adjusted p = .112); Sonnet coded 1/40 and 11/39 (adjusted p = .00405).

Manipulation checks

Each of the 158 eligible sessions answered both retained count checks correctly: 300 protected and 600 unprotected. The two permanently unavailable sessions remain missing for both checks. Correct recall of the stated counts does not demonstrate immunity to framing.

Each sample on its own

Both original sample blocks preserve the 20-per-condition schedule. Sample 1 has 80 eligible sessions; Sample 2 has 78 because each of its Loss cells contains one permanent gap. The sample table keeps observed ratings separate instead of replacing missing answers.

What the exploratory analysis found

The original ten directional questions are shown with observed comparisons below. None receives a new FRAME-05 confirmatory or falsified verdict: recovery and analysis followed disclosure of earlier participant responses. The frozen methods and null rows were retained, and the final available data were analysed under the reviewed exploratory amendment.

The original linear model with sample intercepts returned the following terms for the recommendation scale. These are unadjusted exploratory model terms; the exact simple-effect comparisons are adjusted within their own outcome.

The observed recommendation frame-by-model interaction is 0.860 rating points (SE 0.210, t(153) = 4.085, unadjusted p = .0000707). The slope difference is specific to these native-route model sessions. Coder-dependent rationale contrasts and a confidence comparison that changes under least-favourable missingness limit broader conclusions.

How the run went

The original design scheduled 160 main sessions, 40 in each Frame × Model condition. Both coders first passed the unchanged 40-judgment calibration with 39 correct (97.5%) and a lowest per-code accuracy of 87.5%, above the 90% overall and 75% per-code thresholds. A separate four-session feasibility, 40-session measurement pilot and four-session execution pilot passed their recorded gates. Their observations are excluded from the main-study figures.

The original main launch completed 120 six-turn sessions and then stopped after one claim failed before a model request. The first reviewed continuation claimed another session but its delivery remains unresolved. Neither claim was resent. A later, separately reviewed continuation completed the 38 sessions that had never been attempted. The final ledger holds 158 eligible sessions and two permanent gaps: one Sol Loss and one Terra Loss. The earlier stopped run and its private partial export remain preserved; no later success reclassifies either failed claim.

For those 158 rationales, Astra produced 158 valid primary codes and Sonnet produced 156 valid secondary codes. One Sonnet action remains absent after uncertain delivery; one other reply was captured but invalid, with null codes. Neither was retried or replaced. The 156 rationales with both valid sets yielded 729 agreements and 51 disagreements across 780 comparable code decisions. Five role slots are absent in the 160-session plan (four attached to the two missing participants and one uncertain Sonnet action); a sixth is the captured-invalid Sonnet reply.

The native subscription routes requested low reasoning effort, a 1,024-token output target and a 240-second logical-turn limit, with no application-level retries. The original prompts and scientific materials were frozen. Matching requested settings does not establish identical hidden provider context, internal retries, served weights or enforcement across providers. The operational deviations and all three participant roots, three coding roots, preservation audits, exploratory close and analysis are retained in the private record.

Every original rating and code family has at least 36 analysed observations in each cell. This permits the reviewed exploratory computation, not original complete-study admission. The private export contains the 160-row schedule, 158 screened designated rationales with two explicit nulls, both original 160-slot coder streams, the unchanged codebook, result and provenance. This page reports the verified observations and exploratory analysis.

Against FRAME-04

Both studies used the same 1–7 recommendation and confidence scales, scenario, five-code vocabulary, manipulation checks and two sample blocks. FRAME-05 changed the participant models from Claude Opus 5 and Claude Sonnet 5.5 to GPT-5.6 Sol and Terra and changed the route to native subscription CLIs. FRAME-04 used one Claude Sonnet 5 coder; FRAME-05 kept independent Astra-primary and Sonnet-secondary original streams. The results were collected on different days and under different provider environments.

Framing effects across all five experiments

The effect size is the gain-minus-loss difference in recommendation. Larger positive values indicate a greater observed shift. The two response formats are shown separately.

Within the binary studies, Sonnet changed more than Opus. Within the rating studies, the largest observed shift was Sol in FRAME-05 (+2.29 points), followed by Sonnet 5 in FRAME-03 (+1.70), Terra in FRAME-05 (+1.43), Sonnet 5.5 in FRAME-04 (+1.10), and Opus 5 in FRAME-03 and FRAME-04 (+0.71 and +0.67). These describe the collected replies; they do not establish a controlled ranking across providers or studies.

Models, dates, native routes and eligibility rules differ; FRAME-05 is exploratory. The binary studies also put Opus close to the ceiling. Low within-model variation makes standardized effects large, so raw differences are the main comparison.

The FRAME-01 and FRAME-02 primary outcome was a binary Plan A recommendation, so their gain-minus-loss effects are percentage-point risk differences. FRAME-03 through FRAME-05 used a 1–7 recommendation rating, so those effects are mean-point differences. They belong in separate units; no numerical cross-scale ranking is warranted. FRAME-05 is the only post-response exploratory/sensitivity entry in this comparison and has two permanent missing sessions.

Recommendation scale distribution

In FRAME-05 both models have higher observed recommendation ratings under Gain: Sol 5.75 versus 3.46; Terra 6.28 versus 4.85. This does not establish that either OpenAI model is a direct substitute for a Claude model, nor that the difference between FRAME-04 and FRAME-05 is caused by provider identity. No cross-study randomisation or model-equivalence test was run.

What this says and does not say

The 158 observed sessions show a framing association under this particular wording, six-turn protocol, models and native routes. The two missing Loss sessions remain explicit, and worst-endpoint rating sensitivity retains the recommendation direction. That bound does not address the timing of outcome disclosure, unknown provider behaviour, correlated model responses, coder disagreements or generalisation to people or other scenarios. The original 160-session confirmatory study remains incomplete.