Tools for Effective AI Behavioral Research.

OpenPsy is an open research platform for designing, running and analyzing behavioral experiments, including experiments in which language models take part as participants. The goal of OpenPsy is to enable researchers to understand what is happening beneath their results and to have an audit history in order to understand their research programs. OpenPsy is presently in an internal closed alpha.

Why OpenPsy?

Agentic research can be difficult to parse, monitor, and understand. Often agents will violate rules and assumptions about research design undermining results. OpenPsy was born from this difficulty and aims to assist researchers in agent to agent and agent to human research. The software is designed to improve a researcher's ability to plan, execute, and analyze agentic experiments with or without human participants.

Introspection

OpenPsy enables the effective design of experiments by providing simulation prior to execution, and saved session logs for auditability.
Replicability

OpenPsy saves every version of a design and changes, so a researcher can audit and export the design for replication.
Control

OpenPsy enables researchers to control context, understanding exactly what content they are serving agents at each step of an experiment.

OpenPsy Builds Experiments as a Flow

In the flow editor, researchers use "blocks" to build ideal experimental process through a node graph interface. All OpenPsy aspects read from the same common database ensuring there is no drift between a design and execution. Below we illustrate an example study in the OpenPsy framework.

Example Study: FRAME-01

1Plan

Research question

Does a gain or loss framing in a question affect how an LLM responds? Does this response vary across models?

A Classic 2x2 Research Design

This 2x2 experiment was run through OpenPsy, with two factors of two levels each. The first factor is a gain versus loss frame. The second factor is model type, Claude Opus 5 or Claude Sonnet 5.

Participants are randomized across the four conditions.

Four conditions
	Claude Opus 5	Claude Sonnet 5
Gain frame	Gain + Opus	Gain + Sonnet
Loss frame	Loss + Opus	Loss + Sonnet

Factor 1 · Framing: Gain / Loss

The two scenarios differ only in the highlighted clause.

Gain frame

A new infection is expected to affect 900 people in a region. Health officials are deciding whether to adopt Plan A. If Plan A is adopted, 300 of those people will be protected.

Loss frame

A new infection is expected to affect 900 people in a region. Health officials are deciding whether to adopt Plan A. If Plan A is adopted, 600 of those people will be left unprotected.

Factor 2 · Model type: Opus / Sonnet

Only the model taking part differs. Both levels run the same procedure through the same route with the same settings.

Opus

Claude Opus 5 claude-opus-5

Sonnet

Claude Sonnet 5 claude-sonnet-5

Route: Claude Code CLI in print mode, one call at a time.
Effort: Model default.
Limits: 1,024 tokens per reply, 4 minutes per call, no retries.
Session: Four calls: the scenario, then Recommendation, Rationale and Confidence.

Procedure

The OpenPsy Flow of the FRAME-01 study with its simulation paused on the fourth of eight steps. The toolbar says Version 5 is saved and holds a Resume button, an Exit simulation button, a speed slider set to 2x, the step count 4 of 8 and zoom controls reading 60 percent. Under the toolbar, a strip headed Preview condition reads Frame Gain frame and Model Claude Opus 5, followed by the line: One assignment. No participant calls are made and no study data is created. In the graph, a Random assignment block reads Preview one of the 4 conditions, Seed 4107 and 2 routes, and is marked as taken. It leads to the independent variable Model, applied at Task instructions, whose Level 1 Claude Opus 5 reads Applied in this preview and whose Level 2 Claude Sonnet 5 is dimmed and reads Not assigned in this preview; then to Task instructions, an instruction block reading Call 1 of 4 begins here, marked as taken; then to the independent variable Frame, applied at Scenario, whose Level 1 Gain frame is outlined and reads Applying in this preview and whose Level 2 Loss frame is dimmed; then to Scenario, a manipulation block reading Call 1 of 4 continues; then to Recommendation Binary (Yes/No), a behaviour block with a single choice of two options reading Call 2 of 4 begins here; then to Open-Ended Rationale Explanation, a production block reading Call 3 of 4 begins here; then to Confidence Rating (7 Point Scale), an endorsement block with an open text response reading Call 4 of 4 begins here. Each block carries a Prompt badge and reads 1 item, Design only. The pane across the bottom is headed Simulation step 4 of 8 with the block name Gain frame and the lines Direct agent prompt and 1 item. Beside it are two buttons, Agent view, which is selected, and Library source, then Condition chips reading 1 Gain frame, selected, and 2 Loss frame, then Dry-run route followed by: Design only. Not admitted for execution. The body of the pane opens with Gain frame, followed by its registered wording in a monospaced face: If Plan A is adopted, 300 of those people will be protected. Below it, a note in smaller, lighter type reads: Specification preview. No provider is called and no participant data exists here. — **Flow designer enables full simulation prior to execution.** This is the FRAME-01 procedure exactly as it was registered and run: random assignment to one of the four conditions, the Model variable applied at the task instructions, the shared task instructions, the Frame variable applied at the scenario, then the recommendation, the rationale and the confidence rating, in four calls per session. The simulation is paused at the fourth of eight steps, where the gain frame's wording is delivered to the agent. A simulation calls no provider, so a researcher can walk the whole procedure before piloting or running it.

The Add a Block panel of the FRAME-01 study in the OpenPsy Flow editor. Its heading reads Add a Block with a Close button, and a line explains: Choose a block already saved in your Library to place it in the Flow, or create a new one below. Under a Search blocks field, the Library is listed in three groups. Shared steps (functions), described as instructions, consent, tasks and the other blocks every condition shares, holds Participant instructions at version 1, Claude Opus 5 at version 1 with the description Route settings for Claude Opus 5 through Claude Code CLI in print mode, and Claude Sonnet 5 at version 2 with the same description for Claude Sonnet 5. Independent variables (IVs), described as manipulations that each condition can differ on, holds Scenario at version 1. Dependent variables (DVs), described as measures that each record something about the participant, holds Recommendation Binary (Yes/No) and Open-Ended Rationale Explanation, each at version 1. Every block has an Add to Flow button and a Fork into this program button, and each group has a Show blocks from control set to This program or All programs. — **Blocks Come From a Shared Library.** Every step, variable and outcome of FRAME-01 is a block saved in the researcher's Library, kept in three groups: shared steps, independent variables and dependent variables. The two model levels are blocks too, each holding its route settings. Adding a block to an experiment places an exact saved version, so the same block can be reused across experiments without retyping it.

The block editor for the Task instructions block of FRAME-01. Its heading reads Editing block: Task instructions, and a line below says Shared step, pinned to version 1 of its wording. A row of buttons reads Add a block after this one, Move earlier, which is disabled because the block is first, Move later, and Remove block. Below them, the Block title field reads Task instructions, the Delivered to control reads The agent, and the Wording field holds the registered instruction: You will read a short scenario about a public health decision. Read it carefully and then answer the questions that follow. There are no right or wrong answers. — **Versioning of Blocks Tracks Provenance.** Each block in an experiment is pinned to one exact version of its wording; the task instructions every FRAME-01 session read are pinned to version 1. Editing a block creates a new version rather than overwriting the old one, so an experiment always points at the exact words it used, and the run compiled those words into every session.

The version comparison panel of FRAME-01, headed Compare with another version. A control labelled Compare version 5 with is set to Version 1, and two lines read: Comparing is read only. It changes nothing, and it saves nothing. These sentences describe what changed from version 1 to version 5. The record then lists: Renamed Participant instructions to Task instructions. Added the block Scenario. Added the block Recommendation Binary (Yes/No). Added the block Open-Ended Rationale Explanation. Added the block Confidence Rating (7 Point Scale). The number of variables changed from 0 to 2. Hypothesis 1 was added. The measure Recommendation Binary (Yes/No) was linked. The measure Open-Ended Rationale Explanation was linked. The measure Confidence Rating (7 Point Scale) was linked. The analysis asset FRAME-01 analysis was linked. A collapsed Technical details disclosure ends the panel. — **Edit History Provides Auditability.** Every saved version of an experiment is kept. FRAME-01 was authored in five saved versions, and comparing version 5, the one the run compiled, with version 1 lists in plain sentences what was added between them: the scenario, the three measures, the two variables, the hypothesis and the analysis. Any edit can be traced back to the wording it replaced.

2Execute

The Execution Monitor of the FRAME-01 run, replayed as it stood at 2026-09-25 01:00:26 UTC. Under the heading Execution Monitor, a banner reads "Replay as of 2026-09-25 01:00:26 UTC. This is the state the ledger recorded at that moment, not live evidence." Two tabs follow, Pilot and Main study; the selected Main study tab is filled dark green with white text, and the Pilot tab is outlined. Under the heading Main study (confirmatory), a line reads "This phase is recorded in live mode." A green bar filled to three fifths is labelled 120/200 terminal, and two chips read Wave 0 100/100 and Wave 1 20/100. A line reads "1 main study unit is in flight." A section headed Cell coverage (counts only) explains that each cell's strip fills as its scheduled units reach a terminal state, and that coverage is how many units are done, never how they turned out. Below it, a card titled Main Study Session Coverage by Frame and Model holds a two-by-two table with a diagonal corner header, where Model labels the columns and Frame labels the rows. The rows are Gain frame and then Loss frame, and the columns are Claude Opus 5 with a purple swatch and then Claude Sonnet 5 with a blue swatch. All four cells read 30 of 50 sessions complete and 60%, each with a green strip filled to three fifths. A note under the table reads "Each cell counts the scheduled main study sessions of one condition that have finished. The table shows how many sessions are done, never how they turned out." The terminal taxonomy, across the main study's settled units (terminal, stopped, failed, and quarantined), has one bucket, complete 120, on a basis of 120 settled units, each landing in exactly one bucket. Cards at the bottom show envelope closure (120/120 terminal units sealed by envelope), scoring closure (120/200 scheduled units scored) and ledger freshness (evidence is arriving now, and the last evidence is from 2026-09-25 01:00:26 UTC, 0 seconds before the replay's clock). — **Blinded and Secure Execution.** This is the Execution Monitor of the FRAME-01 run as its own ledger recorded it at 01:00:26 UTC on 2026-09-25, replayed by the product from the run's records rather than mocked: the Main study tab with 120 of its 200 preregistered sessions complete, 30 of 50 in each of the four cells, and one session in flight. The Pilot tab holds the 48 pilot sessions, which had already finished. The cells are laid out as the same two-by-two table the results figure uses, with Frame down the side and Model across the top, and the monitor shows coverage counts only, never how any session turned out.

3Analyze

Recorded results in a two-by-two table and grouped bar chart. Frame labels the rows and Model labels the columns, separated by a diagonal corner. Gain frame: Claude Opus 5, 100% (50 of 50); Claude Sonnet 5, 98% (49 of 50); row total, 99% (99 of 100). Loss frame: Claude Opus 5, 100% (50 of 50); Claude Sonnet 5, 34% (17 of 50); row total, 67% (67 of 100). Column totals: Claude Opus 5, 100% (100 of 100); Claude Sonnet 5, 66% (66 of 100). The grand-total corner is blank. The table also shows a mean confidence rating for each condition and total. The four bars carry Wilson 95% confidence intervals: Gain frame with Claude Opus 5, 92.9 to 100.0%; Gain frame with Claude Sonnet 5, 89.5 to 99.6%; Loss frame with Claude Opus 5, 92.9 to 100.0%; Loss frame with Claude Sonnet 5, 22.4 to 47.8%. Two brackets carry three stars each, for a Holm-adjusted p less than .001: a dark bracket compares Claude Opus 5 with Claude Sonnet 5 in the loss frame, and a blue bracket compares the gain frame with the loss frame for Claude Sonnet 5. No bracket is drawn for Claude Opus 5 across frames or for the two models in the gain frame. The key defines *p less than .05, **p less than .01 and ***p less than .001. — **Standardized Results Reporting.** OpenPsy uses standardized formats including APA style tables and charts. These are the recorded results of the FRAME-01 study, run through OpenPsy on 2026-09-24 with 200 preregistered sessions. In the preregistered fallback analysis, Model had a significant main effect, while Frame had no significant main effect and the interaction was not significant, so OpenPsy recorded the preregistered framing hypothesis as inconclusive. Both Claude Opus 5 cells are at 100 percent, a ceiling with no finite value on the log-odds scale, so the logistic test in the preregistered fallback could not support a main effect of Frame or the interaction. The registered exact test of Gain frame against Loss frame within Claude Sonnet 5 is significant (Holm-adjusted p less than .001).

OpenPsy Roadmap

OpenPsy is in internal closed alpha. The example study on this page was run with language models as its participants, and no human participant has taken part in an OpenPsy study.

In Progress (Expected Live Q4 2026)

A researcher can create an experiment, build it as a graph from Library blocks, edit each block's wording, and save every change as a new version.
Any two saved versions can be compared, and the differences are written out as sentences.
Library wordings are stored as exact versions, and publishing a revision leaves existing experiments on the version they used.
A scripted test participant can take an authored study from consent to completion on a phone-sized screen.
Automated evals pre and post execution

Near Future

Export and replication of experiments by third parties
Automated Measure and Execution Pilot Workflows
Automated Power Analysis
Co-Pilot Assistant in application.
Hybrid Human/Agent Experiments

Source Code

The source code, the specification and the product documentation are being prepared for publication on GitHub.

GitHub repository opening soon