The question travels, the records stay. When the sites disagree, an investigator plans the next question and a person signs it.
Three labs hold data they cannot pool. CohortLens sends each of them the same approved statistical question and brings back only an effect, its standard error and a count. If the pooled answer shows the sites disagree, an LLM investigator uses a handful of deterministic tools on the pooled numbers and proposes one follow-up from a fixed menu. Nothing runs until a person approves a new contract, and the loop stops when the query budget is spent.
- Personal project
- Emulated three-site federation
- Public GDSC2 cancer cell-line data, not patients
- Flower 1.37
- v0.2.0 tool loop verified offline only
Step through a real run ledger, entry by entry
Every gate decision, tool call, proposal, approval and released aggregate is appended to a JSONL ledger in which each entry hashes the one before it. These are the two sample ledgers committed in docs/sample-run/, embedded unchanged. Your browser recomputes every SHA-256 as you go.
Lineage-enriched split44 entries · 3 rounds · ended BUDGET_EXHAUSTED
What each step does
Eight steps, the last of which loops back to the gate. Each card says what crosses the wire and where the code lives. Press a number to replay that step in the diagram.
A human approves the primary contract
The question is a frozen contract: variant, outcome, covariates, estimator, site roster, release policy, round index and query budget (default: primary + 2 follow-ups). Its canonical JSON is hashed with SHA-256 and a time-limited approval is bound to that digest.
Preflight at every site
Each SuperNode re-validates the approval, checks the request against its own closed allowlist (template, covariates, strata, round), its own budget cap, minimum sample size and group support. It answers READY or a refusal code such as QUERY_BUDGET_EXHAUSTED.
StudyPreflightRequest → PreflightReplyLocal association
Ready sites fit ordinary least squares on their own rows and compute an HC3 robust standard error for the BRAF V600E coefficient. Payloads are capped at 64 KiB.
AssociationReply: β, SE, N and the contract hashValidate and pool
The ServerApp binds each reply to its node, study, query, contract hash and stratum. Missing, invalid, duplicate and mismatched replies become explicit states. With two or more estimates it runs an inverse-variance fixed-effect meta-analysis.
AssociationResult: β, 95% CI, Cochran Q, I²A deterministic gate
Plain code, no model. After the primary: fewer than two estimates stops, I² ≥ 0.50 investigates, otherwise finish. After a follow-up it only checks that the round released an aggregate and that budget remains. It never judges whether another query is worth it.
InvestigationDecisionThe investigator's tool loop
The AgentApp runs a bounded plan → act → observe loop. Six deterministic tools read the aggregate-only evidence, describe the action catalog and check preconditions; submit_proposal validates the proposal and returns any errors to the model.
InvestigationEvidence → InvestigatorRecord (FollowUpProposal + full trace)A human signs a parent-bound contract
The operator sees the proposal, its rationale and the exact digest of the one contract the catalog compiles from it. That contract binds the hash of the parent result and of the proposal. Compilation re-checks preconditions, so a person cannot approve what the catalog forbids. The agent cannot start a run: the operator is the only bridge to the Grid.
AnalysisContract + ApprovalFollow-up round, then back to step 5
The ServerApp checks the contract against the proposal (proposal hash, parent hash, round, template, budget) and dispatches. Leave-one-site-out re-runs the primary model and releases the pooled result with each site omitted; stratification runs steps 2–4 per stratum.
LeaveOneSiteOutResult or StratifiedAssociationResultA bounded tool loop over pooled numbers
The investigator never sees rows or site-level estimates, and cannot run an analysis. It can look at aggregates, read a fixed menu of follow-ups with their disclosure costs, check what is currently allowed, and submit one proposal. The prompt describes the role, constraints and tools; it never contains the answer.
Plan → act → observe
One model request per step. The loop, not the model, executes each tool and enforces every cap.
Also max_output_tokens 2000 and an OpenAI client built with max_retries=0. All caps are overridable in run config. The call sequence shown is the scripted policy's round-2 trace from the sample ledger.
Two tool protocols
The same tools, two ways to call them.
tools; the model emits function_call items and gets function_call_output back.{"tool": "check_action", "arguments": {"action": "LEAVE_ONE_SITE_OUT"}}
Anything else is an invalid turn: the loop answers with INVALID_TURN … feedback and counts it against the cap of 3. A native request rejected with HTTP 400 before any native turn has succeeded restarts the conversation in json, and the trace records the fallback.
Why JSON by default: whether Flower Endeavor supports native tool calling is unconfirmed. The request carries only model, instructions, input, max_output_tokens (and, for native, tools); input is rebuilt from stored messages each turn because the runtime's default provider does not support previous_response_id.
Six tools
Pure functions of the evidence, standard library only (tools.py).
| Tool | Returns |
|---|---|
get_current_evidence | binding fields, next round, budget, threshold, latest round |
get_history | every released round's aggregate summary |
heterogeneity_diagnostics | Q, df, Q p-value, I², H², margin to the 0.50 threshold; leave-one-out ranges |
list_actions | the catalog: what each action tests, releases, discloses, and which contract fields it changes |
check_action | named preconditions passed and failed, resulting round and budget |
submit_proposal | accepted, or the list of schema, binding and precondition errors |
Mistakes come back as tool output
A rejected submission is not a crash; it is information the model can act on.
tools.py){"accepted": false, "errors": ["<field path>: <validation message>", …]}
Submissions are validated against the strict FollowUpProposal schema, the parent-result binding and the action's preconditions. The model reads the errors, fixes them and resubmits, within the caps.
{"action": "LEAVE_ONE_SITE_OUT", "allowed": false,
"failed": ["NOT_ALREADY_RUN"], "would_be_round_index": 2, …}
The action catalog
Closed: three actions per data mode, each compiling deterministically to one contract shape that sites allowlist (catalog.py).
| Action | Sites release | Investigator receives | Preconditions |
|---|---|---|---|
LEAVE_ONE_SITE_OUT | the primary β, SE, N again; no new site-level quantity | range of pooled β and I² over omissions; how many omissions clear the flag; no site identities | BUDGET_REMAININGNOT_ALREADY_RUNPRIMARY_HETEROGENEITY_FLAG_SETLATEST_ROUND_RELEASED_AGGREGATEAT_LEAST_THREE_PRIMARY_ESTIMATES |
STRATIFY_GDSC_LINEAGE | β, SE, N per stratum (SKCM, non-SKCM): new site-level information | per-stratum pooled β, CI, Q, I² | BUDGET_REMAININGNOT_ALREADY_RUNPRIMARY_HETEROGENEITY_FLAG_SETLATEST_ROUND_RELEASED_AGGREGATEAT_LEAST_TWO_PRIMARY_ESTIMATES |
STOP_NO_FOLLOWUP | nothing | nothing | ALWAYS_PERMITTED |
The synthetic data mode swaps the stratification for STRATIFY_SYNTHETIC_CONTEXT. Leave-one-site-out runs as its own approved round so that releasing the omission summaries is a separately authorised act.
Who sees what
The data gets smaller at every hop. The model's whole view is InvestigationEvidence and the tool outputs derived from it, which have no field that could carry a site-level result.
SuperNode
one per site, holds the data
- Cell-line rows and outcomes
- Its own β, SE, N
- Contract, approval, its own allowlist
Coordinator
ServerApp and operator
- Cell-line rows and outcomes
- Each site's β, SE, N and status
- Each omission, by site (LOSO)
- Pooled β, 95% CI, Q, I²
Investigator model
AgentApp, via tool outputs only
- Cell-line rows and outcomes
- Site β, SE, N or site identities
- Pooled and per-stratum β, CI, Q, I²
- LOSO ranges, identity-free
- Binding hashes, budget, catalog
Human operator
the only one who can authorise
- The proposal and its rationale
- The exact digest of the compiled contract
- Approves or declines each round
- Sole bridge between agent and Grid
Rows stay on the node
Only an allowlisted reply leaves a site. Payloads are capped at 64 KiB.
The model gets aggregates
Site-level results are stripped before any model call. Tests check every request sent to a model for site-level fields.
A proposal is not permission
A follow-up runs only under a new contract digest and a fresh, expiring approval, within the budget the primary declared.
A gate made of plain code, with a budget
I² measures how much the sites disagree beyond chance. The gate reads it after the primary round; after a follow-up it only checks that the round produced an aggregate and that budget remains. Whether another query is worth running is the investigator's proposal and the operator's approval.
How it is built
Each follow-up contract binds the hash of the result it follows and of the exact proposal the operator reviewed. This is the chain from the lineage-enriched sample ledger:
Hash-bound contract chain
Every analysis is a frozen Pydantic model (extra="forbid", no NaN/Inf), serialised to canonical JSON and identified by its SHA-256. Approvals bind to that digest and expire. Follow-ups carry their round index and the chain's budget.
A closed policy table at each site
SITE_POLICIES is site-local and deliberately does not import the shared catalog. A valid approval for an analysis outside it, or beyond the site's cap of 3 rounds, is refused (QUERY_BUDGET_EXHAUSTED, ROUND_TEMPLATE_MISMATCH).
OLS-HC3 in plain NumPy
Written by hand and matched to statsmodels 0.14.5 at a relative tolerance of 1e-10. The agent bundle has no numerical dependency: its Q p-value is a standard-library chi-square tail, checked against SciPy to 1e-9.
Leave-one-site-out
Pools the network with each site omitted and matches direct recomputation to 1e-12. The operator sees each omission by site; the investigator gets only identity-free ranges and a count of omissions that clear the flag.
Hash-chained ledger
Append-only JSONL; each entry's entry_hash covers all its other fields, including prev_hash. cohortlens verify-ledger rejects edited, removed, inserted or reordered entries. Full released results go to the output directory, not the ledger.
Two bundles, staged separately
A Flower App Bundle holds either an AgentApp or a ServerApp + ClientApp. stage_apps.py copies only what each imports: schemas, catalog, tools, investigator into the agent; schemas, catalog, statistics, gdsc into analytics. A test fails on drift.
Local and SuperGrid backends
run-study drives the loop behind one interface. local runs the real handlers in-process over an in-memory grid; supergrid calls flwr run … --stream and flwr log. The latter is tested only for command construction and log parsing.
Log-framed transport
Each app prints one framed line (COHORTLENS_RESULT_V1 or COHORTLENS_AGENT_RESULT_V2); the operator carries validated artifacts between bundles. Missing, invalid and duplicate replies are recorded; results are COMPLETE, PARTIAL or INSUFFICIENT.
What actually ran
Three different kinds of evidence, kept apart on purpose.
GDSC2 primary
Is BRAF V600E associated with response to PLX-4720 across three SuperNodes? Flower 1.37.0, 2026-09-16.
Pooled β in ln µM. Lower LN_IC50 means greater measured sensitivity, so the negative coefficient is in the expected direction. The sites agreed, so no investigator call or second query was authorised.
Synthetic full loop
Three sites of 1,500 generated rows, with a hidden context variable that changes the effect.
Heterogeneity disappears within each stratum, consistent with the generator. Two failed live probes are part of the record: one model reply had an unapproved JSON shape and validation stopped the run before any analytics.
What is verified, and what is not
110 offline tests against fake grids and fake model clients, about 3 seconds.
- ✓Both tool protocols with fake models, including malformed turns, cap exhaustion, repair after a rejected submission, and protocol fallback
- ✓No request to a model contains a site-level field
- ✓Site refusals, including over-budget contracts with valid approvals
- ✓The orchestrated loop end to end; ledger tamper detection
- ✗Not yet run against Flower Endeavor or any real model; Endeavor's native tool-calling support is unconfirmed
- ✗
run-study --backend supergridnot executed; the investigate–approve–follow-up loop has not run on GDSC2 data on SuperGrid
GDSC2 numbers from the replayed runs
Local backend, no Flower runtime, no language model. Lineage-enriched-v1 is a deliberately non-IID split built to exercise the loop.
Two different single omissions each clear the flag, so no one site alone explains the disagreement. Within SKCM the sites agree and the association is stronger; within non-SKCM they still disagree. The hash-split primary reproduces the live SuperGrid primary exactly.
What this is not
The project demonstrates a governed workflow. It makes no scientific or privacy claim beyond what is listed here.
About the federation
- The federation is emulated. Three partitions of one public dataset on one development host, not independent laboratories.
- No secure aggregation or differential privacy. The coordinator receives site summaries, and result frames are visible to anyone who can read run logs.
- Validation is not attestation. Hashes, approvals and reply checks catch protocol mistakes; they cannot prove a site used honest records or calculations.
- The budget is per contract. Sites keep no cross-run memory, so re-approving a new primary starts a new chain. The ledger records it; nothing prevents it.
- The site roster is a label, not an enforced list of node identities.
About the science and the agent
- Cell lines, not patients. Associations here are not clinical findings.
- Three sites is very few. Heterogeneity estimates are unstable, leave-one-site-out pools two, and the LOSO range plus the full pooled estimate are close to site-level information even without identities.
- The lineage-enriched split is artificial. It is deliberately non-IID to exercise the loop, not an estimate of real between-laboratory heterogeneity.
- Follow-ups are exploratory. They reuse the same records; they are not replication.
- The tool loop has not met a real model yet. The sample runs use a scripted policy; the loop has not run against Flower Endeavor.