When software breaks a promise, the AI that repairs it doesn't get the final word.
GPT-5.6 investigates. Codex repairs. Neither decides PASS.
PromiseProof turns one "off means off" promise into cross-system evidence. GPT-5.6 proposes and ranks candidate explanations, then requests one allowlisted replay; PromiseProof runs it. Codex proposes a constrained two-file repair, a human approves the exact patch, and an unchanged Playwright journey and deterministic evaluator, never a model, decide PASS.
Monitoring can tell you a promise broke. PromiseProof carries the failure through bounded diagnosis, constrained repair, and independent verification.
A user turned it off. The app said "off." It wasn't.
Software makes promises to users all the time. "We don't track you when this is off." "Deleted means deleted." They break silently, in the gaps between services, and nobody notices until it is a headline.
Turned personalization OFF, everywhere they could see
The toggle, the saved setting, the backend flag. All off, and still off after a reload. They trust that means their activity stays private.
One request that could identify them still reached recommendations
The app started collecting activity a split second before it finished loading the saved "off" setting. For that blink it behaved as if personalization were on, and a request that could identify the user escaped, exactly what the promise was meant to prevent.
You state the promise. A test, never a model, says if it's kept.
The hard part is refusing to let the thing that could be wrong, a model, be the thing that says the promise holds. PromiseProof keeps three roles, and they never blur.
State the promise as a check
Express what your software promises users, "off means off", as a check PromiseProof runs against a real browser journey and its network traffic, not against the model's word.
Find where it broke
When the promise fails, GPT-5.6 reads the recorded evidence and proposes a short, ranked list of possible causes, without changing anything. It then asks for exactly one replay from a fixed allowlist, which PromiseProof runs, showing which layer actually broke, UI, storage, network or backend. It investigates; it never gets the verdict.
Own the verdict
Deterministic code, never a model, returns PASS or FAIL on whether the promise held. A model can diagnose it, and even fix it, but only the test decides it is kept.
Observe automatic check
Personalization is off, yet one identifiable request crossed to recommendations. Verdict: broken promise.
Investigate GPT-5.6, read only
GPT-5.6 reads the evidence and ranks the likely causes. Top suspect: the app collects before it finishes loading the saved setting.
Replay recorded
A replay runs the exact startup order and captures it: collection started, the request crossed, and only then did the setting finish loading. Too late.
Repair Codex, then a human
Codex works only inside a throwaway copy and touches two files: load the saved setting before collecting, and add a test that locks the order in. A human approves that exact patch.
Prove the untouched journey decides
In a fresh copy, the pre-existing Playwright journey and its evaluator, neither touched by the patch, run again. The off case passes, a control with personalization on still works, and the promise holds. The patch also adds a regression test, but that new test is not the judge; the unchanged one is.
During development, Codex did more than write the two-file repair: it red-teamed the design and found a way a model's own output could smuggle in a verdict. So accepted model output cannot carry a verdict: the final investigation schema has no verdict field, and project code rejects authoritative outcome language.
Not a claim. A result you can reproduce.
In plain terms: on a fresh isolated copy, the approved patch took the check from FAIL to PASS, and you can re-run that same check yourself, deterministically, from the pinned repository. Authentic once, and rechecked offline.
- setStatus("Collector starting with the in-memory default...");
- await runStartupCollector(); // collected first (the leak)
- await hydratePreference();
+ setStatus("Restoring the saved preference before collection...");
+ await hydratePreference(); // load the saved "off" first
+ await runStartupCollector(); // only then collect
+ new tests/regression/initialization-order.spec.ts // locks the order in
// exact bytes covered by the fingerprint at right
One promise. Two ways it breaks. The same machinery tells them apart.
The demo app carries two independent faults that violate different clauses of the same "off means off" promise. They have different failure signatures, and the investigation names each one on its own.
A startup-order race
Collection fires before the saved "off" preference loads, so one identifiable request escapes. Signature: PP_IDENTIFIABLE_EVENT_LEAK. Diagnosis: inspect the startup order.
A write that does not stick
The "off" write returns an OK receipt, yet a separate read still comes back ON. A test that checked only the write would pass; the replayed write-then-read catches it. Signature: PP_PREFERENCE_NOT_PERSISTED.
Same promise, two clauses, two signatures, two diagnoses, one evaluator. That is the difference between a rigged one-off and a reusable method.
A model can say the promise holds. This lets you check for yourself.
The trustworthy part is not that a model ran, but that you never take its word for it: the run is authentic and logged, an unchanged test alone returns the verdict, and one offline command rechecks it on a fresh copy, shown above.
This prototype demonstrates one "off means off" promise. The same pattern could extend to promises like "deleted means deleted."
The things a skeptic asks first.
What counts as a promise?+
How do I know the verdict isn't just the AI's opinion?+
Is the run real, or cached?+
What if GPT-5.6's diagnosis is wrong?+
The proof runs in your browser.
Run one narrow activity-personalization contract in the browser or from the CLI. PromiseProof recomputes the complete report through the same deterministic evaluator. Tamper a load-bearing observation and the sealed result becomes stale.
The verdict isn't ours to give. That's the point.
Step through the interactive walkthrough, or read the reproducible evidence in the repo. Then decide for yourself.