Case study · P/02 · AI agents · causal measurement
RevenantA payment-recovery agent that proves what it caused
Most recovery tools report how much they recovered. Revenant reports how much it recovered that would not have recovered anyway, and it is willing to say "no evidence of value" about its own work. Built for the Razorpay AI Buildathon 2026, AI Revenue Recovery track.
- coverage of the true lift over 2,000 runs
- 93.8%
- interval error caught by its own harness
- 38%
- stratified randomised holdout
- 50/50
The problem
Gross recovery is a vanity metric. It counts payments that customers would have retried on their own, so a recovery tool can look valuable while changing nothing. The only honest number is the incremental lift: recovery with the agent minus recovery without it.
What I built
- Real recoveries. The agent diagnoses each failed Razorpay test-mode payment from Razorpay's own error attribution fields, picks a bounded action from an inspectable policy grid, and retries through a real, Playwright-driven checkout.
- A randomised holdout. Treatment and control are assigned by stratified 50/50 randomisation on a synthetic population whose true recovery process is fixed in code before any experiment runs. The lift is reported with a stratified bootstrap interval.
- A calibration harness. The estimator is rerun 2,000 times against that frozen world to measure how often its 95% interval actually contains the truth.
- A tamper-proof trail. Every agent action goes into a hash-chained, append-only audit log that PostgreSQL itself refuses to modify.
Keeping the model on a leash
The AI never decides whether money moves. A deterministic policy chooses the recovery action. The LLM only maps unfamiliar decline reasons to a broad failure class, and trusted code resolves that class to a specific action, so a prompt injection cannot target a specific retry. This closed a real injection path found in adversarial testing.
A guardrail layer can veto the agent and every veto is recorded as evidence. Terminal declines are never retried, attempts are capped, and a circuit breaker halts a batch on genuine execution faults.
Results
- 93.8% coverage of a known true lift over 2,000 simulated runs, reported as slightly anti-conservative rather than tuned to look clean.
- The harness caught its own bootstrap interval being 38% too wide, which was then corrected on the record.
- When the primary estimate straddles zero, the system reports no evidence of value instead of a flattering number.
Stack
- Python
- FastAPI
- TypeScript
- PostgreSQL
- Playwright
- Razorpay API
- LLM tooling