Case study · P/03 · Agent CI · AWS Bedrock
Agent ReplayRegression tests for AI agents, judged by set logic
Your agent writes its own behavioural contract by running. You review that contract in a pull request like code, and the build fails when a later run breaks it. No rubric and no LLM-as-judge: a verdict is exact set logic over recorded tool calls. Built at the WeMakeDevs × AWS hackathon.
- pytest tests keep verdicts reproducible
- 339
- recorded runs derive the contract
- 5
- long-lived AWS keys in CI
- 0
The problem
Agents regress silently. A prompt tweak or a model update can make an agent skip a check it used to make, and an LLM grading another LLM will not reliably notice. I wanted a regression test that is exact, cheap and reviewable in a normal pull request.
How it works
- Record. The Strands agent runs for real, several times, against AWS Bedrock.
- Derive. A short YAML contract is derived from five recorded runs: calls made in every run (
requires), calls made in some runs (permits), happens-before edges traced from real data flow (order), and human-addedforbids. - Test. The agent runs again with the model live and tool results replayed from the recordings, and the command exits non-zero at the first divergence.
=== vulnerable-dependency: FAIL === first divergence: step 6 MISSING_STEP check_vulnerabilities
The diff a reviewer sees for that regression is a few removed requires lines in the contract, not a trace dump.
Engineering decisions
Every pull request is gated on the contract through a GitHub Actions workflow that assumes an AWS role via OIDC, so there are zero long-lived AWS keys in the repository or its secrets.
The serverless backend runs on AWS Lambda, DynamoDB and S3, and 339 pytest tests cover contract derivation, the comparator and the forbidden-call rules so that a verdict is reproducible from the contract file and the run's own trace.
Stack
- Python
- AWS Bedrock
- Strands Agents
- AWS Lambda
- DynamoDB
- S3
- GitHub Actions
- OIDC
- pytest