Walkcast

Solo build: the recording engine, the lab harness, the desktop app and Tauri shell, the Postgres queue with SKIP LOCKED leases, the S3 and Supabase media layer, the 15-provider model gateway, and the architecture documentation.

One engine in packages/core, two apps on top of it: a static marketing site and a Tauri desktop app bundling a Node server. The recorder drives a real browser through sign-in, walks same-origin links breadth-first, and writes narration only from recorded events. The browser pauses after each action for exactly as long as its line takes, so the audio and the screen never drift. Walkcast Lab is the other half: it takes a failed agent run, pins the failure to the step that caused it using before/after application state, classifies the cause from a fixed vocabulary with cited evidence, and replays it as a fixture. Fixtures run as a CI release gate, so a new agent version has to prove it fixed the bug and did not break something else.

An agent's own summary can't be trusted, so verification had to come from the application's records, not from the agent or an LLM judge. On top of that, recordings have to survive a worker dying mid-job without leaking the credentials that were sealed into the run.

The lab reports numbers. Across five injected faults it reproduced the original cause 5/5, and the fixed agent passed 5/5 replay trials where the naive baseline passed 0/5, while the deliberate regression (v2, fixed with read-back removed) dropped to 4/5, which is the harness catching a real regression. It also measures what agents get wrong: 3 of 4 success claims were false successes, meaning the agent reported a task complete while the application's records disagreed. 30 test files and 129 test cases run on node:test, covering secrets, jobs, queue leases, auth scoping, and the classifier.

The agents under test are deterministic scripts, not LLMs, so the numbers in results/ validate the harness (capture, oracle, classifier, fixtures, replay) and say nothing about LLM agent behaviour. The published run uses one seed per fault, so the confidence intervals are wide and the figures are a harness check rather than a benchmark. Fixtures reproduce failures in an application you control; arbitrary third-party production sessions cannot be reproduced exactly. Installers are not code-signed, and redaction targets harness secrets rather than customer PII, so real customer data should not be recorded.

I stopped trusting the agent's report and started checking the app. Comparing the application's state before and after every action turns 'the agent said it worked' into something I can verify, and it points at the step that failed. Background jobs taught me the rest: fenced finalization, atomic tenant limits, and at-least-once execution with idempotent effects were the only way to keep a worker crash from double-charging or losing a recording. Secrets are replaced with {{secret:name}} placeholders before anything touches disk, and restored only in memory during replay.