The loop

In episode 0 I wrote a spec for a heating controller and hid the checks that grade it. Nothing had been built. This episode is about the first thing that builds it, and about how wrong my first idea of that thing was.

The plan for H0, the first version of the harness, was to own as little as possible. A standard coding agent, Claude Code with Sonnet 5.5, gets the spec and an empty repository and works until it says it's done. I put a cage around it and check what comes out. Anything I'd want to improve later should first show up here as a failure, and I didn't expect to wait long.

##The cage, first§

Before letting an agent loose, I wanted to know what it could touch. It installs whatever it likes and runs whatever it writes, so every build runs in a Docker Sandbox, a small virtual machine of its own, with the repository mounted and nothing else from my machine.

My first network allowlist was tight: a handful of package registries and the model's endpoint. Then the dry run started bouncing off redirects I hadn't listed, and I had to ask myself what I was actually protecting. An agent pulling a Mosquitto image is doing its job. What it mustn't do is read about the project, because this repository and the site describe how builds are graded and what earlier builds got wrong, and it mustn't ever reach the house, where the real relays are. So the rule became: a build may fetch anything it installs, reads nothing about this project, and never reaches the house. GitHub and Docker Hub are open, this repository is closed by path in every spelling I could think of, and web search is off. A script checks all of it from inside the sandbox before each build, and refuses to start if anything is open that shouldn't be (harness/README.md, Network).

##Attempt one: trust the agent§

The first build took 201 seconds and about $0.45. It wrote 1,516 lines of Go, 245 of them tests nobody asked for, and announced it was done.

Then I looked at what "done" meant. The tests ran against a fake message bus, and the agent noted, quite calmly, that no MQTT broker was available to test the real thing against. A broker was one docker pull away, inside a sandbox where Docker works. So the controller had passed its own exam, set by itself, without ever meeting anything that looked like a device.

That's the first lesson, and it's not a subtle one: an agent left alone declares done as soon as its own tests pass, and its tests are as narrow as it likes.

##Attempt two: add a reviewer§

If the builder won't check its work, maybe a second session will. So H0 got a loop. After the build, a fresh session of the same agent reviews it against the spec and lists issues as high, medium or low. If anything is high, another fresh session fixes only those, and a new review starts from scratch. Medium and low are written down and left alone. The whole loop, and why it's in the baseline rather than a later version, is in ADR 001.

It ran, and it didn't help much. The first reviewer, asked to review the build against the spec, read the code and ran nothing. Fair enough, I hadn't told it to run anything. So I told it to run the program against a broker and play the devices, and it did, and found two medium bugs. Better. Then I tried what sounded like the stricter instruction, to confirm full adherence to the spec, and the reviewer went straight back to reading code, and skipped 14 clauses while it was at it.

So I had a loop that worked mechanically and proved very little. A reviewer allowed to test will happily not test, and "confirm adherence" reads, to a model, a lot like "have a look".

##Attempt three: change a few words§

The change that worked was small. Instead of asking the reviewer to confirm anything, the prompt now says to test every clause and every acceptance criterion, by whatever means it likes (harness/run/review-prompt.txt).

The next three reviews each started a broker, each by a different route, and none of them was told how. The second one found a real bug: two zones on the same relay device shared its availability topic, so one zone going offline took the other with it. The fix session fixed it, for about ten cents. That was the first time the loop earned its keep.

One more thing kept it from finishing. The reviewers decided, reasonably, that a relay turning itself off when its timer runs out needs real hardware, and sent the same three requirements back as high, round after round. One sentence fixed that: a simulated device counts, as long as it behaves the way the spec describes. With that and an example for each severity, the next review tested 62 of the 64 clauses, found another real bug, a target that could end up as "not a number", and once it was fixed, the loop ended on no high issue for the first time. Voilà.

That run cost $2.38 and 35 minutes for five sessions. Fix sessions turned out to be cheap, seven to twelve cents each, and occasionally inventive: asked to make an untested requirement testable, one built a simulated Shelly relay, timer and all, without being asked.

##How I know any of this§

All of the above is the agents' own opinion of their work. What decides is the hidden checks, and building those taught me some humility.

The grading model reads each build's README, installs it, and writes down how to start the controller, validate a file and ask for its status, from the README alone. Code then plays the devices on a fresh broker and decides pass or fail. With no reference app to test the checks on, each one has to fail on an empty build and on one whose README names commands that don't exist, and it has to catch builds I broke on purpose, one change at a time. There are 32 of those, from a relay that's never turned on to a controller that ignores the broker coming back.

Here's the humbling part: in this episode I found more bugs in my checks than in the builds. A reading published a few milliseconds before its step was marked made a fast build look like it reacted before the cause. Stopping the controller by name also stopped the test that had the name in its arguments. A timestamp taken from the wrong clock, 50 milliseconds off, failed a build for switching a relay too early when it had switched exactly on time. Every one of them looked like the agent's fault until I read the evidence, so that's now a rule: no verdict counts before I've looked at what it read.

The grader is DeepSeek V4.1 Flash. It's cheap and quick, and it doesn't come from the builder's family. On the first counted build Sonnet graded alongside it, and the two gave the same outcome in all 111 runs, three of each of the 37 checks. Each check runs three times per build, and the build gets whatever most runs say.

##Three builds, and the twist§

With the spec frozen at 1.0, H0 built it three times for real. Each build took 17 to 21 minutes and $1.21 to $1.65, and each one stopped after its first review, which found no high issue. The fix step, the part I'd spent most of the week on, never ran.

Then the hidden checks ran, and passed one build of the three.

The other two share a bug I'd never have thought to look for. When a Shelly sensor can't take a reading, it reports "tC": null. Both builds took that as 0 °C, because Go turns a JSON null into a zero without complaining. The room's temperature dropped to 10.5, then to 0, and in the house that turns the heating on. The spec says a reading that isn't a number is ignored (B1). Both reviews, told to test every clause, missed it. The same two builds also accepted a relay channel of 1.5 and reported one bad configuration value by line number instead of by name, which F1.AC2 doesn't allow.

So the number for this episode is 1 of 3: one build passed every machine check, after three reviews that each found nothing serious. On the verification set, the ten expectations I report on every version, F1.AC2 passed on one build and the other nine on all three.

Looking back at the trials, I'd been warned. One review had passed code that the two before it had failed, on a real bug, and had listed as confirmed two clauses it had just reported broken. One review is a sample, not a verdict. The loop got the builds from "never ran against a broker" to "tests every clause", which is a long way, and it still can't stand in for checks it can't see.

The builds wrote 3, 6 and 10 test functions, unprompted. The one that passed everything wrote the most. That's one data point, and I'm resisting the urge to make it a pattern.

##The house§

Build 3 goes into the house in a few days and gets three days to prove itself. The checks say it's the clean one. The bathroom will have the final word, and if it's wrong, I'll find out in my bathrobe.

##What I don't know yet§

Three builds can't say how often a review misses a bug like the null one, only that it can. I don't know whether the fix loop pays for itself on counted builds, because it never ran on one. And everything here is one model on one spec. When the soak is in, the cost per accepted change goes into ADR 001 and the changelog.

/enrico/