Blog post
Building to Learn and Building to Deliver
In Thesis 3, we address the objection that because building has become cheap, we can simply implement every idea and learn in production whether it works. Certain assumptions, such as those about usability or feasibility, can indeed be tested far more cheaply through agentic building than before. Whether what we build should also end up in production and only be tested there, however, is an entirely different question.
In Continuous Discovery Habits, Teresa Torres draws a clear distinction between assumption tests and experiments. Assumption tests are used during discovery to check individual assumptions. They help improve a solution idea and choose between several ideas. Experiments, such as A/B tests, measure whether something that has been shipped achieves the intended impact, and they need a correspondingly large amount of traffic to do so. Torres gives two reasons for this separation. First, you don’t want to build everything only to find out it was the wrong thing. Second, if you test an entire idea instead of a single, small assumption, you don’t know why it fails. When an individual assumption test fails, it is fairly clear what needs to change.
At first glance, agentic development has made the first argument obsolete: if building costs little, building the wrong thing isn’t a big deal. The second argument still holds. A shipped feature is a test of the entire idea. If the feature is barely used, there are several possible causes, such as a lack of need, poor discoverability, usability problems, or a well-practiced workaround. The fact that building it was cheap does nothing to help you find the cause.
Cheap building can also help where a single assumption is being tested. Torres counts prototype tests and research spikes among the four types of assumption tests. With coding agents, we can test usability and feasibility assumptions that used to be too expensive to check. Such an artifact has one job: to answer a question. After that, it gets thrown away. It doesn’t need production quality, and it doesn’t run where users get their work done.
Building to deliver is a completely different activity. The result is meant to stay and will have to be maintained in the future. Afterwards, its impact is measured, in the sense of an experiment.
It may be tempting to blur the lines. A prototype built by an agent quickly looks like a product, and throwing it away often takes some resolve. If you keep it, the test turns into unplanned delivery without a harness. That increases the maintenance burden and creates technical debt. The remedy is discipline before building: Which assumption is being tested, what would a given result mean, and who will delete the prototype afterwards? Torres recommends defining success criteria before running a test anyway.
Where the test takes place matters, too. Prototype tests with people who know they are trying something out don’t interfere with the interface that production users rely on. The same naturally applies to other types of assumption tests, such as data mining on existing data. Those who continuously roll out ideas to production, on the other hand, use up conformity with user expectations: users build mental models through repeated use, and every change resets parts of them. More on this in “Faster Is Not Better”. One exception is brief fake door tests, which should be used rarely and for a limited time.