There used to be a useful interval between wanting a piece of software and possessing it. During this interval, known as development, a person had several opportunities to discover that the idea was bad. We have made tremendous progress in reducing this unnecessary friction.
Now the application exists before you have decided what it should do when something goes wrong. It has a tasteful icon. It has settings. It may have more confidence about your computer than you do.
In March, Simon Willison described experimenting with AI-built SwiftUI utilities for his Mac. His account was enthusiastic and unusually explicit about its limits: he had inspected little of the code, lacked relevant platform expertise, and had encountered a misleading memory reading that required correction. He did not present the experiment as a production assurance program.1
The demo has become a very persuasive witness
Trouble begins when the answer is promoted.
The interface becomes evidence of correctness. Correctness becomes evidence of security. Security becomes evidence that someone else can safely depend on the application. These promotions may occur without an intervening test. The button is very well aligned, and we are all busy.
Consider a hypothetical expense tool. It imports a sample receipt, extracts a total, and displays a chart. The demonstration is real. So are the questions it has not answered. What happens with a refund? A foreign currency? Two receipts that look alike? A file containing something other than what its extension suggests? Who can read the stored material?
Cheap software is welcome. The bill that worries us is the one attached to a free tool after someone has quietly made it indispensable.
The new cost center is judgment
When typing the code gets cheaper, the work that used to wait behind it becomes visible. Someone must decide what the tool should do, check that it does it, and keep the thing running after the exciting afternoon is over.
That is why “I built it in an hour” can be both true and incomplete. Perhaps the useful experiment really took an hour. Perhaps the application will require ongoing updates, monitoring, and decisions about what it is allowed to do. Those future obligations did not become free because they were absent from the stopwatch.
An hour is an excellent price for learning something. It is a less informative price for assuming indefinite responsibility.
Permission to make small things
There is a joyless reaction to all this in which every personal script must be operated like a bank. That would be a mistake. The ability to make small, disposable tools is one of the most appealing uses of coding assistants. A person should be allowed to solve a minor annoyance without founding an enterprise software company.
Start with the radius of the experiment. Who can be affected? What data does it touch? Can its actions be undone? Does anyone believe it is more reliable than its builder has established?
A tool that displays a rough local statistic and a service that writes to customers’ accounts should not inherit the same confidence merely because the same assistant produced them. One is a convenient experiment. The other is a promise to people who were not present when the experiment was conceived.
Give the experiment room to fail where failure is cheap. Before it starts handling somebody else’s work, decide who will notice a mistake and what they can do about it.
The part we cannot ship as a screenshot
The strongest demonstration of the new tools is not a founder claiming to have become an expert in every field before lunch. It is someone doing something previously inaccessible while remaining clear about what they have not learned.
A prototype can stay a prototype. It does not need to acquire customers, a subscription tier, and the word “platform” before dinner.