This Is Not How You Build Software

Software is for users. Building it requires observation, judgment, experimentation, and honest feedback—not a mandate to use a particular tool.

I have been building software for more than 25 years. The technology stacks move around, but the job does not.

We build tools to help users solve problems.

I also enjoy exploring new technology. I want developers to try unfamiliar tools, build prototypes, and discover uses nobody planned. Some experiments will fail. Some will turn out to be useless. That is fine. Learning what does not work is one of the purposes of an experiment.

Production is different.

Production software is something people depend on. They use it to do their jobs, make decisions, serve customers, move money, and manage information. When it fails, those people absorb the cost. They lose time. They repeat work. They develop workarounds. Sometimes they make decisions based on information that was wrong.

This is why I try to be liberal about exploration and conservative about production.

The current push to require generative AI tools throughout organizations often reverses that posture. A tool is selected before the user is observed. Employees are expected to use it before the problem is understood. Evidence is collected after the decision.

Language-model assistants can be useful. Some have produced real and measurable gains. That is not the question.

I should be clear about my own experience: I have found a language-model-based coding toolset that works well for me, for now. The tools are changing quickly, and so is the way I use them. That personal success is a reason to keep experimenting. It is not a reason to prescribe the same tools to every developer, team, or codebase.

The question is whether choosing the tool first is a sound way to build software.

I do not think it is.

Software is for users

Software is not primarily for the developer who wants to try a new tool. It is not primarily for the executive who wants an AI strategy.

It is for the person who has to use it.

That user may happen to be a developer or an executive. Their job title is not the important part. What matters is that they are the person trying to accomplish something, and they are the person who will live with the system’s behavior.

Developer enthusiasm can justify an experiment. Executive interest can justify funding one. Neither tells us whether the resulting software helps its users.

A system has not succeeded because the engineering team enjoyed building it, leadership approved it, employees activated their licenses, or a demonstration looked impressive. Those things may be encouraging, but they are not the result we are trying to produce.

The useful question is more ordinary:

Who has to live with this software?

Can that person complete the work more easily? Does the system reduce mistakes, or create new ones? Does it save time after its output has been checked and corrected? Does it make the work safer and more understandable, or does it move effort somewhere less visible?

If users avoid a system, work around it, check everything it produces, or push the resulting problems onto another team, the software is not working well. An adoption dashboard does not change that.

Users receive the benefit when software succeeds. They also absorb the cost when it fails. Their experience must be the center of the decision.

Explore freely. Deploy carefully.

Exploration and production are related, but they are not the same activity.

Exploration should be inexpensive, open, and easy to abandon. Let people try new models. Let them build prototypes and test strange ideas. Give them room to discover useful applications that could not have been specified in advance.

I do not want fear of failure to prevent that work. A small experiment should not require a lengthy business case or a promise that it will eventually transform the company.

A failed experiment can still be valuable because it produces information at limited cost.

Production creates obligations.

Before putting a language-model assistant into production, we should know what happens when it is wrong, how users will recognize a mistake, who will be responsible for correcting it, and whether the system still saves time after review, training, security, and support are included.

An agent deserves additional caution. By “agent,” I mean software that can perform several steps or take actions with limited human involvement. An assistant may suggest an email; an agent may send it. An assistant may draft a database query; an agent may run it and use the result to trigger another action.

Those differences matter because the consequences are different.

My rule is simple:

Be liberal about exploration. Be conservative about production.

Encourage curiosity. Ask for evidence before creating dependence.

A mandate breaks the loop

I use OODA to keep the order of operations straight: observe, orient, decide, and act. Then look again.^[1]

Applied to software, the process is straightforward.

Observe the user doing the work. Understand what happens today, including the delays, mistakes, workarounds, and unmet needs.

Orient yourself to that situation. Consider the users, the available information, the technical limits, the cost of failure, and the possible approaches.

Decide what to try. Prefer the smallest experiment that can answer a useful question. Define what success and failure will look like before running it.

Act. Run the experiment, measure what happened, and return to observation.

That return matters. An action changes the situation and produces new information. The new information should be allowed to change the team’s understanding and its next decision.

A tool mandate often begins near the end of the loop. The decision has already been made:

Use generative AI. Buy the licenses. Add an agent. Report adoption.

Observation and orientation become secondary. Instead of determining whether a tool fits a user’s problem, the organization begins looking for places to apply a tool it has already chosen.

Shopify provided a public example in April 2025. CEO Tobi Lütke told employees that “reflexive AI usage” was now a baseline expectation. He said AI use would become part of performance and peer reviews. Teams asking for additional staff or resources would first have to explain why they could not accomplish the work using AI. The memo also asked employees to consider how their work would change if autonomous agents were already part of the team.^[2]

That is one company, not evidence that every organization works this way. The policy may produce useful experiments and discoveries. It may also improve work in areas where language-model tools are a good fit.

But it changes the default question.

A team may no longer begin by explaining why an assistant or agent is appropriate. It may have to explain why it is not.

Research on failed AI and machine-learning projects gives us reason to take that reversal seriously. RAND interviewed 65 experienced data scientists and engineers about why such projects fail. Recurring causes included misunderstanding the problem, lacking necessary data or infrastructure, focusing on new technology rather than the user’s need, and asking the technology to do something it could not reliably do.^[3]

RAND studied AI and machine learning broadly, not only the current generation of language-model products. Its findings do not establish a failure rate for assistants or agents. They do describe a durable software failure: choosing the technology before understanding the problem.

Mandates also make usage a tempting measure of success. Companies can count activated licenses, prompts, tokens, and projects with “AI” in the name. Those numbers are easy to collect and easy to increase.

Once a metric becomes a target, people can optimize the number instead of the result it was meant to represent.^[4] If employees are evaluated by how often they use language-model tools, they will find reasons to use those tools more often.

That does not tell us whether users received better service, software shipped faster, mistakes declined, or work became easier.

An LLM-usage dashboard measures LLM usage.

It does not tell us whether the software is useful.

A demonstration is not production

Generative AI systems produce compelling demonstrations. Give a language model a narrow task, a clean prompt, and carefully prepared information, and it can return an impressive answer in seconds.

That experience can cause otherwise careful people to skip ordinary software questions:

• How often is it wrong?
• Can users tell when it is wrong?
• How much work does checking its answer require?
• What happens if someone acts on a bad answer?
• Does it still save time after correction, training, security, and support?
• Does it work with the incomplete and inconsistent information found in the real organization?

Consider a language-model-based tool that lets someone question a company database in plain English.

At risk of seeming like an endorsement: Snowflake reported 92% SQL accuracy for Cortex Analyst in a benchmark involving joins across standard star and snowflake database layouts.^[5] That is a strong result within the conditions Snowflake tested.

It does not mean Cortex Analyst answers 92% of all possible business questions correctly. It means Snowflake measured 92% SQL accuracy in a particular set of scenarios using its own benchmark.

That is evidence worth considering. It is enough to justify further evaluation. It is not enough to trust every answer.

Snowflake provides tools for that additional work. Customers can execute generated SQL, compare its results with verified queries, and identify regressions when a query that previously produced the correct result begins to fail.^[6]

That is responsible software development. The demonstration establishes that the idea is possible. Testing determines whether users can depend on it.

Before putting a language-model system into production, I want clear answers to three questions:

1. How often is it wrong?
2. How likely are we to catch the mistake?
3. How much harm could a missed mistake cause?

The same assistant may be safe for drafting an internal email and unsafe for approving a payment. A mistake in one context is an inconvenience. A mistake in the other may move money to the wrong place.

NIST recommends testing AI systems before putting them into everyday use, continuing to check them afterward, documenting what remains uncertain, comparing results with meaningful standards, and involving people who understand both the subject and the consequences of a mistake.^[7]

Its framework covers artificial intelligence broadly, not just generative AI. The principle is still relevant here: test the actual system, with the actual users, under the conditions where it will operate.

What the evidence actually says

The evidence does not support a simple verdict about language-model tools.

A study by Erik Brynjolfsson, Danielle Li, and Lindsey Raymond examined the introduction of a language-model assistant among 5,179 customer-support agents. The assistant suggested responses while human agents continued handling the conversations.

Access to the assistant increased the number of customer issues resolved per hour by 14% on average. Newer and lower-skilled workers improved by 34%. Experienced and highly skilled workers received little measurable benefit.^[8]

That is a credible success.

The assistant addressed a specific user and a specific job. People remained responsible for the conversation. The researchers measured an actual result. The benefit was meaningful but uneven.

Research on coding assistants provides a useful counterexample. METR studied experienced open-source developers working on their own projects. Developers using early-2025 coding assistants took 19% longer to complete the assigned work, even though they believed the tools had made them faster.^[9]

METR warned readers not to stretch the result beyond what was studied. It did not show that language models slow down most developers, other kinds of workers, or people using newer tools.

A later METR study produced some numbers suggesting that newer coding tools might save time. The researchers could not draw a firm conclusion, however. Developers who disliked working without AI tools were less likely to participate, and developers running several coding agents at once made working time difficult to measure accurately.^[10]

Stanford’s 2026 AI Index reports that generative AI was used in at least one business function by 70% of surveyed organizations. Agents capable of performing multi-step work remained much less common, with deployment in the single digits across nearly every business function measured.^[11]

That tells us adoption is widespread. It does not tell us that every use is valuable.

The honest answer is not especially dramatic: it depends.

Language-model assistants have produced real gains for some users doing some tasks. They have produced disappointing or negative results in others. Agents that perform longer sequences of work are a separate and less mature category.

Software does not succeed in the abstract.

It succeeds for a particular user, doing a particular job, under particular conditions.

It has to be safe to call bullshit

People need to be able to call bullshit — but with kindness and empathy. The point is not to punish enthusiasm or embarrass someone for being wrong. It is to get accurate information back into the loop while there is still time to change course.

A project cannot improve unless information about failure reaches someone who can act on it.

A tool is not being used. Its answers cannot be trusted. Checking its work takes too long. Customers dislike it. The promised savings never appeared.

Those observations need to reenter the loop.

An engineer must be able to say that checking and repairing code from a language model takes longer than writing it directly. A support team must be able to show that a chatbot has moved work to another department rather than eliminating it. A finance team must be able to reject an agent because its mistakes are difficult to catch or a wrong action could cause serious damage.

People learn what is safe to say.

If visible enthusiasm for generative AI affects performance reviews, promotions, budgets, or staffing, bad results become dangerous to report. Public agreement can grow while private doubt disappears from view.^[12]

No conspiracy is required. Each person may make a reasonable decision about personal risk while the organization as a whole receives worse information.

This does not mean every enthusiastic executive is secretly skeptical or every generative AI program suppresses disagreement. It means leaders should check whether disagreement remains safe.

Can users report that the assistant is not helping? Can employees say that an agent creates more work than it removes? Can someone question the mandate in a group meeting? Would an anonymous survey produce the same answers as a public discussion?

If people cannot tell you that the software failed, you do not know whether it worked.

A practical review does not need to be elaborate:

1. Who is the user?
2. What are they trying to accomplish?
3. What happens today?
4. Why is this particular tool a reasonable approach?
5. How will users detect and recover from a failure?
6. What result would cause us to expand, change, or stop the project?
7. Can users and employees report bad results without putting themselves at risk?

Measure the result, not the ritual.

Count correctly resolved problems, completed work, mistakes, customer satisfaction, time, cost, and harm. License activation and token consumption may show that a service is being used. They do not show that the use is worthwhile.

Software is for users.

Observe their work. Understand the problem in context. Choose a small experiment. Act, measure what happened, and begin again.

Be liberal about exploration. Be conservative about production.

Use language-model assistants when they help users complete real work more quickly, accurately, or safely. Be very cautious with agents that can take actions without waiting for a person. Stop or change either kind of system when it creates more work, mistakes, or risk than it removes. Make it safe for people to say so.

That is how you build software.

Choosing the tool first and measuring obedience is not.

References