A prototype can work and still leave the question that prompted the project unanswered. It shows that a solution is possible. It does not yet tell us whether it addresses a significant enough need, whether teams will be able to use it or whether it is worth the investment.
To make an experiment useful, we start with the decision to be made. Depending on the uncertainty, we may need to observe users, compare results or test a technical constraint. The prototype then takes the form needed to answer that question.
Be clear about what you want to learn
“Test AI on our documents” leaves the scope wide open. “Find out whether a team can prepare reliable summaries of its case files more quickly” already gives us something to work on. We still need to specify who the summaries are for, which files they cover and what level of quality is needed.
Let us take this as an illustrative example. The experiment needs to help us decide whether a summarisation assistant is worth making available. We therefore need to understand how case files are handled today, what the summary needs to support and which errors would make it unacceptable to use.
The question may also lie further upstream: is summarising really what takes the time? If most of the effort goes into finding missing documents or resolving contradictions between sources, improving the writing will address only part of the problem.
Choose cases that represent the work
The first, straightforward examples help us understand what the tool can do. To make a decision, we then need to include less favourable cases: incomplete documents, conflicting versions, ambiguous wording or missing information.
People who know the work help assemble this set of cases. They know where an error is costly and what a usable result needs to contain. Frequency matters, as does severity: a rare situation may warrant a restriction if it exposes the business to a significant consequence.
Some examples are used to improve the approach. Others are held back to check that it works beyond the cases already worked on. Otherwise, the result may simply reflect the adjustments made to get the demonstration to work.
For a generative feature, the same request can also produce different results. Where that variability matters, several trials are needed. Anthropic’s guide to evaluating agents emphasises defining tasks, success criteria and ways to judge the result. The evaluation should remain proportionate to the decision being made.
Compare the work through to a usable result
In our example, timing text generation would not be enough. We need to include document preparation, checking claims, corrections and any rework.
We therefore compare the complete task with current practice, using comparable situations. A quickly generated summary can be difficult to review if it mixes accurate information with claims that cannot be traced. Conversely, an incomplete but well-sourced result can effectively help a professional prepare their work.
The criteria need to reflect that use. Accuracy, omissions and the ability to trace sources may matter more than fluent writing. For another use, response time or the quality of the interaction may be decisive.
Finally, we need to consider the full cost: the tool, integration, human time and maintenance. An experiment cannot give a precise cost for all future operations, but it can reveal the factors that will affect the decision.
Involve people in the experiment
A demonstration led by the person who prepared the system tells us little about whether future users can work independently. It is useful to let them carry out the work, then examine what happened.
Do they understand what they can delegate to the tool? Can they spot an uncertain answer? Do they know how to return to the original document? Does checking fit into their work, or does it become an additional task they will skip under pressure?
Practical conditions are also part of the experiment. It requires data that can legitimately be used, appropriate access and someone able to make decisions. A difficulty here may lead us to revise the scope before refining the prototype.
Finish with a decision and its conditions
Several next steps are possible. An existing tool may meet the need. A bespoke solution may be worth building. The experiment may also show that the intended use needs rethinking or that this avenue should be dropped.
A useful result sets out what was observed, the cases for which it holds and the remaining uncertainties. It also identifies the conditions for moving forward: data access, responsibility for checking, integrations and criteria to monitor.
For a product idea, the approach will be different. Testing a mock-up can shed light on how people understand a journey without demonstrating willingness to pay. An initial commitment from a user can indicate interest without establishing technical feasibility. The conclusions drawn from each experiment must stay within what it can actually demonstrate.
This is the starting point for our Explore AI through experimentation offer: choose a scope small enough to learn from and representative enough to support a decision.