AI implementation: plan your first business pilot
A first pilot should answer a specific question: can the system complete the chosen task at acceptable cost and quality? Document the current workflow and acceptance conditions before development. Those inputs support a decision to launch, revise or stop.
5 min read
Choose the question the pilot will answer
Implementing AI across a company is too broad for a first test. Narrow the task to one action: preparing a request, finding a supported answer in documents or filling a record for staff review. Define the input and the form of the result.
Illustrative example: a team wants to reduce the time needed to prepare a record from an incoming request. The pilot tests agreed fields on anonymised examples while a member of staff approves production CRM writes. This illustrates project boundaries, not an IOANN client deployment.
Assign a process owner to select examples, explain the rules and accept the result. Without that involvement, a working demonstration may still be impossible to assess against the actual workflow.
Record the baseline and alternatives
Measure volume, processing time, corrections and staff handovers on comparable tasks. Describe uncommon but costly failures separately. Measure the baseline before the pilot; recollections of the old process are a weak basis for comparison.
Consider an option without a model. Structured inputs and stable steps may suit a conventional integration. For free text, compare an existing tool and custom development using the same examples. Let these checks inform the technology choice.
- What does staff receive, and where is it stored?
- Which fields or documents are needed?
- How is a correct result identified?
- What happens when information is missing?
- What costs and delays exist today?
Agree acceptance criteria before the first demo
Define an acceptable outcome for each example category. Some tasks should complete automatically; others should reach a person. Declining to act can be correct when data or permission is missing. Do not collapse these outcomes into one accuracy number.
Choose quality thresholds according to the workflow and consequences of failure. Agree them before testing, alongside acceptable latency, cost and human workload. Do not adjust the criteria to fit a successful demonstration.
| Check | Example | What to record |
|---|---|---|
| Routine task | Sufficient data and available services | Required fields, correct target and completed action |
| Ambiguity | Multiple matching records | Clarification or handover without an arbitrary choice |
| Unsupported answer | The required fact is absent from documents | A stated limitation instead of an invented answer |
| Integration failure | Unavailable service or lost acknowledgement | Clear status, no duplicate write and a recovery path |
| Access restriction | User cannot access some data | No answer or action based on restricted records |
| Human review | System hands an exception to staff | Context completeness and time to completion |
Prepare the data and evaluation
Collect anonymised routine and difficult examples. Confirm they may be used for development and that the expected outcome is known. A separate final-evaluation set shows behaviour on cases the system was not tuned against.
Repeat evaluations when results vary between runs. Record the model, instructions, data and tool versions so changes can be traced. Recheck affected scenarios after permission or integration changes.
For our own product, Inner Voice, we prepared 30 evaluation scenarios with five repetitions and compared four model candidates. Those numbers describe a particular engineering evaluation of our product. They neither prescribe a universal test size nor establish client savings.
Decide what follows the evaluation
Summarise agreed outcome categories, full processing costs and staff workload, then compare with the baseline. Separate technical limits from data problems: changing instructions will not repair an inaccessible source or an outdated policy.
Launch requires the acceptance conditions and an operational plan: an owner, event logs, human handover, spending limits and a stop mechanism. Where gaps remain, define a specific next test. If an option without AI solves the task better, the pilot can end there.
A first implementation discussion needs a process description, system list, anonymised examples and the expected result. Identify any missing inputs. Those materials support pilot scope and evaluation planning, followed by a development estimate.