Five questions before building an AI assistant

An AI assistant is easy to demo and hard to run. A model answers fluently on day one. The work is everything around it: what it reads, what it is allowed to do, what it costs, and how you know it is right.

Ask these five questions before the build starts. If your team or your partner cannot answer one, you have found the first task.

1. What will it answer from?

An assistant is only as good as the documents behind it. List them before you talk about models.

• Where does the knowledge live: a help center, PDF files, a CRM, a shared drive, people’s heads?
• Who keeps it current? An assistant that quotes last year’s price list is worse than no assistant.
• What language is it in? Scanned Arabic PDFs need their text extracted and checked before any system can search them.
• Who may see what? The assistant must never show a user a document they could not open themselves.
• Does it hold personal data? In Saudi Arabia the Personal Data Protection Law applies. Decide where the data is processed before you choose a model provider.

If the answer is “we will clean the data later”, the project is a data project first. That is fine. Plan it that way.

2. How does it stay grounded, and can it show its source?

A language model on its own answers from its general training. It sounds as sure when it is wrong as when it is right. In a business, the answer has to come from your documents. The usual method is retrieval: the system finds the relevant passages, hands them to the model and instructs it to answer only from them.

Ask for three behaviors:

• Every answer cites the passage it came from, with a link a person can open and check.
• When the documents do not hold the answer, the assistant says so and hands over to a person.
• Search quality is tested separately from writing quality. A wrong answer often starts with the wrong passage.

3. Where does a person approve?

Reading and acting carry different risks. An assistant that summarizes a contract can be wrong cheaply. One that sends an email, issues a refund or edits a record cannot.

• List every action it can take. Mark each as reversible or not.
• Anything a customer sees, anything that moves money and anything that deletes needs a named person’s approval, at least at the start.
• Keep a log: what was asked, what was retrieved, what was answered, who approved.

Approval steps can be removed later, action by action, as the numbers earn it. Adding them after an incident is harder.

4. What does it cost to run?

Building is a one-time cost. Running is a monthly cost that grows with use. Models are priced per token, roughly a piece of a word, for what goes in and what comes out:

cost per request = (tokens in × input price) + (tokens out × output price)

What goes in is usually the larger part: the instructions, the retrieved passages and the conversation so far. Ask for these numbers before you sign:

• Cost per conversation at the expected length.
• Monthly cost at expected volume, and at ten times that.
• What lowers it: a smaller model for simple requests, caching of repeated context, shorter retrieved passages.
• Who pays for upkeep: updating the documents, re-testing when the model changes, reading the logs.

A partner who cannot give a number has not designed the system yet.

5. How will you know it works?

“It feels good” is not a measurement. Before the build, collect real questions from your support inbox or your staff, each with the answer you would accept. A hundred is a workable start. That set is the exam. The assistant sits it every time something changes: the model, the instructions or the documents.

Then agree on a few numbers:

• Correct answers on the test set.
• Answers that carry a correct citation.
• Correct refusals when the answer is not in the documents.
• Conversations handed to a person.
• Cost per resolved request.

After launch, read a sample of real conversations every week. Each failure becomes a new question in the exam.

Then build

The dabb looks first, then moves. These five questions are the looking. With the answers on paper, the first version can be small: one task, one source of documents, one team using it. Its numbers tell you where to go next.