A new showroom employee asks where to record a damaged delivery. An AI assistant returns a clear set of steps and links to a company document. Before the employee follows those steps, someone needs to know whether the assistant found the current procedure, kept its conditions, and named the right person to contact.

Datum recommends testing a company knowledge assistant with questions your staff actually ask. Start with a narrow area, such as receiving materials. Write down the accepted answers before running the test, including the questions your documents cannot answer.

Build an answer key with the person who owns the work

Anthropic’s January 2026 evaluation guide describes tests with defined inputs and success criteria. It also recommends repeated trials because an AI system can respond differently across runs. For a building-industry team, the useful translation is a small, repeatable set of staff questions reviewed by the person responsible for the procedure.

Ask the receiving lead to gather recent questions without customer names or private order details. Use the words employees used, including informal names for forms and locations. Give the assistant access only to material approved for the intended staff role.

  • For an answerable question, record the required steps, conditions, current source passage, and procedure owner.
  • For a question with missing information, record what the assistant should ask before answering.
  • For a question the documents do not answer, record the person or role that should receive it. Do not invent a company policy to fill the gap.
  • For conflicting documents, record whether an authorized owner has resolved the conflict. If not, the expected response should expose it.

Keep this answer key outside the material supplied to the assistant during the test. The goal is to find out whether it can answer from the working documents your employees will use.

Include a question that sounds easy but has no answer

Consider a hypothetical receiving procedure that explains how to photograph damage and notify purchasing. It says nothing about where to store a damaged cabinet while awaiting instructions. Ask the assistant where that cabinet should go.

A useful response identifies the missing storage instruction and routes the question to the designated receiving lead. An invented storage location fails the test, even if the rest of the answer accurately describes the photo process. The procedure owner can then decide whether the document needs a storage rule.

Also test the same question with an older procedure present. Mark the old document as superseded and give the current document a clear effective date. Check whether the answer uses the current rule. If the records lack that status information, fix the records before expecting AI to resolve their authority.

Read the passage behind the citation

Anthropic’s Citations documentation explains how its API can return references to passages in supplied documents. Those references help a reviewer locate the evidence. Datum’s recommendation is to review whether each passage supports the instruction beside it and whether that document applies to the employee’s question.

In the cabinet example, a citation to the damage-photo procedure would not support an invented storage location. Record that as an unsupported instruction. Separately record an answer that uses the wrong revision, omits a condition, or asks an unnecessary follow-up when the answer is already available.

Choose the release rule before reading the results

For an initial receiving trial, we recommend blocking staff rollout if the assistant invents a required action or presents an unresolved document conflict as settled. Treat these as separate failures so a pile of correct answers cannot hide them. This is a proposed test standard, not a claim that a particular assistant has passed it.

Have the procedure owner review repeated answers to each question. Record the failures and how long verification takes. Keep some questions aside while adjusting the tool, then use them to check whether improvements extend beyond the examples used to tune it. Repeat the checks when the source procedures or the assistant change.

Bring a redacted procedure, a few staff questions, and the answers your team accepts to a conversation with Adam. Datum Training helps your people build and run AI tools around a defined task. This packet gives that work a concrete starting point: what the assistant must get right before someone relies on it.

Sources Read

Next step, if this note maps to a problem on your desk: book a conversation with Adam. Thirty minutes, weekday evenings, Eastern time.

Related Field Notes