Skip to main content
Every answer Mole Agent writes is scored. Rules score it at once, without a model: whether its tools worked, whether it cites evidence, whether its queries parse, whether it came out in the right shape, and how whoever asked rated it. Evaluators you set up score a sample of the answers with a model, by a rubric you write. Answers that score low, that someone rated not helpful, or that are drawn at random wait for review; the ones a reviewer accepts become the regression set. Before an organization prompt becomes the default, a check answers some of these questions again with the new version and with the present one, and an evaluator scores both. Scores read from 0 to 100. The Quality tab shows them, and every score is also written to the stream _agent_score_traces.

​
Before starting

  • The Quality tab needs agent.manage, which the owner and admin roles have by default. Its holders can already read every team chat; personal chats only count in the totals, and nobody reads them there.
  • An evaluator uses one of the organization’s model providers. See Mole Agent. Without an evaluator, every answer is still scored by rules.

​
What is scored

An answer is scored when it is written, and again when whoever asked rates it or takes the rating back.

​
Rules

Rules look at what the answer and its run left behind, and score each of these from 0 to 100. The rules score of an answer is the mean of the ones that apply.

​
Evaluators

An evaluator gives an answer 1 to 5 by its rubric, and says why in a sentence or two; the score reads as 0, 25, 50, 75, or 100. It reads the question, what the tools found, and the answer; for an RCA, the facts of the incident and the RCA.
  • An evaluator scores the percentage of new answers you set, drawn by the answer, so drawing again gives the same answers.
  • Answers rated Not helpful are always scored, unless the evaluator already scored them.
  • An answer the evaluator scores below the review threshold is scored once more, and the mean of the two counts.
  • An evaluator never scores personal chats.
  • An evaluator writes its reasons in the interface language of whoever saved it last.
  • The model calls of evaluators count toward the organization’s daily token budget, as Quality scoring in the usage by source. While the budget is used up, evaluators skip new answers.
The instructions every evaluator starts from are the built-in prompt Answer evaluator (purpose evaluation); the organization can override it under Settings → Prompt management. Whatever the prompt says, the evaluator returns its score in a fixed format. The score of an answer is the mean of its evaluators’ scores when an evaluator scored it, and its rules score otherwise.

​
The Quality tab

Open Mole Agent → Quality. Overview and Answers look at the last 7, 30, or 90 days and at one source or all of them.

​
Review answers

An answer waits for review when: An answer comes to review once. Open it, write in Note what is wrong and what a good answer gets right, and select:
  • Accept: the answer is wrong. It joins the regression set, and its note goes to the evaluator when a prompt check answers the question again.
  • Reject: the answer is fine.

​
Evaluators and settings

Under Evaluators, New evaluator opens the form: An organization has at most 10 evaluators. Deleting an evaluator keeps the scores it gave. The settings below the evaluators:

​
Check a prompt before it becomes the default

When Set as default is selected on an organization prompt for chats (System instruction) or for incident RCAs (Root-cause analysis), the organization’s setting decides what happens: A check takes up to 8 answers: first the regression set, then the answers the first evaluator that scores these answers scored most recently. Each is answered again twice, with the present default and with the new version, by the same model and without running tools again: the model gets the question and what the tools found that time. For an RCA, both versions write it from the facts of the incident. The evaluator then scores each pair. The new version is Worse when its mean is more than 5 below the present default’s, Better when more than 5 above, No worse otherwise. When fewer than 3 answers could be compared, the check cannot tell, and does not hold the prompt back. The dialog shows the progress and then the result for each answer. A check is of one version of the prompt against one default: after the prompt is edited or another default is set, it has to be run again. The model calls of a check count as Quality scoring. Prompts for other purposes become the default without a check. Editing the prompt that is already the default is not checked either; to check a change, save it as a new prompt and set that as the default.

​
Scores in a stream

Each score is also a row of the internal stream _agent_score_traces, which you can query in Logs: To hear about a drop in quality, create an anomaly alert rule over the stream, for example on the evaluators’ mean:
Last modified on October 21, 2026