> ## Documentation Index > Fetch the complete documentation index at: https://docs.molesignal.com/llms.txt > Use this file to discover all available pages before exploring further. # Quality > How good Mole Agent's answers are: rules score every answer, evaluators score a sample of them with a model, low scores wait for review, and a new default prompt is checked first. Every answer Mole Agent writes is scored. Rules score it at once, without a model: whether its tools worked, whether it cites evidence, whether its queries parse, whether it came out in the right shape, and how whoever asked rated it. Evaluators you set up score a sample of the answers with a model, by a rubric you write. Answers that score low, that someone rated not helpful, or that are drawn at random wait for review; the ones a reviewer accepts become the regression set. Before an organization prompt becomes the default, a check answers some of these questions again with the new version and with the present one, and an evaluator scores both. Scores read from 0 to 100. The **Quality** tab shows them, and every score is also written to the stream `_agent_score_traces`. ## Before starting * The **Quality** tab needs `agent.manage`, which the owner and admin roles have by default. Its holders can already read every team chat; personal chats only count in the totals, and nobody reads them there. * An evaluator uses one of the organization's model providers. See [Mole Agent](https://docs.molesignal.com/en-US/agent#set-up-a-model-provider). Without an evaluator, every answer is still scored by rules. ## What is scored | Source | What it is | | - | - | | **Team chats** | Answers in team chats. | | **Personal chats** | Answers in personal chats. Scored by rules only, and only counted in the totals. | | **Automations** | The answer of each run of an organization or team automation. See [Automations](https://docs.molesignal.com/en-US/automations.md). The runs of a personal automation are personal chats. | | **Incident RCAs** | RCAs that root cause analysis generates, automatically or on demand, and any RCA someone rates. | An answer is scored when it is written, and again when whoever asked rates it or takes the rating back. ## Rules Rules look at what the answer and its run left behind, and score each of these from 0 to 100. The rules score of an answer is the mean of the ones that apply. | Rule | Score | Applies to | | - | - | - | | **Tool calls** | The share of tool calls that did not fail. | Answers that called tools. | | **Evidence** | No cited evidence gives 0, cited evidence that no successful tool call backs 50, one piece 75, two or more 100. | Answers in the requested shape that called tools. | | **Queries** | The share of SQL and PromQL in the tool calls that parses. | Answers whose tools ran queries. | | **Format** | 100 when the answer is in the shape answers are asked for. | Every chat answer except generated queries. | | **Completion** | 100 for a finished answer, 50 for an automation run that needed someone, 0 for a failed, stopped, or empty one. | Every answer. | | **Rating** | 100 for **Helpful**, 0 for **Not helpful**. On an RCA one **Not helpful** counts. | Answers someone rated. | ## Evaluators An evaluator gives an answer 1 to 5 by its rubric, and says why in a sentence or two; the score reads as 0, 25, 50, 75, or 100. It reads the question, what the tools found, and the answer; for an RCA, the facts of the incident and the RCA. * An evaluator scores the percentage of new answers you set, drawn by the answer, so drawing again gives the same answers. * Answers rated **Not helpful** are always scored, unless the evaluator already scored them. * An answer the evaluator scores below the review threshold is scored once more, and the mean of the two counts. * An evaluator never scores personal chats. * An evaluator writes its reasons in the interface language of whoever saved it last. * The model calls of evaluators count toward the organization's daily token budget, as **Quality scoring** in the usage by source. While the budget is used up, evaluators skip new answers. The instructions every evaluator starts from are the built-in prompt **Answer evaluator** (purpose `evaluation`); the organization can override it under **Settings → Prompt management**. Whatever the prompt says, the evaluator returns its score in a fixed format. The score of an answer is the mean of its evaluators' scores when an evaluator scored it, and its rules score otherwise. ## The Quality tab Open **Mole Agent → Quality**. **Overview** and **Answers** look at the last 7, 30, or 90 days and at one source or all of them. | View | What it shows | | - | - | | **Overview** | The answers scored, the mean of the rules and of the evaluators, the answers rated helpful and not helpful, and how many wait for review. Below: the score by day, each rule, and the answers by source, prompt version, Skill, and model. | | **Answers** | The scored answers, newest first, with **Low scores only** to keep those below the review threshold. A row opens the whole question and answer, each rule with what it found, each evaluator's score and reason, and the answer's review. **Open the chat** or **Open the incident** goes to where it was written. | | **Reviews** | The answers waiting for review, accepted, or rejected. | | **Evaluators** | The evaluators and the settings. | ### Review answers An answer waits for review when: | Cause | When | | - | - | | **Low score** | An evaluator's score is still below the review threshold after its second scoring, or, for an answer no evaluator scores, its rules score is. | | **Rated not helpful** | Whoever asked rated it **Not helpful**. | | **Drawn at random** | It was drawn by the **Random review** percentage, to see whether the scores can be trusted. | An answer comes to review once. Open it, write in **Note** what is wrong and what a good answer gets right, and select: * **Accept**: the answer is wrong. It joins the regression set, and its note goes to the evaluator when a prompt check answers the question again. * **Reject**: the answer is fine. ### Evaluators and settings Under **Evaluators**, **New evaluator** opens the form: | Field | Description | | - | - | | **Name** | Unique in the organization. | | **Rubric** | What a good answer does and what counts against it, up to 4000 characters. A rubric to start from is filled in. | | **Answers** | Team chats, automations, incident RCAs. | | **Sample** | The percentage of new answers it scores, 0 to 100. | | **Model provider** and **Model** | Empty uses the organization's first available provider and its default model. | | **Temperature** | 0 to 1; 0 keeps the scores steady. | | **On** | An evaluator that is off scores nothing. | An organization has at most 10 evaluators. Deleting an evaluator keeps the scores it gave. The settings below the evaluators: | Setting | Default | Description | | - | - | - | | **Review threshold** | 60 | Answers scoring below it wait for review. | | **Random review** | 0 | The percentage of the other answers drawn for review. | | **Before a prompt becomes the default** | Check, and ask again when worse | See below. | ## Check a prompt before it becomes the default When **Set as default** is selected on an organization prompt for chats (**System instruction**) or for incident RCAs (**Root-cause analysis**), the organization's setting decides what happens: | Setting | When the prompt has no check, or the check found it worse | | - | - | | **Check, and ask again when worse** | The dialog offers **Run the check**, and **Set it anyway**. | | **Check, and refuse when worse** | Only a prompt a check found no worse can be set. | | **Do not check** | The prompt is set at once. | A check takes up to 8 answers: first the regression set, then the answers the first evaluator that scores these answers scored most recently. Each is answered again twice, with the present default and with the new version, by the same model and without running tools again: the model gets the question and what the tools found that time. For an RCA, both versions write it from the facts of the incident. The evaluator then scores each pair. The new version is **Worse** when its mean is more than 5 below the present default's, **Better** when more than 5 above, **No worse** otherwise. When fewer than 3 answers could be compared, the check cannot tell, and does not hold the prompt back. The dialog shows the progress and then the result for each answer. A check is of one version of the prompt against one default: after the prompt is edited or another default is set, it has to be run again. The model calls of a check count as **Quality scoring**. Prompts for other purposes become the default without a check. Editing the prompt that is already the default is not checked either; to check a change, save it as a new prompt and set that as the default. ## Scores in a stream Each score is also a row of the internal stream `_agent_score_traces`, which you can query in **Logs**: | Field | Description | | - | - | | `target_kind`, `target_id` | `chat_answer` with the message ID, or `incident_rca` with the incident ID. | | `source` | `personal_chat`, `team_chat`, `automation`, or `incident_rca`. | | `scorer`, `scorer_kind` | `rules`, or the evaluator's ID with `evaluator`. | | `score`, `samples` | From 0 to 1, and how many scorings it is the mean of. | | `tool_success`, `evidence`, `query_validity`, `answer_format`, `completion`, `feedback` | The rules, from 0 to 1, when they apply. | | `reason`, `evaluator_model` | The evaluator's reason and model. | | `chat_id`, `team_id`, `automation_id`, `user_id`, `prompt_template_id`, `prompt_version`, `prompt_key`, `provider_id`, `model`, `skill`, `skill_version` | Where the answer came from and what it used. | To hear about a drop in quality, create an [anomaly alert rule](https://docs.molesignal.com/en-US/alerting) over the stream, for example on the evaluators' mean: ```sql theme={null} SELECT avg(score) AS value FROM "_agent_score_traces" WHERE scorer_kind = 'evaluator' ``` This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.