Log inspections

A log inspection has Mole Agent read the logs of a period and write a report: whether something is wrong, what it is, and which logs show it. An inspection only reads logs. It changes no data and no configuration.

Kind	Use
On demand	Inspect the logs that the current query matches, once, from the AI inspection view of Logs. The report streams onto the page. The inspection is a Mole Agent chat, so it can continue in Mole Agent. It is not kept with the scheduled reports.
Scheduled	Save a stream, a query, and what to attend to. MoleSignal inspects on the schedule, keeps every report, and notifies when a report says something is wrong.

Before starting

Enable at least one AI model under Mole Agent → Settings → Models & reasoning. See Mole Agent. Without an enabled model, saved inspections stay listed and cannot run.
Check the permissions:

Task	Permission
Inspect on demand, and read scheduled inspections and their reports	`agent.use` and `streams.query`
Create, edit, switch on or off, delete, and run now	`inspections.manage`, granted to the owner, admin, and editor roles by default
Choose a notification channel in the form	`alerts.read`, which lists the notification connectors

Mole Agent reads the same inspections and reports in a chat with list_log_inspections, list_log_inspection_runs, and get_log_inspection_run, under the same permissions. run_log_inspection runs one now: it needs inspections.manage and, by default, a confirmation in the chat. See Tool catalog and execution.

Inspect on demand

Open Logs, select a stream and a time range, and run a query. Leave the query empty to inspect every log of the stream.
Open the AI inspection view.
Choose the AI model. Optionally describe What to attend to, such as the service to watch or an outcome to treat as expected.
Select Start inspection.

The inspection sends query results of these logs to the chosen model. Select Continue in Mole Agent to ask follow-up questions about the report.

Save a scheduled inspection

Open Logs → AI inspection, then select either entry:

Save as scheduled inspection opens the form with the current stream, query, requirements, and model. The inspection can be saved even when the query matches nothing now.
Scheduled inspections opens the list, where New inspection starts an empty form.

Field	Meaning
Name	Unique in the organization. At most 120 characters.
Stream	One log stream. The inspection reads this stream only.
Query	Optional. Only logs that match are inspected. The syntax is the same as on Logs.
What to attend to	Optional, at most 2,000 characters. What matters, and what counts as normal.
Schedule	At an interval, Daily, or Weekly at a time of day. The shortest interval is an instance setting, 1 hour by default.
Period covered	How much time before each run is inspected: 15 minutes to 7 days in the form. The default follows the schedule.
AI model	One enabled model of the organization.
When to notify	Notify when abnormal (default), Notify on attention or worse, or Notify every time.
Notification channels	Optional, at most 10. Each is a fixed group or address on a notification connector.
Run on the schedule once saved	When off, the inspection is saved and runs only on Run now.

A daily or weekly time is read in the time zone of the interface at the time of creation: the time zone of the personal preferences, or the time zone of the browser when none is set. The form names the time zone, and a later edit does not change it. Notifications and reports use the interface language at the time of creation. An organization keeps at most 20 scheduled inspections by default.

How a run works

Aspect	Behavior
Identity	A run reads the logs as the user who saved the inspection last. Before every run, MoleSignal checks that this user can still sign in, is still a member of the organization, and still has `agent.use` and `streams.query`. Otherwise the run fails with the reason that the identity can no longer be used, and another user with the permission saves the inspection again to run it as that user. An API Token cannot save a scheduled inspection.
Tools	The model uses three read-only tools: `query_logs`, `get_stream_schema`, and `search_field_values`, and only on the stream of the inspection. A tool that the Tool Policy of the organization disables or puts behind approval is not used, because nobody approves in the background.
Time window	A run covers the period just before it starts. Runs that a stopped node missed are not made up: once the node is back, a due inspection runs once and the schedule continues from there.
Limits	A run takes at most 5 minutes by default, after which it counts as timed out. A tool result above 64 KiB is not handed to the model, which is told to narrow the query instead. A failed run is not retried before the next scheduled run.
Concurrency	One inspection has one run at a time. One node runs at most 2 inspections at once by default.
Run now	The run is queued and starts within 15 seconds, under the identity of the inspection. The record names the user who started it. A switched-off inspection can also run this way.

Reports and verdicts

Select an inspection in the list to open its reports. Past runs are on the left. The selected run shows the report, the period covered, the duration, the tools called, the tokens used, and the notifications sent. Open the logs of this period opens Logs on the same stream, query, and period. A scheduled report carries a verdict:

Verdict	Meaning
Normal	Nothing that needs action.
Needs attention	Something deserves a look, and nothing is failing.
Abnormal	Something is wrong and needs action.
No verdict	The answer of the model carried no readable verdict. The report is kept, and notification treats it as abnormal.

A run that did not finish has no report. The record shows its state and the reason:

State	Reason
Timed out	The run exceeded its time limit.
Failed	The identity can no longer be used; the chosen AI model is missing, disabled, or has no key; the model call failed; the organization’s Mole Agent token budget for today is used up; the node stopped during the run; no node picked up the run; or an internal error occurred.

Reports are kept for 30 days by default and removed hourly after that. Deleting an inspection deletes all its reports.

Notifications

When the verdict of a report meets When to notify, each channel receives one notification: the title states the verdict and the name of the inspection, and the body holds the stream, the period, the verdict, the summary of the report, and a link to the report. The link needs http.external_url to be set.
A failed run notifies once, when the run before it was not a failure. Repeated failures notify only the first time, whatever When to notify says.
Without a channel, nothing is sent and the reports are still kept.
Every notification appears under Settings → Notify → Delivery log with the stage Direct. A direct delivery does not pass through notification policies and cannot be retried from the delivery log.
A notification connector that a scheduled inspection uses cannot be deleted.

Instance settings

Setting	Default	Purpose
`[log_inspection].min_interval_secs`	`3600`	The shortest time between two runs of one inspection. Values below 60 count as 60.
`[log_inspection].max_per_org`	`20`	Scheduled inspections that one organization can keep. `0` allows none.
`[log_inspection].concurrent_runs`	`2`	Inspections that one node runs at the same time.
`[log_inspection].run_timeout_secs`	`300`	The time limit of one run, including every model call and query. Clamped to 30–480.
`[log_inspection].report_retention_days`	`30`	How long reports are kept.

Nodes with the alert role run scheduled inspections and check every 15 seconds for a due or queued run. With several alert nodes, each run is picked up by one node only. See Role-based deployment. Two metrics follow the runs:

Metric	Meaning
`log_inspection_runs_total{outcome}`	Finished runs by outcome: `succeeded`, `failed`, or `timed_out`.
`log_inspection_run_duration_seconds`	The duration of a run.

API

The endpoints are under /api/v1/logs/inspections:

Method and path	Purpose
`GET /logs/inspections`	List the scheduled inspections of the organization.
`POST /logs/inspections`	Create an inspection.
`GET /logs/inspections/limits`	The limits that the instance sets.
`GET`, `PUT`, `DELETE /logs/inspections/{id}`	Read, replace, or delete an inspection with its reports.
`POST /logs/inspections/{id}/enabled`	Switch an inspection on or off.
`POST /logs/inspections/{id}/run`	Run once now.
`GET /logs/inspections/{id}/runs`	Past runs, newest first, without the report text. `limit` defaults to 50 and is at most 200. `before` pages to older runs.
`GET /logs/inspections/{id}/runs/{run_id}`	One run with its report.

Logs

Write the query that an inspection reads.

Notification channels

Connect the channels that receive the notifications.

Get started

Data

Observe

Query and correlate

Reliability

AI ops

Collaboration

Admin

Deploy & operate

Best practices

Resources

Before starting

Inspect on demand

Save a scheduled inspection

How a run works

Reports and verdicts

Notifications

Instance settings

API

Logs

Notification channels

​Before starting

​Inspect on demand

​Save a scheduled inspection

​How a run works

​Reports and verdicts

​Notifications

​Instance settings

​API

Logs

Notification channels

Before starting

Inspect on demand

Save a scheduled inspection

How a run works

Reports and verdicts

Notifications

Instance settings

API