Synthetic monitoring

Synthetic monitoring tests a service from the outside before a customer reports a failure. Open Synthetics to create checks, publish schedules, inspect execution evidence, and manage private Probe locations.

What can be monitored

Check type	Use it for
HTTP / API	Validate a URL, method, headers, body, response status, content, and timing.
Browser	Run a browser journey and assert page or step outcomes.
TCP	Confirm that a host and port accept a connection.
DNS	Resolve records and validate the returned answer.
ICMP / Ping	Measure basic host reachability and latency.
TLS	Validate a TLS handshake, certificate identity, and expiry.
gRPC	Call a gRPC endpoint and validate the response.
Heartbeat	Expect an external job or service to report before its deadline.

For a gRPC check, write the Address as host:port. Use TLS alone decides whether the call uses TLS. An address with a scheme that contradicts the switch, such as http:// with Use TLS on, is rejected.

Create and publish a check

Create a draft

Open Synthetics → Checks, select Create check, then choose a type. Configure the target, timeout, assertions, schedule, and one or more locations. A new check starts as a draft.

Test the revision

Select Test revision. MoleSignal runs the draft without activating its schedule. Review the resolved target, attempts, timings, assertion evidence, and errors.

Publish the revision

A successful test enables Publish revision. Publishing makes the revision active and calculates its first scheduled run.

Operate the monitor

Use Run now for an on-demand execution. Editing creates another draft revision, so the published configuration continues running until the replacement is tested and published. Pause, resume, or archive the check without deleting its result history.

The draft gate prevents an untested target or assertion from silently replacing a working production schedule.

Schedule and retries

Use an interval or a cron expression with an explicit timezone. Most active checks accept intervals from 10 seconds to one day; browser checks start at 60 seconds. Set the timeout between 100 milliseconds and five minutes, and keep it shorter than the interval. A run can retry up to two times before MoleSignal records its final outcome. Heartbeat checks use a reporting deadline rather than an outbound schedule. Send the heartbeat from the workload that owns the job, then use the result history to distinguish a late report from a failed active probe.

Assertions and health

Assertions inspect protocol-specific fields such as response status, body, headers, DNS answers, certificate properties, or browser outcomes. A failed Warning assertion produces a degraded result; a failed Critical assertion produces a failing result. The monitor health state is calculated from consecutive successes and failures across its selected locations. Configure the failure threshold, recovery threshold, and location policy to avoid flapping or declaring a global outage from one isolated network.

Locations and Probe Agents

MoleSignal supports two location models:

Location	Behavior
Embedded Local	A Standalone deployment starts an in-process Probe runner automatically. It is useful for local evaluation and checks that should originate beside MoleSignal.
Agent Pool	One or more external `probe-agent` processes execute checks from a private network, region, or edge location.

Create an Agent Pool location under Synthetics → Locations, generate a one-time register token, and register the Agent:

probe-agent register \
  --endpoint https://molesignal.example.com:5084 \
  --token ONE_TIME_TOKEN \
  --ca-certificate-base64 BASE64_CA

Then start its control stream:

probe-agent run

Registration uses server-authenticated TLS. The Agent stores a private identity under —state-dir and uses mTLS for the control stream on port 5085. The default state directory is /var/lib/molesignal-probe. Configure the server endpoints in conf/config.toml: The Probe control plane starts automatically on Standalone and Alert nodes; it has no separate enable switch.

[probe]
bind = "0.0.0.0"
register_port = 5084
control_port = 5085
register_endpoint = "https://molesignal.example.com:5084"
control_endpoint = "https://molesignal.example.com:5085"
server_names = ["molesignal.example.com"]

An egress policy applies before every network request. Loopback, link-local, or private addresses are rejected unless the selected location policy explicitly allows them. This protects the MoleSignal host and private network from server-side request forgery.

HTTP checks honor HTTP_PROXY, HTTPS_PROXY, ALL_PROXY, and NO_PROXY in the environment of the Probe Agent. Through a proxy, the connect time of a result measures the connection to the proxy, including the request for the tunnel. The proxy makes the connection to the target, so the address that the egress policy approved is not enforced for the target. Limit what the proxy can reach.

Variables, secrets, and evidence

Use variables for reusable non-sensitive values and encrypted secrets for credentials. Secret values are write-only and never returned by the API. Result detail records bounded request and response evidence, timings, attempt metadata, the selected location, and the active revision without exposing stored secrets.

Alert incidents

A failing check opens an alert incident, which pages the people on call the same way a rule does. Set it up under Alerting in the check’s form.

Check state	Incident
Failed	Opens one, severity `error`.
Degraded	Opens one, severity `warning`, when Alert when degraded is on.
Flaky	Opens one, severity `warning`, when Alert when flaky is on.
Passed	Resolves it.

A check has at most one active incident. When the check moves between failed, degraded, and flaky, the open incident takes the new severity, and a higher one restarts escalation. An acknowledged incident stays as it is until someone resolves it.
The check’s Escalation policy pages people. With No escalation policy nobody is paged, and the notify policies still deliver the incident.
The incident is titled with the check’s name. It carries the labels alert.source = synthetic, synthetic.kind, and synthetic.state, and synthetic.tags and team_id when the check has tags and a team, so notify policies can route it. Its detail links back to the check.
Notifications are titled with the severity and the check’s name, for example ERROR · Checkout API, and say Synthetic check Checkout API is failing.
Only the state of the whole check counts: the state its consecutive failures, recoveries, and location policy give. A single location does not open an incident.

Publish failures to a status page

A status-page automation rule can use Synthetic monitor as its source. Select the monitor, match its transition and severity, map the impact to components, and choose either manual approval or automatic publishing. See Status pages.

Permissions

Permission	Capability
`synthetics.read`	Read checks, revisions, schedules, runs, results, locations, and Agent state.
`synthetics.manage`	Create, test, publish, pause, resume, run, and archive checks.
`synthetics.locations.manage`	Manage locations, Agent pools, register tokens, and egress policies.
`synthetics.secrets.manage`	Create, rotate, and remove encrypted synthetic-monitoring secrets.

Status pages

Turn monitor transitions into reviewed or automatic customer communication.

Alerting

Route operational incidents to responders before publishing customer updates.

Get started

Data

Observe

Query and correlate

Reliability

AI ops

Collaboration

Admin

Deploy & operate

Best practices

Resources

What can be monitored

Create and publish a check

Schedule and retries

Assertions and health

Locations and Probe Agents

Variables, secrets, and evidence

Alert incidents

Publish failures to a status page

Permissions

Status pages

Alerting

​What can be monitored

​Create and publish a check

​Schedule and retries

​Assertions and health

​Locations and Probe Agents

​Variables, secrets, and evidence

​Alert incidents

​Publish failures to a status page

​Permissions

Status pages

Alerting

What can be monitored

Create and publish a check

Schedule and retries

Assertions and health

Locations and Probe Agents

Variables, secrets, and evidence

Alert incidents

Publish failures to a status page

Permissions