Connectors

A connector is a managed integration with an external platform. Connectors work in two directions:

Intake sources — a platform pushes data into MoleSignal (Kinesis Firehose, Cloudflare Logpush, Heroku log drains).
Egress sinks — a pipeline fans pipeline output out to S3 or Kafka.

Configure a connector

Create connectors under Pipelines → Connectors, or over the API. Each has a kind and a config_json whose shape depends on the kind. Sensitive fields (push_token, access_key, secret_key, …) are masked in API responses.

curl -X POST http://localhost:5080/api/v1/connectors \
  -H "authorization: Bearer $MS_JWT" \
  -H 'content-type: application/json' \
  -d '{
        "name": "cf-prod",
        "kind": "cloudflare_logpush",
        "config_json": { "push_token": "ms_cf_9f3c…", "target_stream": "cloudflare" }
      }'

Intake sources (push)

The three push sources share one model: create a connector with a push_token and a target_stream, then point the platform at the matching endpoint. These endpoints do not use the JWT — the request authenticates with the connector token, supplied whichever way the platform allows:

Method	Used by
`X-Connector-Token` header	Platforms with configurable headers (e.g. Cloudflare)
`X-Amz-Firehose-Access-Key` header	Kinesis Firehose (sent automatically)
`?token=` query parameter	Platforms that can’t set headers (e.g. Heroku)

Bodies sent with Content-Encoding: gzip are decompressed automatically. Events flow through the same intake path as everything else — schema-on-write, pipelines, and masking all apply.

Kinesis Firehose
Cloudflare Logpush
Heroku

Create the connector with kind: "aws_kinesis_firehose", then add an HTTP endpoint destination to the Firehose delivery stream:

URL — https://molesignal.example.com/api/v1/_kinesis_firehose
Access key — the connector push_token (Firehose sends the value as X-Amz-Firehose-Access-Key)

MoleSignal base64-decodes each record, splits the record on newlines, and parses each line as JSON (falling back to a message field for plain text). MoleSignal returns the 200 ACK expected by Firehose.

Create the connector with kind: "cloudflare_logpush", then create a Logpush job whose destination is:

URL — https://molesignal.example.com/api/v1/_cloudflare
Header — X-Connector-Token: <push_token>

Cloudflare ships gzip-compressed NDJSON; each line becomes one event. The endpoint replies 204 No Content.

Create the connector with kind: "heroku_drain", then add the drain — Heroku can’t set custom headers, so the token rides in the URL:

heroku drains:add "https://molesignal.example.com/api/v1/_heroku?token=<push_token>"

Each log line becomes an event with message and source: "heroku". The endpoint replies 204 No Content.

The target_stream is created on first delivery, and the schema evolves as new fields appear — same as any other stream. Omit target_stream to fall back to a per-source default (kinesis, cloudflare, heroku).

Pull sources

CloudWatch Logs is a pull source: MoleSignal polls the AWS API on a schedule instead of receiving a push. Create a connector and MoleSignal periodically calls FilterLogEvents (signed with SigV4 — no AWS SDK needed), advancing a per-connector checkpoint so each run reads only new events.

curl -X POST http://localhost:5080/api/v1/connectors \
  -H "authorization: Bearer $MS_JWT" \
  -H 'content-type: application/json' \
  -d '{
        "name": "cw-prod",
        "kind": "aws_cloudwatch_logs",
        "config_json": {
          "region": "ap-southeast-1",
          "log_group": "/aws/lambda/checkout",
          "access_key": "AKIA…",
          "secret_key": "…",
          "target_stream": "cloudwatch"
        }
      }'

Field	Notes
`region` / `log_group`	Required — the CloudWatch region and log group to read.
`access_key` / `secret_key`	IAM credentials with `logs:FilterLogEvents`.
`session_token`	Optional, for temporary STS credentials.
`target_stream`	Destination stream (default `default`).

The poller runs as a singleton (on the alert-manager / standalone node), so exactly one node polls regardless of cluster size. Each event becomes a log with message, log_stream, and log_group.

Egress sinks

S3 and Kafka connectors are sinks: select these connectors in a pipeline to fan transformed events out as the pipeline runs.

Kind	Payload
`s3`	Batched JSON Lines `PUT` to `s3://<bucket>/<prefix><id>.jsonl`.
`kafka`	One JSON record produced per event to the configured topic.

Permissions

pipelines.read lists and inspects connectors. pipelines.edit creates, updates, or deletes connectors. Push receivers authenticate with the connector token and do not accept a normal login JWT as a replacement.

Intake API

Endpoint details for the Kinesis, Cloudflare, and Heroku push receivers.

Getting Started

Observe

Reliability

AI Ops

Data

Collaboration

Admin

Deploy & Operate

Configure a connector

Intake sources (push)

Pull sources

Egress sinks

Permissions

Intake API

​Configure a connector

​Intake sources (push)

​Pull sources

​Egress sinks

​Permissions

Intake API

Configure a connector

Intake sources (push)

Pull sources

Egress sinks

Permissions