> ## Documentation Index
> Fetch the complete documentation index at: https://docs.molesignal.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Connectors
> Receive logs straight from Kinesis Firehose, Cloudflare Logpush, and Heroku — and fan pipeline output out to S3 or Kafka.
A **connector** is a managed integration with an external platform. Connectors work in two
directions:
* **Intake sources** — a platform pushes data **into** MoleSignal (Kinesis Firehose, Cloudflare
Logpush, Heroku log drains).
* **Egress sinks** — a [pipeline](/en-US/pipelines) fans pipeline output **out** to S3 or Kafka.
## Configure a connector
Create connectors under **Pipelines → Connectors**, or over the API. Each has a `kind` and a
`config_json` whose shape depends on the kind. Sensitive fields (`push_token`, `access_key`,
`secret_key`, …) are masked in API responses.
```bash theme={null}
curl -X POST http://localhost:5080/api/v1/connectors \
-H "authorization: Bearer $MS_JWT" \
-H 'content-type: application/json' \
-d '{
"name": "cf-prod",
"kind": "cloudflare_logpush",
"config_json": { "push_token": "ms_cf_9f3c…", "target_stream": "cloudflare" }
}'
```
## Intake sources (push)
The three push sources share one model: create a connector with a **`push_token`** and a
**`target_stream`**, then point the platform at the matching endpoint. These endpoints do not use the
JWT — the request authenticates with the connector token, supplied whichever way the platform allows:
| Method | Used by |
| ---------------------------------- | ----------------------------------------------------- |
| `X-Connector-Token` header | Platforms with configurable headers (e.g. Cloudflare) |
| `X-Amz-Firehose-Access-Key` header | Kinesis Firehose (sent automatically) |
| `?token=` query parameter | Platforms that can't set headers (e.g. Heroku) |
Bodies sent with `Content-Encoding: gzip` are decompressed automatically. Events flow through the
same intake path as everything else — schema-on-write, [pipelines](/en-US/pipelines), and
[masking](/en-US/security) all apply.
Create the connector with `kind: "aws_kinesis_firehose"`, then add an **HTTP endpoint**
destination to the Firehose delivery stream:
* **URL** — `https://molesignal.example.com/api/v1/_kinesis_firehose`
* **Access key** — the connector `push_token` (Firehose sends the value as `X-Amz-Firehose-Access-Key`)
MoleSignal base64-decodes each record, splits the record on newlines, and parses each line as JSON
(falling back to a `message` field for plain text). MoleSignal returns the `200` ACK expected by Firehose.
Create the connector with `kind: "cloudflare_logpush"`, then create a Logpush job whose
destination is:
* **URL** — `https://molesignal.example.com/api/v1/_cloudflare`
* **Header** — `X-Connector-Token: `
Cloudflare ships gzip-compressed NDJSON; each line becomes one event. The endpoint replies
`204 No Content`.
Create the connector with `kind: "heroku_drain"`, then add the drain — Heroku can't set custom
headers, so the token rides in the URL:
```bash theme={null}
heroku drains:add "https://molesignal.example.com/api/v1/_heroku?token="
```
Each log line becomes an event with `message` and `source: "heroku"`. The endpoint replies
`204 No Content`.
The `target_stream` is created on first delivery, and the schema evolves as new fields appear — same
as any other stream. Omit `target_stream` to fall back to a per-source default (`kinesis`, `cloudflare`,
`heroku`).
## Pull sources
CloudWatch Logs is a **pull** source: MoleSignal polls the AWS API on a schedule instead of receiving
a push. Create a connector and MoleSignal periodically calls `FilterLogEvents` (signed with SigV4 — no
AWS SDK needed), advancing a per-connector checkpoint so each run reads only new events.
```bash theme={null}
curl -X POST http://localhost:5080/api/v1/connectors \
-H "authorization: Bearer $MS_JWT" \
-H 'content-type: application/json' \
-d '{
"name": "cw-prod",
"kind": "aws_cloudwatch_logs",
"config_json": {
"region": "ap-southeast-1",
"log_group": "/aws/lambda/checkout",
"access_key": "AKIA…",
"secret_key": "…",
"target_stream": "cloudwatch"
}
}'
```
| Field | Notes |
| --------------------------- | ------------------------------------------------------- |
| `region` / `log_group` | Required — the CloudWatch region and log group to read. |
| `access_key` / `secret_key` | IAM credentials with `logs:FilterLogEvents`. |
| `session_token` | Optional, for temporary STS credentials. |
| `target_stream` | Destination stream (default `default`). |
The poller runs as a **singleton** (on the alert-manager / standalone node), so exactly one node polls
regardless of cluster size. Each event becomes a log with `message`, `log_stream`, and `log_group`.
## Egress sinks
S3 and Kafka connectors are **sinks**: select these connectors in a [pipeline](/en-US/pipelines#sinks-and-connector-egress)
to fan transformed events out as the pipeline runs.
| Kind | Payload |
| ------- | --------------------------------------------------------------- |
| `s3` | Batched JSON Lines `PUT` to `s3:///.jsonl`. |
| `kafka` | One JSON record produced per event to the configured topic. |
## Permissions
`pipelines.read` lists and inspects connectors. `pipelines.edit` creates, updates, or deletes connectors.
Push receivers authenticate with the connector token and do not accept a normal login JWT as a
replacement.
Endpoint details for the Kinesis, Cloudflare, and Heroku push receivers.