Kubernetes Ingress

The configuration below follows the recommended way in Public access: two ingress-nginx controllers run in the cluster, one behind an internal load balancer and one behind a public load balancer, and one name, molesignal.example.com, resolves to each of them in the internal and the public DNS. MoleSignal’s Service is molesignal on port 5080 in the namespace molesignal; in a role-split deployment, use the router’s Service instead.

Two controllers

The two controllers need different IngressClasses and different controllerValues, or each takes the other’s Ingresses. The internal controller’s Service carries the cloud provider’s internal load balancer annotation:

# ingress-internal-values.yaml
controller:
  ingressClassResource:
    name: nginx-internal
    controllerValue: k8s.io/ingress-nginx-internal
  ingressClass: nginx-internal
  electionID: ingress-nginx-internal-leader
  service:
    annotations:
      # Per cloud provider, for example Alibaba Cloud ACK:
      service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: intranet

The public controller keeps the client source address and logs paths, never query strings; heartbeat and calendar paths carry a token, so only their prefix is logged:

# ingress-public-values.yaml
controller:
  ingressClassResource:
    name: nginx-public
    controllerValue: k8s.io/ingress-nginx-public
  ingressClass: nginx-public
  electionID: ingress-nginx-public-leader
  service:
    externalTrafficPolicy: Local
  config:
    http-snippet: |
      map $uri $molesignal_log_uri {
          ~^(/api/v1/(?:intake/heartbeats|public/calendar)/)  $1<token>;
          default                                             $uri;
      }
    log-format-upstream: '$remote_addr [$time_local] "$request_method $molesignal_log_uri $server_protocol" $status $body_bytes_sent "$http_user_agent" $request_time'

Install each:

helm upgrade --install ingress-internal ingress-nginx --repo https://kubernetes.github.io/ingress-nginx --namespace ingress-internal --create-namespace -f ingress-internal-values.yaml

helm upgrade --install ingress-public ingress-nginx --repo https://kubernetes.github.io/ingress-nginx --namespace ingress-public --create-namespace -f ingress-public-values.yaml

Internal Ingress

The internal Ingress forwards every path. Live queries are streamed responses, so buffering is off and the timeouts are long:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: molesignal-internal
  namespace: molesignal
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "10m"
    nginx.ingress.kubernetes.io/proxy-buffering: "off"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
spec:
  ingressClassName: nginx-internal
  tls:
    - hosts: [molesignal.example.com]
      secretName: molesignal-tls
  rules:
    - host: molesignal.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: molesignal
                port:
                  number: 5080

Public Ingress

The public Ingress lists only the paths to open; the controller’s default backend answers 404 for every path not listed. Keep only the paths in use:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: molesignal-public
  namespace: molesignal
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "10m"
    nginx.ingress.kubernetes.io/limit-rps: "20"
spec:
  ingressClassName: nginx-public
  tls:
    - hosts: [molesignal.example.com]
      secretName: molesignal-tls
  rules:
    - host: molesignal.example.com
      http:
        paths:
          # External alerts
          - { path: /api/v1/intake/alerts, pathType: Exact, backend: { service: { name: molesignal, port: { number: 5080 } } } }
          # Heartbeat check-ins, the Jira ticket webhook, on-call calendar subscriptions
          - { path: /api/v1/intake/heartbeats, pathType: Prefix, backend: { service: { name: molesignal, port: { number: 5080 } } } }
          - { path: /api/v1/public/tickets/jira, pathType: Prefix, backend: { service: { name: molesignal, port: { number: 5080 } } } }
          - { path: /api/v1/public/calendar, pathType: Prefix, backend: { service: { name: molesignal, port: { number: 5080 } } } }
          # Mole Agent in Teams, automation API triggers, replies to texts and calls
          - { path: /api/v1/public/im/teams, pathType: Prefix, backend: { service: { name: molesignal, port: { number: 5080 } } } }
          - { path: /api/v1/public/automations, pathType: Prefix, backend: { service: { name: molesignal, port: { number: 5080 } } } }
          - { path: /api/v1/public/notify, pathType: Prefix, backend: { service: { name: molesignal, port: { number: 5080 } } } }
          # Push sources
          - { path: /api/v1/_kinesis_firehose, pathType: Exact, backend: { service: { name: molesignal, port: { number: 5080 } } } }
          - { path: /api/v1/_cloudflare, pathType: Exact, backend: { service: { name: molesignal, port: { number: 5080 } } } }
          - { path: /api/v1/_heroku, pathType: Exact, backend: { service: { name: molesignal, port: { number: 5080 } } } }
          # RUM
          - { path: /api/v1/rum/events, pathType: Exact, backend: { service: { name: molesignal, port: { number: 5080 } } } }
          - { path: /api/v1/rum/replay, pathType: Exact, backend: { service: { name: molesignal, port: { number: 5080 } } } }
          # Synthetics Probe Agent artifact uploads
          - { path: /api/v1/synthetics/artifacts, pathType: Prefix, backend: { service: { name: molesignal, port: { number: 5080 } } } }
          # Status pages
          - { path: /status, pathType: Prefix, backend: { service: { name: molesignal, port: { number: 5080 } } } }
          - { path: /api/v1/public/status-pages, pathType: Prefix, backend: { service: { name: molesignal, port: { number: 5080 } } } }
          # Resource sharing
          - { path: /s, pathType: Prefix, backend: { service: { name: molesignal, port: { number: 5080 } } } }
          - { path: /shared, pathType: Exact, backend: { service: { name: molesignal, port: { number: 5080 } } } }
          - { path: /api/v1/public/share, pathType: Prefix, backend: { service: { name: molesignal, port: { number: 5080 } } } }
          # Web files of status pages and shared resources
          - { path: /assets, pathType: Prefix, backend: { service: { name: molesignal, port: { number: 5080 } } } }
          - { path: /favicon.ico, pathType: Exact, backend: { service: { name: molesignal, port: { number: 5080 } } } }
          - { path: /favicon.svg, pathType: Exact, backend: { service: { name: molesignal, port: { number: 5080 } } } }

Prefix matches by path element: /s matches /s/<share token> but not /shared, and /api/v1/public/share matches /api/v1/public/share/query. An Ingress cannot filter by method, so MoleSignal does: each path above takes only the methods it uses and answers 405 or 404 to the rest. Annotations apply to a whole Ingress object: to give some paths their own body limit or source addresses, move them to another Ingress, for example to let only the ranges of the Azure service tag AzureBotService reach the Teams callback:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: molesignal-public-teams
  namespace: molesignal
  annotations:
    nginx.ingress.kubernetes.io/whitelist-source-range: "203.0.113.0/24,198.51.100.0/24"
spec:
  ingressClassName: nginx-public
  tls:
    - hosts: [molesignal.example.com]
      secretName: molesignal-tls
  rules:
    - host: molesignal.example.com
      http:
        paths:
          - { path: /api/v1/public/im/teams, pathType: Prefix, backend: { service: { name: molesignal, port: { number: 5080 } } } }

Remove a moved path from molesignal-public; one path belongs in one Ingress only.

Custom domains of status pages

When a status page uses status.example.org, add a rule for it on the public controller that opens only the status page’s root paths, endpoints and web files:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: molesignal-status-domain
  namespace: molesignal
  annotations:
    nginx.ingress.kubernetes.io/limit-rps: "20"
spec:
  ingressClassName: nginx-public
  tls:
    - hosts: [status.example.org]
      secretName: status-example-org-tls
  rules:
    - host: status.example.org
      http:
        paths:
          - { path: /, pathType: Exact, backend: { service: { name: molesignal, port: { number: 5080 } } } }
          - { path: /history, pathType: Exact, backend: { service: { name: molesignal, port: { number: 5080 } } } }
          - { path: /uptime, pathType: Exact, backend: { service: { name: molesignal, port: { number: 5080 } } } }
          - { path: /incidents, pathType: Prefix, backend: { service: { name: molesignal, port: { number: 5080 } } } }
          - { path: /api/v1/public/status-pages, pathType: Prefix, backend: { service: { name: molesignal, port: { number: 5080 } } } }
          - { path: /assets, pathType: Prefix, backend: { service: { name: molesignal, port: { number: 5080 } } } }
          - { path: /favicon.ico, pathType: Exact, backend: { service: { name: molesignal, port: { number: 5080 } } } }
          - { path: /favicon.svg, pathType: Exact, backend: { service: { name: molesignal, port: { number: 5080 } } } }

Check

Confirm that each controller took its own Ingresses and that they have the addresses of their load balancers:

kubectl -n molesignal get ingress

From the internet, the sign-in endpoint answers 404, and alert intake without a token answers 401:

curl -s -o /dev/null -w '%{http_code}\n' https://molesignal.example.com/api/v1/auth/signin

curl -s -o /dev/null -w '%{http_code}\n' -X POST https://molesignal.example.com/api/v1/intake/alerts

Finally, in the _sys organization, list the network of the public controller’s pods as trusted proxies under Settings → Client IP recognition, see Client IP recognition.

Get started

Data

Observe

Query and correlate

Reliability

AI ops

Collaboration

Admin

Deploy & operate

Security

Best practices

Resources

Two controllers

Internal Ingress

Public Ingress

Custom domains of status pages

Check

​Two controllers

​Internal Ingress

​Public Ingress

​Custom domains of status pages

​Check

Two controllers

Internal Ingress

Public Ingress

Custom domains of status pages

Check