heartbeats

Heartbeats

A lightweight HTTP service for monitoring periodic "heartbeat" pings ("bumps") and notifying configured receivers when a heartbeat goes missing or recovers. Includes an in‐browser read-only dashboard showing current heartbeats, receivers, and historical events.

🚀 Features

• Heartbeat monitoring with configurable interval & grace periods
• Pluggable notifications via Slack, Email, or MS Teams
• In-memory history of recent events (received, failed, state changes, notifications, API requests)
  • Stores up to --history-size entries (default: 10,000)
  • Not persisted across restarts (~1.73 MB at max size)
• Dashboard with:
  • Heartbeats: status, URL, last bump, receivers, quick-links
  • Receivers: type, destination, last sent, status
  • History: timestamped events, filter by heartbeat
  • Text-search filters & copy-to-clipboard URLs
• /healthz and /metrics endpoints for health checks & Prometheus
• YAML configuration with variable resolution via containeroo/resolver

🏁 Flags

Proxy Environment Variables

• HTTP_PROXY: URL of the proxy server to use for HTTP requests
• HTTPS_PROXY: URL of the proxy server to use for HTTPS requests

🌐 Endpoints

⚙️ Configuration

heartbeats and receivers must be defined in your YAML file (default config.yaml).

Examples (config.yaml)

---
receivers:
  dev-crew-int:
    slack_configs:
      - channel: integration
        token: env:SLACK_TOKEN
heartbeats:
  prometheus-int:
    description: "Prometheus → Alertmanager test"
    interval: 15m
    grace: 90s
    receivers:
      - dev-crew-int

Heartbeats

A heartbeat waits for periodic pings ("bumps"). If no bump arrives within interval + grace, notifications are sent.

To reduce noise from race conditions (e.g. pings arriving milliseconds after grace timeout), Heartbeats adds a short internal delay before transitioning to grace or missing. This ensures smoother handling of near-expiry bumps without affecting responsiveness.

Example

heartbeats:
  prometheus-int:
    description: "Prometheus → Alertmanager test"
    interval: 15m
    grace: 90s
    receivers:
      - dev-crew-int

Receivers

Each receiver can have multiple notifier configurations. Supported under receivers::

• slack_configs
• email_configs
• msteams_configs

You may use any template variable from the heartbeat (e.g. {{ .ID }}, {{ .Status }}), and these helper functions:

• upper: {{ upper .ID }}
• lower: {{ lower .ID }}
• formatTime: {{ formatTime .LastBump "2006-01-02 15:04:05" }}
• ago: {{ ago .LastBump }}
• isRecent: {{ isRecent .LastBump }} // isRecent returns true if the last bump was less than 2 seconds ago
• join: {{ join .Tags ", " }}

Variable Resolution

Heartbeats uses containeroo/resolver for variable resolving.

Resolver supports:

• Plain: literal value
• Environment: env:VAR_NAME
• File: file:/path/to/file
• Within-file: file:/path/to/file//KEY, also supported yaml:,json:,ini: and toml:. For more details see containeroo/resolver.

Slack

Defaults:

• subject_tmpl: [{{ upper .Status }}] {{ .ID }}"
• text_tmpl: {{ .ID }} is {{ .Status }} (last bump: {{ ago .LastBump }})"

receivers:
  dev-crew-int:
    slack_configs:
      - channel: "#integration"
        token: env:SLACK_TOKEN
        # optional custom templates:
        title_tmpl: "[{{ upper .Status }}] {{ .ID }}"
        text_tmpl: "{{ .ID }} status: {{ .Status }}"
        # optional: override global skip TLS
        skip_tls: true

Heartbeats adds a custom User-Agent: Heartbeats/<version> header to all outbound HTTP requests. The Content-Type header is also set to application/json.

Email

Defaults:

• subject_tmpl: "[HEARTBEATS]: {{ .ID }} {{ upper .Status }}"
• body_tmpl: "<b>Description:</b> {{ .Description }}<br>Last bump: {{ ago .LastBump }}"

email_configs:
  - smtp:
      host: smtp.gmail.com
      port: 587
      from: [email protected]
      username: env:EMAIL_USER
      password: env:EMAIL_PASS
      # optional
      start_tls: true
      # optional: override global skip TLS
      skip_insecure_verify: true
    email:
      is_html: true
      to: ["[email protected]"]
      # optional custom templates:
      subject_tmpl: "[HB] {{ .ID }} {{ upper .Status }}"
      body_tmpl: "Last bump: {{ ago .LastBump }}"

MS Teams (incomming webhook)

Defaults:

• title_tmpl: "[{{ upper .Status }}] {{ .ID }}"
• text_tmpl: "{{ .ID }} is {{ .Status }} (last bump: {{ ago .LastBump }})"

msteams_configs:
  - webhook_url: file:/secrets/teams/webhook//prod
    # optional custom templates:
    title_tmpl: "[{{ upper .Status }}] {{ .ID }}"
    text_tmpl: "{{ .ID }} status: {{ .Status }}"
    # optional: override global skip TLS
    skip_tls: true

Heartbeats adds a custom User-Agent: Heartbeats/<version> header to all outbound HTTP requests. The Content-Type header is also set to application/json.

MS Teams (Graph API) (NOT TESTED)

Defaults:

• title_tmpl: "[{{ upper .Status }}] {{ .ID }}"
• text_tmpl: "{{ .ID }} is {{ .Status }} (last bump: {{ ago .LastBump }})"

msteamsgraph_configs:
  - channel_id: env:MSTEAMSGRAPH_CHANNEL_ID
    team_id: env:MSTEAMSGRAPH_TEAM_ID
    # optional custom templates:
    title_tmpl: "[{{ upper .Status }}] {{ .ID }}"
    text_tmpl: "{{ .ID }} status: {{ .Status }}"
    # optional: override global skip TLS
    skip_tls: true

Heartbeats adds a custom User-Agent: Heartbeats/<version> header to all outbound HTTP requests. The Content-Type header is also set to application/json.

📈 Metrics

Heartbeats exposes the following Prometheus metrics via the /metrics endpoint:

Heartbeat Metrics

History Store Metrics

🧪 Development

Download the binary and update the example config.yaml according your needs. If you prefer to run heartbeats in docker, you find a docker-compose.yaml & config.yaml here. For a kubernetes deployment you find the manifests here.

🔧 Debugging

Heartbeats includes optional internal endpoints for testing receiver notifications and simulating heartbeats. These are only enabled when the --debug flag is set.

Internal Endpoints

These endpoints listen only on 127.0.0.1.

Kubernetes

Forward the debug port to your local machine:

kubectl port-forward deploy/heartbeats 8081:8081

curl http://localhost:8081/internal/receiver/{id}
curl http://localhost:8081/internal/heartbeat/{id}

Docker

Bind the debug port only to your host's loopback interface:

docker run \
  -p 127.0.0.1:8081:8081 \
  -v ./config.yaml:/config.yaml \
  containeroo/heartbeats

curl http://localhost:8081/internal/receiver/{id}
curl http://localhost:8081/internal/heartbeat/{id}

✅ This ensures /internal/* endpoints are only reachable from your local machine, not from other containers or the network.

⚠️ Warning: These endpoints are meant for local testing and debugging only. Never expose them in production.

📝 License

This project is licensed under the Apache 2.0 License. See the LICENSE file for details.