Grafana integration¶
Everything Grafana shows is provisioned from grafana/: three datasources, thirteen dashboards, thirteen alert rules, an email contact point and a heartbeat webhook. Locally it's at http://localhost:3000; in production nginx serves it on grafana.integr8scode.cc (see Nginx configuration). Anonymous access is off. The admin login comes from GRAFANA_ADMIN_USER and GRAFANA_ADMIN_PASSWORD (default admin / admin123), and the entrypoint resets the password to that value on every start.
Datasources¶
| Name | Type | UID | Notes |
|---|---|---|---|
| Victoria Metrics | prometheus | victoria-metrics |
default; exemplar trace_id links to Jaeger |
| Jaeger | jaeger | jaeger |
needs Jaeger below 2.21, see Tracing |
| Loki | loki | loki |
trace_id derived field links to Jaeger |
| OpenSearch | grafana-opensearch-datasource | opensearch |
daily logs- indices; traceId links to Jaeger, see OpenSearch |
Grafana 13 downloads the prometheus, jaeger and grafana-opensearch-datasource plugins on first start. preinstall_sync in grafana.ini makes it install them before provisioning runs, so the first start on a fresh volume needs to reach grafana.com.
Dashboards¶
All dashboards land in one folder, Integr8sCode, with the fixed UID integr8scode. The file provider reloads them every 10 seconds and doesn't allow saving edits from the UI. The metrics behind them are listed in the metrics reference.
| File | Title |
|---|---|
| integr8scode.json | System Overview |
| coordinator-execution.json | Queue & Execution |
| dlq-monitoring.json | Dead Letter Queue |
| event-replay.json | Event Replay |
| event-stream-monitoring.json | Event Stream |
| http-middleware.json | HTTP & Middleware |
| kafka-events-monitoring.json | Kafka Events |
| kubernetes-pods.json | Kubernetes & Pods |
| mongodb-monitoring.json | MongoDB Monitoring |
| notifications.json | Notifications |
| rate-limiting-dashboard.json | Rate Limiting |
| security-auth.json | Security & Authentication |
| errors.json | Errors |
Alerting¶
provisioning/alerting/alerting.yml holds the rules, the contact point and the policy:
apiVersion: 1
contactPoints:
- orgId: 1
name: email-alerts
receivers:
- uid: email-alerts-receiver
type: email
settings:
addresses: ${GF_SMTP_ALERT_RECIPIENTS}
singleEmail: true
subject: '{{ template "email.subject" . }}'
message: '{{ template "email.message" . }}'
# An outside dead man's switch; when its pings stop, the host or Grafana is down
- orgId: 1
name: heartbeat
receivers:
- uid: heartbeat-receiver
type: webhook
settings:
url: ${ALERT_HEARTBEAT_URL}
httpMethod: POST
disableResolveMessage: true
# One line per alert from its summary, under the rule's description when it has one
templates:
- orgId: 1
name: email
template: |
{{ define "email.subject" }}[{{ .Status | toUpper }}] {{ .CommonLabels.alertname }}{{ with .Alerts.Firing }}{{ with (index . 0).Annotations.description }}: {{ . }}{{ end }}{{ end }}{{ end }}
{{ define "email.message" }}{{ with .Alerts.Firing }}{{ with (index . 0).Annotations.description }}{{ . }}:
{{ end }}{{ range . }}- {{ .Annotations.summary }}
{{ end }}{{ end }}{{ with .Alerts.Resolved }}
Resolved:
{{ range . }}- {{ .Annotations.summary }}
{{ end }}{{ end }}
{{ with index .Alerts 0 }}{{ or .PanelURL .GeneratorURL }}{{ end }}{{ end }}
policies:
- orgId: 1
receiver: email-alerts
group_by: ['grafana_folder', 'alertname']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
routes:
- receiver: heartbeat
object_matchers:
- ['heartbeat', '=', 'true']
group_wait: 0s
group_interval: 1m
repeat_interval: 5m
# The email links a rule's panel only through the __dashboardUid__ and __panelId__ annotations; dashboardUid and panelId alone don't reach it
groups:
- orgId: 1
name: infrastructure-alerts
folder: 'Integr8sCode'
interval: 1m
rules:
- uid: host-memory-warning
title: "Host Memory > 85%"
condition: C
data:
- refId: A
relativeTimeRange:
from: 300
to: 0
datasourceUid: victoria-metrics
model:
expr: avg_over_time(system_memory_utilization_ratio{state="used"}[5m]) * 100
instant: true
range: false
intervalMs: 15000
maxDataPoints: 43200
- refId: C
relativeTimeRange:
from: 300
to: 0
datasourceUid: __expr__
model:
type: threshold
conditions:
- evaluator:
type: gt
params: [85]
expression: A
for: 5m
annotations:
summary: "Host memory usage is above 85%"
labels:
severity: warning
- uid: host-memory-critical
title: "Host Memory > 95%"
condition: C
data:
- refId: A
relativeTimeRange:
from: 300
to: 0
datasourceUid: victoria-metrics
model:
expr: avg_over_time(system_memory_utilization_ratio{state="used"}[5m]) * 100
instant: true
range: false
intervalMs: 15000
maxDataPoints: 43200
- refId: C
relativeTimeRange:
from: 300
to: 0
datasourceUid: __expr__
model:
type: threshold
conditions:
- evaluator:
type: gt
params: [95]
expression: A
for: 2m
annotations:
summary: "Host memory critically high (>95%), OOM kill risk"
labels:
severity: critical
- uid: host-disk-warning
title: "Host Disk > 80%"
condition: C
data:
- refId: A
relativeTimeRange:
from: 300
to: 0
datasourceUid: victoria-metrics
model:
expr: sum by (mountpoint) (system_filesystem_usage_bytes{state="used"}) / sum by (mountpoint) (system_filesystem_usage_bytes{state=~"used|free"}) * 100
instant: true
range: false
- refId: C
datasourceUid: __expr__
model:
type: threshold
conditions:
- evaluator:
type: gt
params: [80]
expression: A
for: 10m
noDataState: OK
dashboardUid: http-middleware
panelId: 108
annotations:
__dashboardUid__: http-middleware
__panelId__: "108"
summary: "Disk {{ $labels.mountpoint }} is {{ printf \"%.0f\" $values.A.Value }}% full"
labels:
severity: warning
- uid: host-disk-critical
title: "Host Disk > 90%"
condition: C
data:
- refId: A
relativeTimeRange:
from: 300
to: 0
datasourceUid: victoria-metrics
model:
expr: sum by (mountpoint) (system_filesystem_usage_bytes{state="used"}) / sum by (mountpoint) (system_filesystem_usage_bytes{state=~"used|free"}) * 100
instant: true
range: false
- refId: C
datasourceUid: __expr__
model:
type: threshold
conditions:
- evaluator:
type: gt
params: [90]
expression: A
for: 5m
noDataState: OK
dashboardUid: http-middleware
panelId: 108
annotations:
__dashboardUid__: http-middleware
__panelId__: "108"
summary: "Disk {{ $labels.mountpoint }} is {{ printf \"%.0f\" $values.A.Value }}% full; a full disk stopped prod before"
labels:
severity: critical
- uid: container-restarted
title: "Container restarted"
condition: C
data:
- refId: A
relativeTimeRange:
from: 600
to: 0
datasourceUid: victoria-metrics
model:
expr: sum by (container_name) (increase(container_restarts_total[10m]))
instant: true
range: false
- refId: C
datasourceUid: __expr__
model:
type: threshold
conditions:
- evaluator:
type: gt
params: [0]
expression: A
for: 0s
noDataState: OK
dashboardUid: errors
panelId: 3
annotations:
__dashboardUid__: errors
__panelId__: "3"
summary: "{{ $labels.container_name }} restarted {{ printf \"%.0f\" $values.A.Value }} times in 10 minutes"
labels:
severity: critical
# Seen in the last 10 minutes but not in the last one; a service a release removes alerts once too
- uid: container-stopped
title: "Container stopped"
condition: C
data:
- refId: A
relativeTimeRange:
from: 600
to: 0
datasourceUid: victoria-metrics
model:
expr: max by (container_name) (max_over_time(container_uptime_seconds[10m])) unless max by (container_name) (last_over_time(container_uptime_seconds[1m]))
instant: true
range: false
- refId: C
datasourceUid: __expr__
model:
type: threshold
conditions:
- evaluator:
type: gt
params: [0]
expression: A
for: 1m
noDataState: OK
dashboardUid: errors
panelId: 4
annotations:
__dashboardUid__: errors
__panelId__: "4"
summary: "{{ $labels.container_name }} stopped reporting; it was up {{ printf \"%.0f\" $values.A.Value }} s"
labels:
severity: critical
- orgId: 1
name: service-alerts
folder: 'Integr8sCode'
interval: 1m
rules:
- uid: error-lines
title: "Error lines"
condition: C
data:
- refId: A
relativeTimeRange:
from: 300
to: 0
datasourceUid: loki
model:
expr: sum by (service_name, log_record_template) (count_over_time({service_namespace="integr8scode"} | detected_level="error" [5m]))
queryType: instant
- refId: B
relativeTimeRange:
from: 300
to: 0
datasourceUid: loki
model:
expr: sum(count_over_time({service_namespace="integr8scode"} | detected_level="error" [5m]))
queryType: instant
- refId: C
datasourceUid: __expr__
model:
type: threshold
conditions:
- evaluator:
type: gt
params: [0]
expression: A
for: 0s
noDataState: OK
dashboardUid: errors
panelId: 5
annotations:
__dashboardUid__: errors
__panelId__: "5"
description: "{{ printf \"%.0f\" $values.B.Value }} {{ if eq $values.B.Value 1.0 }}error{{ else }}errors{{ end }} in the last 5 minutes"
summary: "{{ printf \"%.0f\" $values.A.Value }}x {{ $labels.service_name }}: {{ $labels.log_record_template }}"
labels:
severity: warning
- uid: api-5xx
title: "API 5xx > 1%"
condition: C
data:
- refId: A
relativeTimeRange:
from: 300
to: 0
datasourceUid: victoria-metrics
model:
expr: sum(rate(http_server_request_duration_seconds_count{http_response_status_code=~"5.."}[5m])) / sum(rate(http_server_request_duration_seconds_count[5m])) * 100
instant: true
range: false
- refId: C
datasourceUid: __expr__
model:
type: threshold
conditions:
- evaluator:
type: gt
params: [1]
expression: A
for: 5m
noDataState: OK
dashboardUid: http-middleware
panelId: 1
annotations:
__dashboardUid__: http-middleware
__panelId__: "1"
summary: "{{ printf \"%.1f\" $values.A.Value }}% of API requests answered 5xx over 5 minutes"
labels:
severity: critical
# Only groups with a consumer: an empty group's lag is a stopped container, which alerts on its own
- uid: kafka-lag
title: "Kafka consumer lag > 100"
condition: C
data:
- refId: A
relativeTimeRange:
from: 300
to: 0
datasourceUid: victoria-metrics
model:
expr: max by (group, topic) (kafka_consumer_group_lag_sum_ratio) and on (group) (max by (group) (kafka_consumer_group_members) > 0)
instant: true
range: false
- refId: C
datasourceUid: __expr__
model:
type: threshold
conditions:
- evaluator:
type: gt
params: [100]
expression: A
for: 10m
noDataState: OK
dashboardUid: kafka-events-monitoring
panelId: 21
annotations:
__dashboardUid__: kafka-events-monitoring
__panelId__: "21"
summary: "{{ $labels.group }} is {{ printf \"%.0f\" $values.A.Value }} messages behind on {{ $labels.topic }}"
labels:
severity: warning
# No data means the receiver can't reach mongod, so it alerts too
- uid: mongodb-health
title: "MongoDB down"
condition: C
data:
- refId: A
relativeTimeRange:
from: 300
to: 0
datasourceUid: victoria-metrics
model:
expr: min(mongodb_health_ratio)
instant: true
range: false
- refId: C
datasourceUid: __expr__
model:
type: threshold
conditions:
- evaluator:
type: lt
params: [1]
expression: A
for: 2m
noDataState: Alerting
dashboardUid: mongodb-monitoring
panelId: 1
annotations:
__dashboardUid__: mongodb-monitoring
__panelId__: "1"
summary: "MongoDB reports unhealthy or stopped reporting"
labels:
severity: critical
- uid: dlq-waiting
title: "DLQ has messages for 30 minutes"
condition: C
data:
- refId: A
relativeTimeRange:
from: 300
to: 0
datasourceUid: victoria-metrics
model:
expr: sum(dlq_queue_size)
instant: true
range: false
- refId: C
datasourceUid: __expr__
model:
type: threshold
conditions:
- evaluator:
type: gt
params: [0]
expression: A
for: 30m
noDataState: OK
dashboardUid: dlq-monitoring
panelId: 1
annotations:
__dashboardUid__: dlq-monitoring
__panelId__: "1"
summary: "{{ printf \"%.0f\" $values.A.Value }} messages have waited in the DLQ for 30 minutes"
labels:
severity: warning
- uid: dlq-large
title: "DLQ > 50 messages"
condition: C
data:
- refId: A
relativeTimeRange:
from: 300
to: 0
datasourceUid: victoria-metrics
model:
expr: sum(dlq_queue_size)
instant: true
range: false
- refId: C
datasourceUid: __expr__
model:
type: threshold
conditions:
- evaluator:
type: gt
params: [50]
expression: A
for: 5m
noDataState: OK
dashboardUid: dlq-monitoring
panelId: 5
annotations:
__dashboardUid__: dlq-monitoring
__panelId__: "5"
summary: "{{ printf \"%.0f\" $values.A.Value }} messages are waiting in the DLQ"
labels:
severity: critical
- orgId: 1
name: heartbeat
folder: 'Integr8sCode'
interval: 1m
rules:
# Always firing; the heartbeat route resends it every 5 minutes, and the outside monitor alerts when that stops
- uid: heartbeat
title: "Heartbeat"
condition: C
data:
- refId: A
relativeTimeRange:
from: 60
to: 0
datasourceUid: victoria-metrics
model:
expr: vector(1)
instant: true
range: false
- refId: C
datasourceUid: __expr__
model:
type: threshold
conditions:
- evaluator:
type: gt
params: [0]
expression: A
for: 0s
noDataState: Alerting
execErrState: Alerting
annotations:
summary: "Grafana is up and can send"
labels:
heartbeat: "true"
Every rule is an instant query evaluated every minute, and each summary names the service or mount and the value. Rules that point at a panel carry its dashboardUid and panelId, and the same two as the __dashboardUid__ and __panelId__ annotations, since Grafana puts the panel link into the mail only from those.
| Rule | Fires when | Severity |
|---|---|---|
| Host Memory, Host Disk | memory over 85% for 5 minutes or 95% for 2; disk over 80% for 10 minutes or 90% for 5 | warning, critical |
| Container restarted, Container stopped | Docker restarted a container in the last 10 minutes; a container seen in the last 10 minutes hasn't reported for a minute, for 1 minute | critical |
| Error lines | any error line in 5 minutes, from Loki's detected_level; one alert per service and line template (Loki), so the mail lists each kind of error with its count |
warning |
| API 5xx, Kafka lag, MongoDB down, DLQ | 5xx over 1% for 5 minutes; a group with consumers 100 messages behind for 10 minutes; health under 1 or no data for 2 minutes; any DLQ message for 30 minutes or over 50 for 5 | critical or warning |
The memory and disk rules read the host-metrics collector's scrapers; system.memory.utilization is off by default and otel-host-metrics.yaml turns it on. The container rules read the collector's docker_stats receiver.
The Heartbeat rule always fires. A child policy sends it to the heartbeat contact point, a webhook to ALERT_HEARTBEAT_URL, every 5 minutes; the outside monitor behind that URL alerts when the pings stop, since nothing inside the host can report the host's or Grafana's own death. Locally the URL defaults to the collector's health endpoint, which answers any POST.
All other alerts go to the email-alerts contact point, which sends one email addressed to everyone in GRAFANA_ALERT_RECIPIENTS (singleEmail: true). SMTP is configured in grafana.ini from env vars that compose fills in: GF_SMTP_HOST, MAILJET_API_KEY as the user, MAILJET_SECRET_KEY as the password and MAILJET_FROM_ADDRESS as the sender. Locally GF_SMTP_HOST is empty, so alerts fire but no mail goes out. Production sets it from the MAILJET_HOST secret.
The email template in the same file writes the subject and the body. The first line is the rule's description when it has one, then comes one line per alert from its summary, the resolved ones under "Resolved:", and the panel link, or the rule's page for a rule without a panel. An Error lines mail from a local run, with the backend unreachable and the worker failing to create pods, read:
16 errors in the last 5 minutes:
- 1x frontend: 2026/10/24 21:36:35 [error] 41#41: *6 backend could not be resolved (3: Host not found), client: 172.19.0.1, server: _, request: "GET /api/v1/x1 HTTP/1.1", host: "localhost:15001"
- 10x frontend: 2026/10/24 <*> [error] <*> <*> backend could not be resolved (3: Host not found), client: 172.19.0.1, server: _, request: "GET <*> HTTP/1.1", host: "localhost:15001"
- 1x integr8scode-worker: {"event": "Pod creation failed", "execution_id": "a1b2", "error": "boom", "logger": "integr8scode", "level": "error", "timestamp": "2026-10-24T21:30:00.000Z", <exception>}
- 4x integr8scode-worker: {"event": "Pod creation failed", "execution_id": <*> "error": <*> "logger": "integr8scode", "level": "error", "timestamp": <*> <exception>}
https://grafana.integr8scode.cc/d/errors?orgId=1&viewPanel=5
The 1x lines are the first of each shape: the collector keeps a line whole until a second one like it arrives.
The rule group names its folder by title, Integr8sCode. If the rules load before the dashboard provider creates that folder, Grafana makes a second Integr8sCode folder with a random UID and puts the rules there. The compose entrypoint avoids that: it starts Grafana without the alerting files, waits for /api/folders/integr8scode, then copies them in and calls /api/admin/provisioning/alerting/reload.
A Grafana volume from before that fix may still have the stray folder holding the rules. Delete it once in the UI (it's the Integr8sCode folder whose URL isn't /dashboards/f/integr8scode/), then docker compose restart grafana so the entrypoint loads the rules into the right one.
grafana.ini¶
[users]
allow_sign_up = false
allow_org_create = false
auto_assign_org = true
[smtp]
enabled = true
host = ${GF_SMTP_HOST}
user = ${GF_SMTP_USER}
password = ${GF_SMTP_PASSWORD}
from_address = ${GF_SMTP_FROM_ADDRESS}
from_name = Integr8sCode Alerts
starttls_policy = MandatoryStartTLS
[auth]
disable_login_form = false
[auth.anonymous]
enabled = false
[auth.basic]
enabled = true
[unified_alerting]
enabled = true
[alerting]
enabled = false
; Grafana 13 downloads these on first start; install them before provisioning runs
[plugins]
preinstall_sync = prometheus,jaeger,grafana-opensearch-datasource@2.34.5
[alerting] is the legacy engine and stays off. [unified_alerting] runs the provisioned rules. There's no root_url here; compose sets GF_SERVER_ROOT_URL from GRAFANA_ROOT_URL.