Log aggregation (Loki)¶
Loki stores the logs of the backend and the worker, and the output of every container, so you can query them in Grafana across processes and time. Without it, docker compose logs still shows each container's output from Docker's local cache.
flowchart LR
Proc["backend and worker"] -->|"OTLP gRPC"| Collector["OTel Collector :4317"]
Docker["every container's stdout and stderr"] -->|"fluentd log driver"| Collector2["OTel Collector :24224"]
Collector -->|"otlp_http/loki"| Loki["Loki :3100/otlp"]
Collector2 -->|"otlp_http/loki"| Loki
Loki --> Grafana
Export from Python¶
setup_log_exporter in app/core/logging.py builds an OTel LoggerProvider with a BatchLogRecordProcessor and a gRPC OTLPLogExporter, then adds a LoggingHandler to the root logger. The handler carries the same RedactingFilter as the stderr handler, so only redacted text leaves the process. The backend calls it in create_app, and the worker calls it right after setup_logger in its main().
The only switch is OTEL_EXPORTER_OTLP_ENDPOINT: empty means no export. config.toml sets it to http://otel-collector:4317; config.test.toml leaves it unset.
The OTel resource on every record has service.name from SERVICE_NAME (integr8scode-backend for the API, integr8scode-worker for the worker), service.version, service.namespace=integr8scode, deployment.environment and service.instance.id from HOSTNAME.
Collector¶
The logs pipeline receives OTLP and container output (fluent_forward), runs memory_limiter, filter/app_json_stdout, filter/mongo_info, groupbyattrs/container, transform/container, transform/level, drain, transform/template, batch and resource, and exports with otlp_http/loki to http://loki:3100/otlp. It also hands every line to forward/search, the pipeline that indexes the same lines in OpenSearch. The full config is otel-collector/otel-collector-config.yaml.
The browser's records come in on a pipeline of their own, logs/browser. Its receiver, otlp/browser, takes OTLP over HTTP on port 4319 with bodies up to 64 KB, and only nginx's /otlp/v1/logs reaches it (Frontend logging). The browser sends every level, and filter/browser is where lines get dropped: below warning when the collector's ENVIRONMENT is production, none anywhere else. Anyone can post to that path, so transform/browser drops whatever resource a record arrives with and sets service.name to browser: a posted record can't pass as the backend or any other service. The Errors dashboard and the error-lines alert group by service_name, so browser shows up there like any container.
Container output¶
Every compose service logs through Docker's fluentd log driver (the x-logging block in docker-compose.yaml) to the collector's fluent_forward receiver on localhost:24224, tagged with the container name. That covers what never reaches the Python logger: Mongo, Kafka, Redis, nginx, gunicorn's own lines and a traceback from a crash before logging starts. groupbyattrs/container and transform/container give each container its own resource with service.name set to the container name, so its lines carry service_name="mongo", service_name="backend" and so on, next to integr8scode-backend and integr8scode-worker for the structured logs. filter/app_json_stdout drops the backend's and worker's JSON lines from stdout, since the same records arrive over OTLP with their trace ids. filter/mongo_info drops Mongo's info lines ("s":"I") and keeps its warnings and errors: Mongo logs every connection, its metadata and each login, and in prod those were 1.24 million of 1.87 million lines over ten days, 85% of the bytes a search had to read.
Docker's lines arrive without a level, and Loki's own guess marks Mongo's and Redis's info lines as errors, so transform/level gives each one from its format before Loki sees it: a JSON or logfmt level, Mongo's "s", a zap tab-separated level, nginx's [error], then a level word such as log4j's or gunicorn's ERROR, a traceback's first line and Redis's marker after the time, and only then a plain error or warn word. Records that already have a level, like the backend's and worker's over OTLP, keep theirs. The Errors dashboard and the error-lines alert read the result through detected_level.
Then drain, the collector's drain processor, gives each line a template: it groups lines of the same shape and puts <*> where they differ, so five failed requests to a dead backend share one connect() failed (111: Connection refused) while connecting to upstream, client: <*> line. Two masks turn a traceback into one token, a plain one after the first line break and the exception field in structlog's JSON, so a deeper traceback doesn't make a new template. transform/template keeps the template, log.record.template, on error lines only, since Loki and OpenSearch store every attribute of every line. Loki has it as the structured metadata field log_record_template, and the error-lines alert groups by it. A new shape keeps its full text as the template until a second line like it arrives, and the collector learns the shapes again after a restart. The browser's pipeline runs both processors too.
With fluentd-async a container starts and buffers its lines while the collector is down, and docker logs reads Docker's local copy (3 files of 10 MB per container). The collector's own container keeps Docker's local driver, since its debug exporter would otherwise print its own lines back to itself.
Loki 3 accepts OTLP natively. Resource attributes such as service.name and service.namespace become stream labels (service_name, service_namespace). The record's trace_id and span_id become structured metadata. For our own events the log line is the JSON that structlog rendered. Third-party library records arrive as plain text, so | json can't pull fields out of them.
Loki¶
Single instance with filesystem storage, TSDB index and schema v13. Structured metadata is on, and retention is 30 days, the same as Victoria Metrics.
auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9096
log_level: warn
common:
ring:
instance_addr: 127.0.0.1
kvstore:
store: inmemory
replication_factor: 1
path_prefix: /loki
schema_config:
configs:
- from: "2026-01-01"
store: tsdb
object_store: filesystem
schema: v13
index:
prefix: index_
period: 24h
storage_config:
filesystem:
directory: /loki/chunks
limits_config:
retention_period: 720h
allow_structured_metadata: true
ruler:
enable_api: false
storage:
type: local
local:
directory: /loki/rules
compactor:
working_directory: /loki/compactor
compaction_interval: 10m
retention_enabled: true
delete_request_store: filesystem
retention_delete_delay: 2h
retention_delete_worker_count: 150
The container has a 512m memory limit (a 30-day search took it to 207 MiB), keeps data in the loki_data volume and listens on 127.0.0.1:3100. Its image is distroless, so there's no healthcheck; Grafana and the collector start once the container has started.
Queries¶
Open Grafana Explore at http://localhost:3000/explore and pick the Loki datasource.
# everything, from all processes
{service_namespace="integr8scode"}
# the worker only
{service_name="integr8scode-worker"}
# errors
{service_namespace="integr8scode"} | json | level="error"
# one execution
{service_namespace="integr8scode"} | json | execution_id="abc123"
# one trace, through structured metadata
{service_namespace="integr8scode"} | trace_id="0af7651916cd43dd8448eb211c80319c"
Expand a line logged inside a span and its Links section has a TraceID button that opens the trace in Jaeger. The datasource's derived field reads the trace_id structured metadata, not the JSON line, so third-party records logged inside a span get the link too.