Log search (OpenSearch)¶
OpenSearch holds the same log lines as Loki, with every field indexed, so a search by execution id, user id or error text over 30 days answers in milliseconds instead of reading every chunk. Loki stays for now: the plan in #373 is to run both on real traffic, compare, and then drop one.
flowchart LR
Logs["logs and logs/browser pipelines"] -->|"forward/search"| Search["logs/search: transform/search"]
Search -->|"opensearch exporter"| OS["OpenSearch :9200, logs-YYYY.MM.DD"]
OS --> Grafana
Collector¶
Both log pipelines in otel-collector/otel-collector-config.yaml send their lines to Loki and to the forward/search connector, so OpenSearch gets exactly what Loki gets, after the same filters. The logs/search pipeline runs transform/search, which parses the JSON body of the backend's and worker's records into attributes (execution_id, user_id, event and the rest of structlog's keys), and the opensearch exporter writes one index a day, logs-2026.10.22, in the exporter's default SS4O document shape:
{
"@timestamp": "2026-10-22T19:00:59.756Z",
"body": "{\"execution_id\": \"7f3c2a10-1111-4abc-9def-0123456789ab\", \"event\": \"Pod failed\", \"level\": \"error\", \"error\": \"ImagePullBackOff\"}",
"attributes": { "execution_id": "7f3c2a10-1111-4abc-9def-0123456789ab", "event": "Pod failed", "level": "error", "error": "ImagePullBackOff" },
"resource": { "service.name": "integr8scode-backend", "service.namespace": "integr8scode" },
"severity": { "text": "ERROR", "number": 17 },
"traceId": "7f08c4fb897b5f96bfa811daf69f6821",
"spanId": "a1b2c3d4e5f60718"
}
The exporter treats a refused connection as a permanent error and doesn't retry it (log_bulk_indexer.go#L48-L51), so lines sent while OpenSearch starts or restarts are dropped from OpenSearch. Loki still has them.
OpenSearch¶
Single node, OpenSearch 3.9, security plugin off, listening on 127.0.0.1:9200 and on the compose network, like Loki. node.store.allow_mmap=false means the host's vm.max_map_count stays as it is. The heap is 512m and the container limit 1536m: 30 days at prod's volume sit near 1.1 GiB. Data lives in the opensearch_data volume, about 1.1 GB for 30 days.
The container's entrypoint is opensearch/start.sh. It starts OpenSearch, waits for the cluster, and then applies two things through the API: the index template and the retention policy. The healthcheck passes only once the template exists.
{
"index_patterns": ["logs-*"],
"template": {
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0
},
"mappings": {
"properties": {
"@timestamp": { "type": "date" },
"body": { "type": "text" },
"attributes": { "type": "flat_object" },
"resource": { "type": "flat_object" }
}
}
}
}
attributes and resource are flat_object fields. Every key under them is searchable by its dotted path, but they add no fields to the mapping, so a key that's a number in one line and a string in another (status: 500, status: "running") can't make OpenSearch reject a line.
{
"policy": {
"description": "Delete a daily logs index 30 days after it was created",
"default_state": "hot",
"states": [
{
"name": "hot",
"actions": [],
"transitions": [{ "state_name": "delete", "conditions": { "min_index_age": "30d" } }]
},
{
"name": "delete",
"actions": [{ "delete": {} }],
"transitions": []
}
],
"ism_template": [{ "index_patterns": ["logs-*"], "priority": 100 }]
}
}
The policy deletes each daily index 30 days after it was created, the same retention as Loki. A stored policy is never overwritten, so a changed policy needs a new id.
The collector starts writing as soon as OpenSearch answers, a moment before start.sh has put the template in. So compose starts OpenSearch with action.auto_create_index=-logs-*,+*: until then a write gets index_not_found_exception and is dropped (Loki has it), instead of creating a logs- index with default mappings. Once the template and the policy exist, start.sh allows auto-creation with a persistent cluster setting, so a restart needs no setup before writes go through. The cluster reports yellow even so: the retention plugin's own .opendistro-ism-config index asks for a replica that a single node can't place.
Grafana¶
The datasource is the grafana-opensearch-datasource plugin, installed by preinstall_sync in grafana.ini. Its index is the daily pattern [logs-]YYYY.MM.DD, so a query reads only the days in its time range. The message field is body, the level field severity.text, and a traceId data link opens the trace in Jaeger.
Queries¶
Open Grafana Explore at http://localhost:3000/explore, pick the OpenSearch datasource and the Logs query type, and write Lucene:
# one execution, across every service
attributes.execution_id:"7f3c2a10-1111-4abc-9def-0123456789ab"
# one user's failures
attributes.user_id:"u-42" AND severity.text:ERROR
# error text anywhere in a line
"ImagePullBackOff"
# one service
resource.service.name:"integr8scode-worker"
Measured locally on 1.94 million synthetic app lines spread over 31 daily indices, prod's volume once Mongo's info lines are dropped: a search by attributes.execution_id took 6 to 8 ms in OpenSearch (624 ms for the first query after a restart) and 25 to 75 ms through Grafana. The same search in Loki on prod's 10 days took 3.4 s, and 8.2 s with | json.