The Log Stack

Metrics tell you something is wrong. Logs tell you why. The first version of this pipeline was Fluent Bit shipping straight to Loki, and it worked fine. The rebuild changed three things: Loki moved to the community Helm chart in HA mode, Fluent Bit and Fluentd now run as a forwarder/aggregator pair managed by fluent-operator, and the whole thing went single-tenant.

None of that was strictly necessary for homelab load. I did it to run the topology you actually see in production, and because the Loki chart forced my hand. The values described here are the production configuration.

What's in the stack

fluent-operator manages the first two through CRDs instead of a static ConfigMap. The pipeline (inputs, filters, outputs) lives as ClusterInput, ClusterFilter, and ClusterOutput objects, versioned in Git like everything else.

The pipeline

flowchart LR
    FB["Fluent Bit (DaemonSet)<br>tail + enrich + filter"] -->|"forward :24224"| FD["Fluentd (StatefulSet)<br>buffer + label"]
    FD -->|"push :3100"| L[Loki]
    L --> M[("MinIO<br>loki-logs")]
    G[Grafana] -->|LogQL| L

Fluent Bit tails /var/log/containers/*.log on every node, enriches each record with pod metadata, drops anything that didn't opt in, and forwards over the forward protocol to Fluentd. Fluentd aggregates the streams from all nodes, buffers to a disk PVC, and pushes to Loki. Loki indexes by labels and writes the log content as compressed chunks to MinIO.

Why add Fluentd

This is the part worth justifying, because for homelab scale Fluent Bit alone is enough. The forwarder/aggregator split is what the Fluent docs recommend for production: a lightweight collector on every node (Fluent Bit, written in C, tiny footprint) forwarding to a heavier central aggregator (Fluentd, with its huge plugin ecosystem) that does the buffering and routing.

The aggregator earns its place through one thing: a persistent buffer. If Loki goes down, Fluentd holds logs on disk and retries instead of every node buffering on its own and dropping data under backpressure. fluent-operator wires this with a PVC mounted at /buffers, and the Loki output uses a file buffer with exponential backoff:

buffer:
  type: file
  path: /buffers/loki
  flushMode: interval
  flushInterval: 5s
  retryType: exponential_backoff
  retryWait: 1s
  chunkLimitSize: 8MB

Without that, Fluentd would just relay in memory on an ephemeral volume, which adds nothing over Fluent Bit talking to Loki directly. The buffer survives both a Loki outage and a Fluentd pod restart.

Loki: HA Monolithic on the community chart

Loki has three deployment modes: monolithic, simple scalable, and microservices. Monolithic (one process runs every role) is the right call for a homelab. The catch is naming: SingleBinary was renamed to Monolithic, and Simple Scalable is now deprecated and slated for removal in Loki 4.0. Grafana's production guidance for anyone outgrowing Simple Scalable is either microservices or HA monolithic, so I went HA monolithic: two replicas gossiping over memberlist, replication_factor: 2, sharing the same MinIO bucket.

The chart itself moved. As of March 2026 the OSS Loki Helm chart forked to grafana-community/helm-charts; the original grafana/loki chart is now Enterprise-oriented. The wrapper pins the community chart:

dependencies:
  - name: loki
    version: 17.4.10
    repository: https://grafana-community.github.io/helm-charts

Config highlights from values.yaml:

• deploymentMode: Monolithic, two replicas in production with soft pod anti-affinity
• auth_enabled: false (single-tenant, more on that below)
• Storage type s3 pointing at the in-cluster MinIO endpoint
• Schema tsdb v13, 24h index period
• Chunks: 4 MB target size, Snappy compression
• WAL enabled, replay on crash recovery
• 5 GB local-path PVC per replica

Retention is 7 days. The compactor runs every two hours and deletes expired data automatically.

The chart default pins a hard requiredDuringScheduling anti-affinity on the single-binary pods. With one node per tier, that leaves the second replica Pending forever, so the production values null it out and use a soft preferred rule instead. Both replicas land on the same medium node today; node-level spread kicks in automatically once a second medium node exists.

Single-tenant

The old setup ran auth_enabled: true with homelab as the active tenant, which meant an X-Scope-OrgID header on every push and every query. Multi-tenancy isolates data and applies per-tenant rate limits. It makes sense when several independent consumers share one Loki.

There's one Loki per cluster here and one consumer: me. Multi-tenancy with a single tenant isn't isolation, it's a header you carry everywhere for nothing. Flipping auth_enabled: false dropped the header from the Fluentd output and the Grafana datasource. One line, reversible, and the pipeline got simpler.

Opt-in with labels

Not every pod needs its logs in Loki. The system is opt-in via a custom label: o11y.ruiz.sh/logs: "true".

Fluent Bit reads everything (it's a DaemonSet, it has to), but a Lua filter checks each record and drops it before it leaves the node if the pod doesn't carry the label. Infrastructure addons all carry it. New apps add it explicitly when they want their logs centralized. In fluent-operator this is a ClusterFilter with inline Lua:

function filter_o11y(tag, timestamp, record)
  local k = record["kubernetes"]
  if k ~= nil and type(k) == "table" then
    local labels = k["labels"]
    if labels ~= nil and type(labels) == "table" then
      if labels["o11y.ruiz.sh/logs"] == "true" then
        return 0, timestamp, record  -- keep
      end
    end
  end
  if record["o11y_ruiz_sh_logs"] == "true" then
    return 0, timestamp, record  -- keep (label already flattened)
  end
  return -1, timestamp, record  -- drop
end

A second Lua step normalizes the mess of level field names across apps. It copies from severity, log_level, logLevel, LOG_LEVEL, severity_text into a single level, then maps values: warning becomes warn, err, fatal and critical become error, anything missing defaults to unknown. The whole filter chain (Kubernetes enrich, opt-in drop, JSON parse, level fold, normalize) is one ordered ClusterFilter, because filter order matters and a single object keeps it explicit.

The three bugs that only showed up at runtime

helm template validates structure. It does not validate CRD required fields, forward-protocol compatibility, or whether a path is writable. All three bit me, in order, during the develop rollout.

Required field, missing. The Fluentd Loki output rejected the sync with spec.outputs[0].buffer.type: Required value. The <buffer> @type is mandatory and the render never caught it. Set it to file.

fluent-bit CrashLoopBackOff on every node. The tail input wrote its position DB to /var/log/fluentbit-tail.db, but /var/log is mounted read-only (it's there to read container logs). Fluent Bit could not open the SQLite file and crashed the input. Moved the DB to the operator's writable /fluent-bit/tail volume.

Fluentd silently dropped every event. Logs reached the aggregator, full records with all the metadata, and Fluentd logged skip invalid event for each one. Fluent Bit v2.1+ forwards events as [[time, metadata], record], and Fluentd v1.19+ (where skip_invalid_event defaults to true) rejects that metadata wrapper. The chart documents the exact flag for this combination:

forward:
  retainMetadataInForwardMode: false

Then a quieter one. Logs landed in Loki, but the labels came through as literal strings: namespace="$.kubernetes.namespace_name", level="$.level". The Fluentd output maps labels to extra_labels, which are static, and its labelKeys only reads top-level record keys, not nested $.kubernetes.*. The fix was to flatten in Fluent Bit: a Lua step promotes kubernetes.namespace_name, pod_name, and container_name up to the top level, then labelKeys turns them into real stream labels.

function promote_k8s(tag, timestamp, record)
  local k = record["kubernetes"]
  if k ~= nil and type(k) == "table" then
    record["namespace"] = k["namespace_name"]
    record["pod"]       = k["pod_name"]
    record["container"] = k["container_name"]
  end
  return 2, timestamp, record
end

After that, a query for {namespace="logging"} came back with container, pod, level, and job populated for real. None of these are exotic. They're the ordinary friction of wiring two tools that each evolved their own assumptions about the forward protocol.

From logs to metrics

The Loki Ruler used to fire log-based alerts at Alertmanager. That path left the cluster when alerting consolidated on Grafana and Alertmanager was removed; a log-based alert is now a Grafana-managed rule querying the Loki datasource directly.

What the Ruler still earns its keep with is recording rules: per-service log volume and error/warning rates are precomputed as loki:* series and written to Prometheus via remote_write. They show up in PromQL and Grafana like any other metric, so volume dashboards don't rerun heavy LogQL scans on every refresh.

flowchart LR
    CM["ConfigMaps (loki_rule: true)"] --> SC[sidecar] --> RU[Loki Ruler]
    RU -->|"remote_write (loki:* series)"| PR[Prometheus]
    PR --> GF[Grafana]

Rules ship as ConfigMaps labeled loki_rule: "true". The chart's sidecar watches every namespace for that label and drops the files into /var/loki/rules, where the Ruler loads them and evaluates every 15 minutes. Adding a rule is a Git commit, same as everything else in the stack.

Loki is not Elasticsearch, and that's the point. Simpler, cheaper, good enough for most homelab cases, with full-text search as the tradeoff. The forwarder/aggregator rebuild didn't make it faster, it made it match the shape of a real production logging tier, which was the whole reason to do it. The opt-in label is still the best decision in the stack: without it, every pod in the cluster would dump noise into Loki.

This is the logs half of the observability stack. The other half is metrics.