Loki vs VictoriaLogs on a four-node homelab

Disclaimer. This is an experiment on my homelab. The point is to try both and see what happens, and I'm not out to trash Loki or VictoriaLogs. My setup may have mistakes or misconfigured settings that skew the results, so read the numbers as what happened on my four nodes with my config.

Update (October 13, 2026). The first load test gave Loki a 150m CPU limit against 500m for VictoriaLogs, and Loki lost on CPU. I reran it the next day with both on the same pod size, a higher ingestion limit for Loki and a ladder that goes to 100k lines per second. This post now covers both rounds, the full test setup, and what each one actually costs me.

Searching a day of logs in Loki took long enough that I would switch tabs while it ran. Counting every line of the last 24 hours took 13 seconds. On a cluster that writes about 15 lines per second, that felt wrong, so I set up VictoriaLogs next to Loki and fed both the same logs for a while.

I wanted numbers before deciding which one stays. So I measured three things on the same data: what a day of logs costs on disk, how long the same searches take, and how much each one can take in before it falls over.

Two backends, one pipeline

Fluent Bit already ships every container log to Loki, as described in The Log Stack. For the trial I added a second output to the same Fluent Bit, so both backends receive every record and neither knows about the other.

                 Fluent Bit (DaemonSet, every node)
              ┌───────────────────┴────────────────────┐
         loki output                              http output
              │                                /insert/jsonline
              ▼                                        ▼
 ┌────────────────────────┐        ┌───────────────────────────────────────┐
 │ Loki, 2 replicas       │        │ vmauth-logs (2 replicas)              │
 │ both on "medium"       │        │      ┌────────────┴────────────┐      │
 │ chunks → SeaweedFS     │        │  vlagent a                 vlagent b  │
 │ + memcached caches     │        │      └────────────┬────────────┘      │
 └────────────────────────┘        │                   │                   │
                                   │      ┌────────────┴────────────┐      │
                                   │      ▼                         ▼      │
                                   │  vlsingle a               vlsingle b  │
                                   │  ("medium")                ("large")  │
                                   └───────────────────────────────────────┘

The two layouts differ in a way that matters later. Loki runs in monolithic mode with two replicas, keeps its chunks in SeaweedFS and has memcached in front for chunks and results. VictoriaLogs runs as a VLDistributed from the VictoriaMetrics operator: two single-node instances, one per worker, each holding a full copy. A vmauth in front sends writes to the vlagents and reads to whichever zone answers.

The workers are VMs on a single Proxmox host, a mini PC. "medium" has 4 vCPUs and 14 GB, "large" has 6 vCPUs and 16 GB. Nothing here is big, which is part of the point.

How each one is configured

Both are wrapper Helm charts in my homelab repo, synced by ArgoCD. These are the parts that matter for a write test, as they were during the second round.

Loki is the grafana-community/loki chart in Monolithic mode, so each pod is a single binary running the distributor, ingester, querier, compactor and ruler together:

loki:
  deploymentMode: Monolithic
  singleBinary:
    replicas: 2
    extraEnv:
    - name: GOMAXPROCS
      value: "2"
    resources:
      requests:
        cpu: 50m
        memory: 128Mi
      limits:
        cpu: 500m      # 150m in the first round
        memory: 512Mi  # 384Mi in the first round
    nodeSelector:
      tier: medium

And the Loki config itself, the part that decides how a line travels from the push API to a chunk:

  loki:
    ingester:
      max_chunk_age: 4h
      wal:
        dir: /var/loki/wal
    limits_config:
      retention_period: 180d
      ingestion_rate_mb: 64        # default 4
      ingestion_burst_size_mb: 96  # default 6
    pattern_ingester:
      enabled: true
    commonConfig:
      replication_factor: 1

replication_factor: 1 means each line is written to one ingester only. Both replicas sit on the same node, so a second copy would double the bucket without surviving a node loss. Chunks are gzip, the default, and they leave the ingester for SeaweedFS once they are full, idle or four hours old. The WAL sits on a local-path volume so a restart replays what was not flushed yet. The pattern ingester is on because Grafana's Logs Drilldown needs it for its Patterns tab, and it costs something on every push, as the logs showed later.

VictoriaLogs has three kinds of pods on the write path. These are the values from its chart:

vmauth:
  replicaCount: 2
  resources:
    requests: { cpu: 50m, memory: 64Mi }
    limits:   { cpu: 500m, memory: 256Mi }
vlagent:
  queueStorageSize: 2Gi
  maxDiskUsagePerURL: 900MiB
  resources:
    requests: { cpu: 50m, memory: 64Mi }
    limits:   { cpu: 500m, memory: 256Mi }
vlsingle:
  retentionPeriod: 180d
  storageSize: 20Gi
  retentionMaxDiskSpaceUsageBytes: 15GiB
  resources:
    requests: { cpu: 50m, memory: 128Mi }
    limits:   { cpu: 500m, memory: 512Mi }

The vlagent keeps an on-disk queue per destination zone. If zone b goes down, its share waits on the vlagent's volume and replays when it comes back, which is how a zone catches up. Each vlsingle drops its oldest days once it passes 15 GiB, even before the 180-day retention.

What a day of logs costs

The first measurement is the boring one: let both run for 24 hours on real traffic and compare. The window closed at 18:45 UTC on 2026-10-12, the first full day after the VictoriaLogs zones went up.

The line counts agree, which told me the two outputs were really sending the same thing. The byte counts don't, and that is expected. Loki receives the log line and three labels. VictoriaLogs receives the whole Fluent Bit record, every Kubernetes field included, and indexes all of it.

Even with five times the input, VictoriaLogs stores less per copy. It keeps two copies, so the total is 63.5 MB a day against Loki's 37.1 MB. At this rate 180 days of retention is about 5.8 GB per zone and about 6.7 GB of Loki chunks.

These are the queries behind the table. Loki's metrics live in Prometheus. VictoriaLogs' are scraped by vmagent, so they live in VictoriaMetrics.

# Loki
sum(increase(loki_distributor_lines_received_total[24h]))
sum(increase(loki_distributor_bytes_received_total[24h]))   # raw, before compression
sum(increase(loki_ingester_chunk_stored_bytes_total[24h]))  # chunks flushed to the bucket

# VictoriaLogs
sum by (job) (increase(vl_rows_ingested_total{job=~"vlsingle-lab-.*"}[24h]))
sum by (job) (increase(vl_bytes_ingested_total{job=~"vlsingle-lab-.*"}[24h]))
# stored: size now minus size 24h ago
sum by (job) (vl_compressed_data_size_bytes{job=~"vlsingle-lab-.*"})
sum by (job) (vl_compressed_data_size_bytes{job=~"vlsingle-lab-.*"} offset 24h)

The same search on both

Next I wrote five searches in both query languages, each pair returning the same rows. I ran every one five times in a row through a port-forward and kept the median and the worst run.

Loki got a Cache-Control: no-cache header, which skips its results cache. Its chunk cache stayed on, and so did VictoriaLogs' own caches. The rare phrase is the one row where the counts really differ: LogQL's |= matches a substring, while LogsQL's phrase filter matches whole words. The other gaps are the few seconds between the two runs.

Twenty-five milliseconds for a full day looked like a cache to me, so I ran the last search five more times with the window shifted by a random 60 to 660 seconds. The count changed every time and the latency stayed between 23 and 31 ms.

The difference comes from what each one indexes. Loki indexes only the stream labels and scans the chunks for everything else, so a search over a day reads a day of chunks. VictoriaLogs indexes every field and uses bloom filters to skip the blocks that cannot match.

The load test

Searches tell you how it feels to use. I also wanted to know where each one stops keeping up, so I installed the k6 operator and wrote one load test that runs against either backend.

The operator turns a TestRun resource into Kubernetes Jobs. An initializer checks the script and splits the load into N parts, N runner pods start paused with one part each, and a starter releases them all at once. Each runner remote-writes its own metrics to Prometheus, tagged with a testid.

  kubectl apply  ──►  TestRun (parallelism: 2)  ──►  k6-operator
                                                         │
                       initializer: checks the script, splits the rate
                                                         │
                   ┌─────────────────────────────────────┴──┐
                   ▼                                        ▼
            runner 1 (large node)                    runner 2 (large node)
            half the requests                        half the requests
                   │  100 JSON lines per request            │
                   └──────────────┬─────────────────────────┘
                                  ├──► loki:3100/loki/api/v1/push
                                  └──► vmauth-logs:8427/insert/jsonline
                                                 (both zones)
  runners ── remote write ──► Prometheus   (testid=logs-loki | logs-vl)

This is the whole script. It lives in a ConfigMap that both TestRuns mount:

import http from 'k6/http';
import { check } from 'k6';
import { SharedArray } from 'k6/data';

const TARGET = __ENV.TARGET; // loki | vl
const BATCH = 100; // lines per request
// Real logs come from many streams; one stream would hit Loki's per-stream limit
// (3MB/s) long before the tenant one and measure the wrong thing.
const STREAMS = 20;
// Requests per second at each step; x BATCH = lines per second.
const STEPS = (__ENV.STEPS || '50,100,200,400,500,700,1000').split(',').map(Number);

const URLS = {
  loki: 'http://loki.logging.svc.cluster.local:3100/loki/api/v1/push',
  vl: 'http://vmauth-logs.victoria-logs.svc.cluster.local:8427/insert/jsonline' +
    '?_stream_fields=namespace,container&_msg_field=msg',
};

export const options = {
  scenarios: {
    ingest: {
      executor: 'ramping-arrival-rate',
      startRate: 0,
      timeUnit: '1s',
      preAllocatedVUs: 50,
      maxVUs: 400,
      // 30s to climb to each step, 2m holding it.
      stages: STEPS.flatMap((r) => [
        { target: r, duration: '30s' },
        { target: r, duration: '2m' },
      ]),
    },
  },
  thresholds: {
    http_req_failed: [{ threshold: 'rate<0.01', abortOnFail: true, delayAbortEval: '30s' }],
    http_req_duration: [{ threshold: 'p(95)<2000', abortOnFail: true, delayAbortEval: '30s' }],
  },
};

// Built once and shared by every VU, so a request costs picking lines, not
// building them.
const LEVELS = ['info', 'info', 'info', 'debug', 'warn', 'error'];
const PATHS = ['/api/orders', '/api/users', '/healthz', '/api/checkout', '/metrics'];
const POOL = new SharedArray('lines', () =>
  Array.from({ length: 5000 }, (_, i) => ({
    level: LEVELS[i % LEVELS.length],
    msg: `request handled path=${PATHS[i % PATHS.length]} status=${i % 17 === 0 ? 500 : 200}`,
    trace_id: (Math.random().toString(16).slice(2) + Math.random().toString(16).slice(2)).slice(0, 32),
    user_id: `u${Math.floor(Math.random() * 5000)}`,
    latency_ms: Math.floor(Math.random() * 900),
  })),
);

const pick = () => POOL[Math.floor(Math.random() * POOL.length)];

function lokiBody(container) {
  const ms = Date.now();
  // A distinct nanosecond per line: Loki drops an identical (timestamp, line) pair.
  const values = Array.from({ length: BATCH }, (_, k) => [
    `${ms}${String(k).padStart(6, '0')}`,
    JSON.stringify(pick()),
  ]);
  return JSON.stringify({ streams: [{ stream: { namespace: 'loadtest', container }, values }] });
}

function vlBody(container) {
  return Array.from({ length: BATCH }, () =>
    JSON.stringify({ namespace: 'loadtest', container, ...pick() }),
  ).join('\n');
}

export default function () {
  const container = `k6-${__VU % STREAMS}`;
  const r = TARGET === 'loki'
    ? http.post(URLS.loki, lokiBody(container), { headers: { 'Content-Type': 'application/json' } })
    : http.post(URLS.vl, vlBody(container), { headers: { 'Content-Type': 'application/stream+json' } });
  check(r, { '2xx': (res) => res.status >= 200 && res.status < 300 });
}

A few choices in there are deliberate. Every request carries 100 synthetic JSON lines of about 220 bytes, picked from a pool built once, so the runner spends its CPU on sending and not on building strings. The lines go to 20 streams in a loadtest namespace, because a single stream would hit Loki's per-stream limit first. Each backend gets the lines in its native format: Loki's push API with a nanosecond timestamp per line, and VictoriaLogs' JSON lines with namespace and container as stream fields.

The ramping-arrival-rate executor is what makes this a capacity test. It starts requests at a fixed rate no matter how long the previous ones take, so a slow backend shows up as rising latency and errors instead of quietly lowering the load. The two thresholds end the run on their own once more than 1% of requests fail or p95 passes two seconds.

The Loki TestRun, as it ran in the second round. The VictoriaLogs one only differs in its name, testid and TARGET:

apiVersion: k6.io/v1alpha1
kind: TestRun
metadata:
  name: logs-loki
  namespace: k6-operator
spec:
  # Two runners split the arrival rate: one used 0.46 cores at 20k lines/s, so alone it
  # would hit its own 2-core limit before 100k and the generator would be the result.
  parallelism: 2
  cleanup: post
  script:
    configMap:
      name: logs-ingest
      file: script.js
  arguments: --out experimental-prometheus-rw --tag testid=logs-loki
  runner:
    nodeSelector:
      tier: large
    resources:
      requests:
        cpu: 500m
        memory: 256Mi
      limits:
        cpu: "2"
        memory: 1Gi
    env:
      - name: TARGET
        value: loki
      - name: K6_PROMETHEUS_RW_SERVER_URL
        value: http://kube-prometheus-stack-prometheus.metrics.svc.cluster.local:9090/api/v1/write
      - name: K6_PROMETHEUS_RW_TREND_STATS
        value: p(95),p(99),min,max

The test lives in Git, in a load-tests/ directory that no ArgoCD Application reads. That was deliberate. With cleanup: post the TestRun deletes itself when it finishes, and if ArgoCD owned it, it would see the object missing, recreate it, and run the test again. Here a run starts only when I apply it:

kubectl --context lab apply -f load-tests/logs/script.yaml -f load-tests/logs/vl.yaml
# wait for the TestRun to disappear, then
kubectl --context lab apply -f load-tests/logs/script.yaml -f load-tests/logs/loki.yaml

Round one: unequal pods

The first round ran on 2026-10-12 with one runner and a ladder of 1k, 2k, 5k, 10k and 20k lines per second. VictoriaLogs went first, from 19:07 to 19:20 UTC. Loki started at 19:20 and aborted at about 19:28.

Loki stopped on CPU. Each replica had a 150m limit, it reached 0.14 cores, and the container was throttled in 20 to 25% of its CFS periods. Once that happened, the distributor's calls to the ingesters started timing out:

level=error caller=ratestore.go:301 msg="unable to get stream rates from ingester"
  err="rpc error: code = DeadlineExceeded desc = context deadline exceeded"

That result said more about my limits than about Loki. Each vlsingle had 500m and each Loki replica had 150m. With more CPU, Loki would have run into its own default ingestion_rate_mb: 4, about 18k of these lines per second, and started answering 429. So I changed three things and ran it again.

Round two: what changed

Loki's single binary got exactly what a vlsingle has: 50m/500m CPU and 128Mi/512Mi memory, two replicas as before. The per-tenant ingestion limit went from 4 MB/s to 64 MB/s, with the burst from 6 to 96. The last step is 100k lines per second of about 220 bytes, roughly 22 MB/s, and VictoriaLogs has no such cap, so leaving the default would have ended Loki's run on a config value. The per-stream limit stayed at 3 MB/s; at 100k lines per second the test sends about 1.1 MB/s per stream.

The ladder now starts where the first one got interesting and goes five times further: 5k, 10k, 20k, 40k, 50k, 70k and 100k lines per second, 2.5 minutes per step. And the TestRun uses two runners. One runner had used 0.46 cores at 20k lines per second against a 2-core limit, so at 100k it would have been the bottleneck.

Equal pods still don't make equal setups, and I wanted to see the remaining differences before reading any result:

The differences cut both ways. VictoriaLogs has three times the CPU limit along its path, and it also writes twice as much and shares its large node with the load generator. Loki's two replicas are packed on one 4-vCPU node and pay for the pattern ingester on every push.

Making them identical would mean changing the architecture of one of them, and then the test would measure a setup I don't run. So I tested them as they are and added a second number next to "where did it stop": how many cores the whole write path burns per 10k lines per second. That normalizes the 1-core vs 3-core difference.

Round two: what happened

VictoriaLogs went first, from 08:12:46 to 08:30:24 UTC on 2026-10-13. Loki ran from 08:46:11 and was gone at 08:58:24. Every number below is taken at the end of each step's two-minute hold, over the last 90 seconds.

VictoriaLogs finished the whole ladder with zero failed requests. Its p95 never went above 2.5 ms in the whole run, and it went down a little as the rate went up. At 100k lines per second the whole path, two vmauth, two vlagent and both zones, used 1.2 cores and 876 MiB. The vlagent queues held zero bytes the whole time, so neither zone ever fell behind.

The one sign of strain was in the vlsingle pods. At 100k each used 0.29 cores on average, but 53% of their CFS periods were throttled. They work in bursts, merging and flushing parts, and those bursts hit the 500m ceiling. Latency didn't notice. I never found VictoriaLogs' limit; it is somewhere above 100k lines per second with these pods.

To check that nothing went missing, I counted the test lines in each zone directly:

_time:[2026-10-13T08:12:00Z, 2026-10-13T08:31:00Z] namespace:=loadtest | stats count() as n

zone a: 42749900
zone b: 42749900

The same number in both zones, and a multiple of 100, which is one full request.

Loki is the more interesting story this time. With 500m it was a different system at the low end: p95 of 4.6 to 4.8 ms at 5k and 10k, where in round one it had 26 ms at 2k. At 20k its latency jumped to 66 ms and stayed there through 40k, without errors. Then on the climb to 50k, around 08:57, it fell over:

Memory had been flat at about 260 MiB per replica through every step and only moved once requests started piling up. The run ended on errors: 1,483 HTTP 500s out of 146,939 requests, exactly the 1% threshold. Loki's own logs, which I read from VictoriaLogs since Loki was the one in trouble, show what the 500s were:

level=warn orgID=fake msg="POST /loki/api/v1/push (500) 971.659251ms Response:
  \"rpc error: code = Canceled desc = grpc: the client connection is closing\n\""

level=error msg="unable to get stream rates from ingester"
  err="rpc error: code = DeadlineExceeded desc = context deadline exceeded"

level=error component=pattern-tee msg="failed to send patterns to pattern ingester"
  err="rpc error: code = DeadlineExceeded desc = context deadline exceeded"

Same failure as round one, at eight times the rate. Each push goes from the distributor to the ingester that owns the stream over gRPC, often on the other replica, and the pattern ingester gets a copy too. When both replicas are throttled half the time, those internal calls miss their deadlines, and the push fails. Loki never rejected anything for rate: loki_discarded_samples_total stayed at zero, so the 64 MB/s limit did its job of staying out of the way.

The lines from the 1,483 failed requests were never written, because k6 doesn't retry. Fluent Bit would have retried them in real life, with the loki output set to retry_limit: no_limits.

Cost per line

Normalizing by throughput puts both on the same scale:

At 20k, Loki spends about 1.8 times the CPU to store half as many copies. Per stored copy it is closer to 3.6 times. That's the number I care about on a small host: it says how much headroom each one leaves for everything else.

These are the queries I used to watch the runs and build the tables:

# k6 side, testid = logs-vl or logs-loki
sum(rate(k6_http_reqs_total{testid="logs-vl"}[1m])) * 100   # lines/s
1000 * max(k6_http_req_duration_p95{testid="logs-vl"})      # p95 in ms
max_over_time(k6_http_req_failed_rate{testid="logs-loki"}[15m])  # per status code

# VictoriaLogs (VictoriaMetrics datasource)
sum by (job) (rate(vl_rows_ingested_total{job=~"vlsingle-lab-.*"}[1m]))
sum by (job) (vlagent_remotewrite_pending_data_bytes{namespace="victoria-logs"})

# Loki
sum(rate(loki_distributor_lines_received_total[2m]))
sum by (reason) (rate(loki_discarded_samples_total[2m]))

# CPU, throttling and memory of the pods doing the work
sum by (pod) (rate(container_cpu_usage_seconds_total{namespace="logging",pod=~"loki-[0-9]+",container!=""}[90s]))
sum by (pod) (rate(container_cpu_cfs_throttled_periods_total{namespace="logging",pod=~"loki-[0-9]+"}[90s]))
  / sum by (pod) (rate(container_cpu_cfs_periods_total{namespace="logging",pod=~"loki-[0-9]+"}[90s]))
max by (pod) (container_memory_working_set_bytes{namespace="logging",pod=~"loki-[0-9]+",container!=""})

One side effect I didn't plan for: the mini PC under the desk sounded like a jet engine during both runs. The Proxmox host was at a load of 3.8 with every core boosting to about 5.1 GHz, even with the powersave governor. I left it alone, because turning boost off for one run and not the other would have broken the comparison.

What each one actually costs me

Throughput is nice to know, but my cluster writes about 15 lines per second. At that rate the question is what each one costs while doing almost nothing, and there VictoriaLogs is cheaper on CPU and memory too. Averaged over the 24 hours before the tests:

Most of Loki's memory is the two binaries at about 225 MiB each and the chunks cache at 139 MiB. VictoriaLogs' biggest pods are the vlsingles at about 75 MiB each.

Where VictoriaLogs costs more is disk. On real traffic it stores 63.5 MB a day across its two copies against Loki's 37.1 MB, about 1.7 times as much. Per copy it is actually smaller, so the extra disk is entirely the second zone. And where the bytes live matters as much as how many there are. Loki's chunks go to SeaweedFS, object storage I already run and can grow. VictoriaLogs writes to node-bound local-path volumes, 20Gi requested per zone and capped at 15 GiB by retentionMaxDiskSpaceUsageBytes, on the same node disks everything else uses. If a node's disk fills, that zone is in trouble no matter how cheap its CPU is.

The load test made this concrete. Its 42.7 million lines added about 770 MiB to each zone, 1.5 GiB in total, and they stay for the 180-day retention. On the synthetic lines VictoriaLogs compressed to about 19 bytes per line per copy, which is good compression. It is still two copies on local disk.

So on this cluster the price of VictoriaLogs is disk, and specifically the second copy I chose to keep for HA. CPU and memory are not where it costs me; it wins on both, idle and under load.

Which one stays

VictoriaLogs did better on everything I measured, and the second round removed the excuse I had for Loki. With the same pod size it took 100k lines per second without an error, while Loki broke between 40k and 50k. Searches were 70 to 500 times faster. It uses less CPU and memory idle and less CPU per line under load. The cost is disk: about 1.7 times Loki's footprint, on node-local volumes instead of object storage.

Loki still has things I didn't measure and that I'd miss. Retention in object storage is the big one, and VictoriaLogs only has it on its roadmap. Loki's HA is mature, while the VLDistributed resource I'm using is still marked experimental. And Loki is a built-in Grafana datasource with Logs Drilldown, while VictoriaLogs needs a plugin.

None of that changes the verdict for my homelab. Both have thousands of times today's load in hand, so capacity was never going to decide it. What decides it is that I search logs every day and wait seconds in Loki for what VictoriaLogs answers in 25 ms, on fewer cores and less memory. A few extra gigabytes on node disks is a price I can see and cap, and I already cap it.

I'm keeping VictoriaLogs. Before Loki can go, a few things that read from it have to move: the log-canary alert rules written in LogQL, the trace link on the Tempo datasource, a Tempo dashboard, Fluent Bit's Loki output, and the kagent prompts that name Loki. That migration is its own piece of work.