systemd-journald writes 55KB+ to disk per log line

systemd-journald writes 55KB+ to disk per log line

A GitHub issue opened against the systemd project on January 3, 2026 by a user named XANi describes systemd-journald, the logging daemon bundled with systemd, writing far more data to disk than the logged volume would suggest. On a Debian 13 virtual machine running systemd 257.9 with journald backed by XFS, a haproxy service was producing only two log lines per second, yet the VM's block I/O sat at about 50 IOPS. XANi's stated expectation was that journal writes should stay within an order of magnitude of what plain syslog would produce; the observed load was far outside that. He added that he has, on separate occasions, seen journal files corrupt after an unclean reboot, arguing the format is not even particularly resilient given how much space it consumes. He linked the report to a previous issue, #15292, which he said covered the same problem and had been closed without good reason, a characterization that is his own rather than something the source explains.

Early replies tried an obvious fix: ValdikSS suggested disabling journald's log compression, but XANi found no meaningful difference, since his log lines were likely short enough that compression never engaged. One commenter, mAd-DaWg, criticized how the earlier issue's author had been treated, calling it 'utterly shocking', and argued that the systemd maintainers, with their deeper understanding of the system, were better placed to diagnose the problem than to dismiss the measurement tools being used. Months later, a different user, joaociocca, reported an unrelated but consistent symptom on a separate system: while chasing intermittent slowdowns, they found via iotop's accumulation mode that journald had written almost 7GB to disk in just 15 minutes, climbing to almost 11GB after 22 minutes. Their journal log for that window was dominated by repeated 'Under memory pressure, flushing caches' entries.

The most detailed numbers arrived in August, when ValdikSS built a dedicated ext4 loop device for /var/log/journal on an otherwise quiet system, with Compress=no, SyncIntervalSec=10s and DefaultIOAccounting=yes set, and wrote a script to sample two counters after each test: the block device's raw write-sector count and the systemd-journald cgroup's io.stat write counter. A single logger -p info test call produces a journal record of about 752 bytes, confirmed by piping journalctl -o json for that exact timestamp. Despite that, ValdikSS found that either one such log message or a burst of ten nearly identical ones both settled at a minimum of 55KB of physical writes. Across the full test run, 14 individual log messages produced 386KB of total block-device writes, of which 319KB was attributed specifically to journald by the cgroup accounting.

For comparison, ValdikSS ran the same instrumentation while appending equivalent lines directly to a plain text file with fdatasync and fsync after each write, the pattern classic syslog daemons use. Three such appends totaled only 21KB of writes altogether, far below what a single journald message alone required. He then repeated the ext4 test on a btrfs volume with copy-on-write disabled, posting comparable raw write-delta samples, though without the same clean single summary total the ext4 run produced.

On the same day, XANi argued that the deeper cause is not the write mechanism but the on-disk format itself, calling it 'extremely wasteful and verbose': every message repeats the boot ID and other duplicated data, and the format is not indexed in a way that makes either writing or reading cheap. ValdikSS separately revised his own assumption about how journald buffers data: he had expected it to hold new entries in volatile memory for the SyncIntervalSec window before writing to disk, but concluded from his tests that journald actually writes to persistent storage the moment any data appears in the log, and only delays the fdatasync and fsync call by that interval. More than a week later, another commenter, amluto, said he once built his own append-only database using mmapped writes and, in hindsight, called that design choice wrong, writing that pwrite would have been far better, and asked whether anyone could explain why journald relies on mmapped writes in the first place. XANi's reply the next day restated that mmap itself is not the core problem: the bloated format is, since it serves goals most users do not need while missing what they actually want, fast lookup and reasonable write performance. Late in the thread, another commenter raised a further, unconfirmed theory from an outside discussion: that a recent kernel change in how memory-mapped files are handled, called large folios, may be making mmap-based writes costly across several filesystems, not only the ones tested here.

The issue remains open. No systemd or journald maintainer has commented, and there is no official explanation or announced fix.

Key facts

  • XANi opened systemd/systemd issue #40262 on January 3, 2026, reporting about 50 IOPS on a Debian 13 VM logging just two lines per second through haproxy on XFS, and linked it to a previous issue, #15292, that he says was closed without good reason.
  • ValdikSS's instrumented ext4 test in August found that a single logger call, which produces a journal record of about 752 bytes, forces at least 55KB of physical block-device writes, and a burst of 10 similar messages cost about the same minimum.
  • Across a full run of 14 test log messages, ValdikSS measured 386KB of total block-device writes, of which 319KB was attributed to journald specifically via cgroup accounting.
  • A baseline test appending equivalent lines to a plain text file with fdatasync and fsync, the syslog-style approach, used only 21KB across three writes, far less than journald's per-message cost.
  • Separately, joaociocca measured journald writing almost 7GB to disk in 15 minutes, almost 11GB after 22 minutes, via iotop's accumulation mode; as of this report, no systemd or journald maintainer has commented and the issue remains open.

Why it matters

systemd-journald is the logging daemon systemd installs and runs by default, so a structural inefficiency in how it writes to disk potentially touches every machine running it, not only the two systems described here. The core finding, that a log record of about 750 bytes can force tens of kilobytes of physical writes, is more than a one-off complaint: two users, on different systems and different filesystems, independently reported the same disproportion between logged data and disk writes. That pattern is what turned a complaint that had previously been closed into a new issue with concrete measurements behind it.

Who it affects

Administrators and users running systemd-based Linux systems with journald writing to persistent storage, at any log volume: the original report came from a haproxy-fronted VM logging only two lines per second, and a second, independent report came from a different user who initially suspected a GPU problem behind some slowdowns, before iotop's accumulation mode pointed to journald instead. That range, from a proxy-fronted VM to an unrelated troubleshooting session, is part of why the thread reads as a systemic problem rather than one tied to a specific workload. The discussion also touches filesystem behaviour more broadly, since the underlying mechanism is debated between ext4 and btrfs specifics and a possibly more general interaction with how the kernel handles memory-mapped files.

How to use it

There is no fix or official guidance from the systemd project. Commenters describe partial workarounds rather than a solution: birdie-github has switched systems that do not need log persistence across reboots to Storage=volatile in journald.conf, which avoids writing to persistent storage at all. Disabling compression, the first suggestion tried, made no real difference in XANi's case, since his log lines are short enough that compression likely never triggers. ValdikSS suggested the ext4 lazytime mount option as a partial mitigation, and another commenter agreed it helps on the kernel side, but XANi pushed back that it fixes basically nothing in this specific bug, since relatime already covers access-time writes on files touched recently, and the extra writes described in the issue are unrelated to atime handling.

How solid is it

This is community testing, not an audit from the systemd project. The original report came from a single VM on XFS; the detailed byte-level numbers come from ValdikSS's own instrumented rig, a purpose-built ext4 loop device with cgroup IO accounting enabled, plus a separate, less fully summarized run on btrfs. The setup is concrete and reproducible: a documented configuration, a script sampling block-stat and cgroup counters, and a baseline comparison against plain-text appends, which is more rigorous than a typical bug report. A second, independent user, joaociocca, separately measured multi-gigabyte journal writes on an unrelated system using iotop's accumulation mode while chasing a different problem; that is consistent with, but not cross-checked against, ValdikSS's block-level figures. No systemd or journald maintainer has commented as of this writing. The issue remains open, and the root cause, whether it is the on-disk format, mmap handling, or a broader kernel-level interaction, is debated among commenters rather than confirmed.

Risks and caveats

The numbers come from individual users' own test rigs, not a systematic benchmark across filesystems and configurations, and the source draws no conclusion about journald's default behaviour generally. The btrfs run produced raw per-sample write deltas but no equivalent clean summary total the way the ext4 run's 55KB and 386KB figures were. XANi's claim that the earlier, related issue, #15292, was 'closed without good reason' is his own characterization; the source never states what reason was actually given. The 'large folios' theory raised late in the thread is a further competing explanation relayed from an outside discussion, not something any systemd, journald or kernel maintainer has confirmed in this issue.

“Either a single log message or 10 of similar log messages result in at least 55 KB of written data.”

— ValdikSS, in a comment on systemd/systemd issue #40262