Glonce

Zsh 5.9.2 fixes a decade-old shell history truncation bug

miscThe Glonce editorsHacker News🔥45Glonce rating45Impact40Novelty50Relevance25Surprise35Players40Credibility80

For years, the author of this post occasionally found that commands typed the day before were missing from their Zsh shell history file, ~/.zsh_history: pressing Ctrl+R for backward search simply turned up nothing for those commands, and the file itself held only old entries, with years of newer history gone. There was no visible corruption, no stray unreadable characters or broken lines, and the file's line count varied from incident to incident, which left it unclear whether the culprit was Zsh itself, another program, or some interaction between multiple concurrent Zsh sessions. The first few times it happened, the author just restored the file from a daily backup and moved on, but the problem kept recurring. Their ~/.zshrc set HISTSIZE=4000 (lines loaded for Ctrl+R search) and SAVEHIST=10000000 (lines saved to disk), enabled HIST_IGNORE_DUPS and INC_APPEND_HISTORY, and explicitly disabled SHARE_HISTORY, so each shell ran as a separate session streaming its own commands into one shared history file rather than sharing history live across sessions.

After asking for help on Mastodon in December 2024, the author was pointed toward filesystem-change monitoring tools and worked through the Linux options one by one. inotify, the oldest such API, dating to 2005, showed that .zsh_history was opened, read and then apparently deleted, until watching the whole home directory revealed the real sequence: Zsh reads the existing history, writes it to a new .zsh_history.new file, then renames that file over the original, an ordinary atomic-replace pattern used to compact the history on exit, but inotify gave no process ID to pin down which Zsh session was responsible. fatrace added the missing PID and process name but still no byte counts, so it could not show whether a given rewrite was reading or writing less data than it should. The author considered strace next but judged it impractical to attach to every interactive Zsh session in advance, and worried it might change Zsh's behavior, returning to it later only once there was a reliable way to reproduce the bug on demand. The breakthrough tool was bpftrace: a program logging every open() call against .zsh_history together with the calling stack trace immediately distinguished the two Zsh code paths involved, readhistfile and savehistfile. The author extended this into a fuller script tracking opens, closes, read and write byte counts, renames, unlinks and symlinks per process, and ran it permanently in the background as a systemd unit, checking its journalctl log whenever a truncation happened. One such log confirmed the mechanism directly: a normal shell exit contains an explicit 'read = 0' line showing Zsh read its history file all the way to EOF before rewriting it, while a truncating exit had no such line, meaning the read had stopped short.

To pin down why the read stopped short, the author patched the Zsh 5.9.1 source, in hist.c's savehistfile function, to deliberately crash if it was about to write fewer than 50,000 lines to the temporary .zsh_history.new file, before that file could be renamed over the real history, and installed systemd-coredump so Linux would capture a debuggable core dump when it happened, with a warning that such a core dump contains the user's full shell history and should stay local rather than being uploaded anywhere. A crash duly turned up a few days later. Debugging the core dump with gdb led back through savehistfile into readhistfile and confirmed the actual root cause: readhistfile can be interrupted mid-read when Zsh receives a signal, since it checks a flag, errflag & ERRFLAG_INT, and breaks out of its read loop early when that flag is set, but savehistfile never checked whether that interruption had happened before writing out and saving the now-incomplete history it had just read, so it saved the short version as if it were complete, permanently truncating the file. In the captured crash, the debugger showed errflag set to 2 and lasthist.interrupted set to 1, confirming a signal had, in fact, fired mid-read. The author also worked out why a stray lseek() call had shown up in the earlier bpftrace traces: POSIX requires fclose() on a seekable stream to reposition the underlying file descriptor to match the stream's logical read position, which glibc's buffered fopen() triggers automatically; because Zsh closes the file immediately afterward anyway, the seek turned out to be a harmless side effect rather than a clue to the actual bug.

The author traced the trigger to their own logout habit: they run a long-lived SSH session multiplexed inside a mosh session, and close everything at the end of the day by repeatedly pressing Ctrl+D and Ctrl+C across the open windows, likely sending SIGINT into a Zsh process while it is mid-way through rewriting its history file on exit. With a standalone reproducer in hand, the author filed a bug report to the zsh-workers mailing list in March 2025; Bart Schaefer investigated and posted a fix the following month, in April 2025. It then took a long time for that fix to actually ship, partly because there was a long gap without any Zsh releases at all, and partly because Bart's fix was accidentally missed by the release engineer once Zsh 5.9.1 finally did come out; after the author flagged the oversight, the fix finally landed in Zsh 5.9.2 on July 12, 2026. The author's own closing assessment calls it remarkable that a data-loss bug could go unfixed for about a decade in a popular shell, noting as context that Apple made Zsh the default macOS login shell in 2019; they add that most users likely do not share their specific logout habit that reliably triggers the bug, but say they 'have to imagine' that some users have lost parts of their history along the way, an explicit guess rather than a measured count. The piece frames the issue throughout as a data-loss and reliability bug, with no CVE or security advisory attached to it.

A separate appendix describes an unrelated footgun with a similar symptom: Emacs's TRAMP mode exports the HISTFILE environment variable when opening a remote shell, and because most shell configs only reassign HISTFILE rather than unexporting it, a later shell can inherit a HISTFILE setting meant for a different shell entirely. On a work computer where bash defaults to HISTSIZE=64000 and HISTFILESIZE=64000, the author once inadvertently truncated their own ~/.zsh_history down to 64,000 lines this way, likely by running Emacs's M-x shell, then zsh, then bash in sequence. The fix adopted afterward was to actively unexport HISTFILE in ~/.zshrc rather than only reassigning it.

A second appendix, written after the main investigation, asks whether today's AI coding agents could have found the same bug given only the evidence the author had: a plain-language description of the symptom, the Zsh 5.9.1 source tree, the exact ~/.zshrc in effect, and both a normal and a truncating bpftrace log, with models explicitly barred from consulting newer Zsh versions, upstream commits, changelogs or mailing-list threads. The author first tried Simon Willison's smevals but found it too bare-bones: unrestricted models would peek at the solution or search online for the newer Zsh git version that already had the bug fixed. Switching to Inspect, an open-source eval framework built by the UK AI Security Institute and Meridian Labs, worked better, though the run was expensive: over $300 in token costs across roughly three attempts, with the published results taken from the last of those. A pass required a model to correctly state that a signal interrupt sets errflag, which aborts readhistfile and produces the truncated file.

Working from just the symptom and the bpftrace logs, with no hint about the author's logout habit, OpenAI's GPT-5.6 Sol and Anthropic's Claude Opus 5 both reached the correct diagnosis in all three attempts; most other frontier models, including GPT-5.5, Claude Sonnet 5 and several Gemini variants, only got there once or twice out of three, and a long tail, including GPT-5, GPT-5.1, GPT-5.2, GPT-5.4, Claude Opus 4.8, Claude Haiku 4.5, Gemini 3.1 Flash Lite and every open-weight model except Kimi K3, failed all three attempts outright. Adding a single hint line about the author's Ctrl+C/Ctrl+D logout habit shifted the picture considerably: GPT-5.6 Sol, GPT-5.5, Claude Opus 5, Claude Opus 4.8, Claude Sonnet 5 and the open-weight Kimi K3 and GLM 5.2 all then reached three-for-three, while Gemini 3.1 Flash Lite was the only model that never reached the correct hypothesis in either version of the eval, which the author attributes to it being a comparatively small model. Token costs varied hugely and did not track success: GPT-5.6 Sol solved both variants for under 500,000 tokens in well under three minutes each, while Claude Sonnet 5's hinted run, despite a perfect score, still used 3.1 million tokens over 14 minutes, and GLM 5.2's hinted run needed nearly 23.7 million tokens over 22 minutes to go from a full fail to three-for-three.

The author's read on where models went wrong was methodological, not factual: the most common failure was fixating on the wrong theory and trying to verify it rather than returning to weigh the alternatives. As one example, they quote GLM 5.2 wrongly concluding from a non-zero lseek() call in the trace that SHAREHISTORY must be enabled, overriding the unsetopt SHARE_HISTORY line plainly present in the supplied ~/.zshrc, in order to keep its incorrect theory alive. The author separately notes that most models, including some that scored zero on the graded rubric such as the Qwen and Minimax entries, did at some point consider the correct hypothesis without ultimately committing to it, so the pass and fail table alone may understate what a model's reasoning actually touched on. Adding more orchestration to the eval, such as having one model propose theories while another checks and falsifies them, raised the success rate, and the author expects that tuning prompts and harnesses further could improve individual models' results beyond what is shown here. The author is explicit that this is their own one-off eval setup built on Inspect, not an official or vendor benchmark, and that the pass and fail results are specific to this one bug rather than a general capability claim.

Key facts

  • The root cause: readhistfile can be interrupted by a signal mid-read, but savehistfile never checked for that interruption before saving the shortened history over the real file on shell exit; Zsh 5.9.2, released July 12, 2026, ships the fix.
  • The author found it by escalating through inotify, fatrace and bpftrace, the last one run permanently via a systemd unit, then deliberately patched Zsh to crash whenever it was about to save fewer than 50,000 history lines, capturing the failure in a core dump via systemd-coredump.
  • The actual fix dates to April 2025, when Bart Schaefer posted it to the zsh-workers mailing list after the author's March 2025 bug report, but it was accidentally left out of Zsh 5.9.1 by the release engineer, delaying it by over a year.
  • In the author's own AI eval, built on the Inspect framework at a cost of over $300 in tokens across roughly three attempts, only GPT-5.6 Sol and Claude Opus 5 reliably (3 of 3) diagnosed the bug from the symptom and bpftrace logs alone; adding one hint about the author's Ctrl+C/Ctrl+D logout habit brought Claude Sonnet 5 and the open-weight GLM 5.2 and Kimi K3 up to 3 of 3, while Gemini 3.1 Flash Lite never reached the correct hypothesis in either version.
  • A separate, unrelated footgun (Appendix A): Emacs's TRAMP mode exports HISTFILE into the shell environment, which once truncated the author's own ~/.zsh_history to 64,000 lines by inheriting bash's default HISTSIZE on a work machine.

Why it matters

This is a real, decade-old data-loss bug in one of the most widely used interactive shells, not a hypothetical: Zsh has been macOS's default login shell since 2019, and the failure mode here, silent truncation with zero visible corruption, is exactly the kind that erodes trust in a history file without ever pointing at a cause. It also doubles as a clean case study in systematic debugging: each tracing tool (inotify, fatrace, strace, bpftrace) added a layer of visibility the last one lacked, and when tracing alone stalled, the author switched strategies entirely, engineering a crash on purpose to get a debuggable artifact rather than continuing to guess from logs. And because the author used this exact bug as a yardstick for whether current AI coding models can perform this kind of root-cause diagnosis from raw evidence, it is also a rare, concretely documented data point, not a vendor benchmark, on how models such as GPT-5.6 Sol, Claude Opus 5, Claude Sonnet 5 and several open-weight models handle a real low-level systems bug.

Who it affects

Anyone who uses Zsh, especially heavy multi-session users on Linux or macOS whose shells get closed by an interrupt rather than a clean exit, since that is specifically what triggers the bug; the author's own case involved a long-running SSH session multiplexed inside mosh and closed with repeated Ctrl+C and Ctrl+D presses. It is also relevant to anyone doing low-level Linux debugging, since the piece works as a tutorial on inotify, fatrace and bpftrace plus deliberate crash-and-coredump debugging; and to anyone evaluating AI coding agents on systems-level root-cause diagnosis, since the author's second appendix is a concrete, if informal, comparison across a wide range of current frontier and open-weight models.

How to use it

Update to Zsh 5.9.2, released July 12, 2026, to get the fix; the author notes that anyone pinning Zsh on Debian should pin both the zsh and zsh-common packages together, to avoid ending up with no zsh package installed at all. The debugging technique itself is directly reusable on any Linux system: inotifywait --monitor on a directory for a first pass, fatrace to add process names and PIDs, a small bpftrace script on the relevant syscalls for byte-level detail, and systemd-coredump plus a deliberate crash if tracing alone is not conclusive, though such a core dump can contain a full shell history and should stay local rather than being uploaded anywhere. Separately, the unrelated exported-HISTFILE footgun from Appendix A is avoided by explicitly unexporting HISTFILE in ~/.zshrc instead of only reassigning it. Anyone evaluating coding agents on this kind of task can reuse the author's eval design too: give the model only the symptom, the relevant source tree, the exact configuration in effect and both a normal and a failing trace, explicitly bar it from searching for the already-fixed upstream version, and grade strictly on whether it names the real mechanism rather than a plausible-sounding guess.

How solid is it

This is first-hand and technically detailed, backed by concrete artifacts: bpftrace logs, a gdb backtrace from an intentionally engineered crash, a diff of the debug patch used to trigger it, and a link to the actual upstream fix commit. It is not a peer-reviewed source, but the root cause was independently confirmed by an actual Zsh contributor, Bart Schaefer, who wrote and posted the real fix to the zsh-workers mailing list, and the fix genuinely shipped in Zsh 5.9.2. On Hacker News, the discussion has drawn 62 points and 21 comments, a modest but genuine reception rather than independent verification of the technical claims. The AI-eval appendix is explicitly the author's own one-off setup built on the Inspect framework, not a vendor or third-party benchmark; the published numbers come from just the latest attempt, and token costs for the same model varied by an order of magnitude between runs, so the model comparison is best read as illustrative on this one specific bug rather than a rigorously powered benchmark.

Risks and caveats

The bug was diagnosed and reproduced on Linux (NixOS), using inotify, fatrace, bpftrace and systemd-coredump; the piece cites Zsh becoming macOS's default shell only as context for how widely it is used, not as evidence the same crash was traced on macOS. No count of affected users exists: the author only speculates that some people likely lost history, while noting that most users probably do not share the specific logout habit, repeatedly sending SIGINT while closing sessions, that made the bug easy to trigger and to notice. Appendix A's exported-HISTFILE footgun produces a similar-looking symptom but is a distinct, unrelated cause and should not be confused with the main bug. In the AI eval, the author separately notes that most models, including some that scored zero on the graded rubric such as the Qwen and Minimax entries, did at some point consider the correct hypothesis without ultimately committing to it, so the pass and fail table alone may understate what a model's reasoning actually touched on.

“savehistfile did not check for interruption when writing the shell history when exiting. Therefore, savehistfile wrote the (incomplete) history, truncating the actual history.”

— Michael Stapelberg