Glonce

SELF replaces ELF with a SQLite-based executable format

infrastructureThe Glonce editorsHacker News🔥78Glonce rating78Impact58Novelty75Relevance85Surprise70Players35Credibility85

The author of the blog post has spent the last few years on two intertwined obsessions: Nix, used as a way to rebuild whole software environments from scratch, and the idea of replacing ELF, the standard Linux executable format, with SQLite. The idea started during a PhD thesis, where early feedback was unmotivating; radical proposals struggle against the inertia of an established standard. That work produced sqlelf, a tool for exploring an ELF file declaratively with SQL, for example running a query for symbol names instead of piping readelf through grep. It also produced an arXiv paper on sqlelf that was not accepted for publication, plus a follow-up post on querying with the tool. The author never let the idea go, and recent progress in large language models made it worth revisiting: can ELF itself, not just a description of it, be replaced by SQLite?

The answer is a fleshed-out working prototype called SELF, the Structured Executable and Linkable Format, posted on GitHub. A SELF file is not a database that describes a program; it is the literal file a user marks executable and runs. Inspecting a compiled SELF binary with a standard file-type check reports it as an ordinary SQLite database carrying a distinct application ID, and the file still runs directly: it prints its normal output, and an ordinary SQLite query against that same file can pull out the program's shared library dependencies.

The author's central claim is that ELF already functions as a database, it just will not admit it: the format hand-implements database primitives, including a bloom filter for symbol lookup, instead of using a real database engine. ELF's terseness, a legacy of an era when disk space and network bandwidth were scarce, makes it hard to modify, since sections often have to be zeroed out and new ones appended, and leaves it with no self-describing schema; its sections are only interpreted a particular way by convention, not enforced by the format. SQLite is the opposite: self-describing, extremely stable, and built to be extended with new features without breaking older consumers, while still supporting a wide range of queries efficiently.

Structurally, a SELF file needs only two tables to run: one storing the ELF header as key and value pairs, and one holding the load image, with a row per program header and the segment's raw bytes stored as a binary blob, alongside fields for its type, offset, virtual address, size and permission bits. A single symbols table, indexed on name and version, replaces several separate ELF sections along with the .gnu.hash lookup structure; SQLite maintains a proper b-tree index in its place, rather than the hand-rolled bloom filter and bucket chains that .gnu.hash uses so the loader can reject a lookup miss without walking a chain. Folding everything into SQLite tables removes other ELF machinery outright: the .dynstr string table disappears because SQLite already interns text, and symbol versioning becomes a plain column instead of the separate .gnu.version_r and .gnu.version_d sections. Further tables exist only to support tooling and metadata, such as sections, notes and dynamic entries; deleting them still leaves a runnable program, so stripping a SELF binary is literally a database transaction, a delete of those tables followed by a vacuum. In a demonstration, that shrinks a sample binary called hello from 57,344 to 49,152 bytes while it keeps running.

Every tool that only reads an ELF file, such as readelf, nm or ldd, reduces to a query over the tables in a SELF file. Tools that modify a binary, such as strip or patchelf, can operate inside a transaction rather than performing fragile offset surgery on a tightly packed format: stripping becomes a delete plus a vacuum, and patching becomes an update. Information not directly represented in the schema can be exposed through an ordinary SQL view; the post gives ldd itself as an example, defined as a view over the table of needed libraries, ordered by load position.

SQLite reserves a 4-byte application_id field at byte offset 68 of its file header for exactly this kind of use, and SELF stamps that field with the value 0x53454c46, the ASCII bytes for SELF, so an ordinary SQLite database never accidentally matches. Linux's binfmt_misc subsystem, which lets the kernel invoke arbitrary binaries as if they were native, is registered to recognize both the SQLite file signature and the SELF application ID and hand matching files to an interpreter; the post gives the exact NixOS configuration for that registration. Two supporting programs make the demonstration work. elf2self is a small converter, wired up as an optional NixOS build hook, that reads an already-built ELF file's program headers and symbol table and writes them into a new SQLite database; the author notes that extending gcc or ld to emit SELF binaries directly is only a possible future direction, not something built yet, so every conversion today starts from an existing ELF file. self-exec is the interpreter itself, a small C program linked against the SQLite library whose logic mirrors ld.so, except that it reads program headers and symbols from the database rather than from an ELF file; it maps the loadable segments into memory, relocates them and jumps to the entry point. self-exec has to remain an ordinary ELF binary itself: if it also matched the SELF binfmt_misc registration, invoking it would recurse straight into an ELOOP error.

Dynamic linking is where the database approach shows its advantage most clearly, and the post explores it two ways. The first keeps glibc's stock ld.so in place and only replaces the symbol lookup step, using glibc's rtld-audit interface: an audit library intercepts every shared-object lookup, including calls made through dlopen, before any filesystem search happens, and answers the question of which library satisfies a given symbol with a SQL query instead of walking RUNPATH and LD_LIBRARY_PATH. Because ld.so still performs the actual mapping and relocation, standard glibc features such as lazy PLT binding, IFUNCs, TLS and symbol versioning keep working; library storage becomes rows in a table, and library lookups become queries against it. The post demonstrates this by deleting a shared library file entirely from disk; running the dependent program normally then fails, unable to find it, but running it again with a prebuilt system database and the audit library loaded finds the dependency through SQL and runs successfully. The second approach, self-ld, is a from-scratch dynamic linker, again a small C program, that performs the entire lookup and binding step in SQL rather than delegating to ld.so. The author calls it a proof of concept, but says it works: it maps every loaded object's segments, publishes their exported symbols, and for each relocation, resolves the target with a SQL query that joins the relocation, symbol and object tables ordered by load order, then patches the program's global offset table and jumps to the entry point.

On cost, the post measures both size and startup latency against ELF. An unstripped SELF file carries SQLite's b-tree bookkeeping overhead and lands at roughly double the size of the equivalent ELF file, though most of that is recoverable because it sits in optional debugging and tooling tables; a stripped build of the coreutils SELF binary measures 1,794,048 bytes against the ELF version's 1,768,632 bytes, within 1 percent. Latency was benchmarked across binaries ranging from a 15 KiB hello program up to a 42 MiB build of gdb linking 47 libraries: there is a fixed cost of about 5 milliseconds to open the SQLite file and start the interpreter, plus a copy cost that scales with the size of the program image. That copy cost is worse than the raw numbers suggest, because SQLite's b-tree pages are not memory-mapped: two processes running the same SELF binary do not share text pages the way two processes running the same ordinary, memory-mapped ELF binary do, since each has to copy its segment bytes out of the b-tree rather than mapping them. The post also notes that a 274 KiB curl binary linking 27 libraries starts more slowly than a 4.6 MiB ELF build of git linking only 5, since startup cost scales with the number of linked objects rather than with raw size, a property of this style of linking that the author describes as a long-standing complaint.

The post's final section treats the executable not as a single binary but as a closure: a single file bundling a program together with all of its transitive dependencies. It starts from the observation that the output of ldd is ambiguous, listing only the library names a program needs rather than the specific files on disk that satisfy them. Nix already resolves that ambiguity for ordinary ELF binaries by pinning every dependency edge to a specific store path through RUNPATH, a topic the author says has been covered in earlier posts, including making that RUNPATH resolution redundant and making it faster. The post proposes doing the same inside SELF, adding tables that record every linked object's resolved path and, for each dependency edge, a reference that points at the exact object satisfying it rather than just a library name. The captured text breaks off mid-sentence inside that table definition, before reaching whatever conclusion the post states.

Key facts

  • SELF is a working prototype executable format that is a literal SQLite database: a compiled SELF binary reports as an ordinary SQLite file stamped with a distinct 4-byte application ID (0x53454c46) at header offset 68, and Linux runs it directly through the binfmt_misc subsystem.
  • A SELF file needs only two required tables to run, one for the ELF header and one for the load image; optional tables for sections, notes and dynamic metadata can simply be deleted, and doing so, plus a vacuum, is how the author implements strip, shrinking a sample binary from 57,344 to 49,152 bytes.
  • Two dynamic-linking prototypes exist: one keeps glibc's stock ld.so and swaps only the symbol-lookup step via the rtld-audit interface, demonstrated by deleting a shared library from disk and still resolving it through a SQL query; the other, self-ld, does the entire lookup and relocation in SQL and is explicitly called a proof of concept.
  • Benchmarked against ELF, a stripped coreutils SELF binary is 1,794,048 bytes versus 1,768,632 bytes for ELF, within 1 percent, while an unstripped SELF file runs roughly double the ELF size; startup adds a fixed cost of about 5 milliseconds plus a copy proportional to image size, since SQLite's b-tree pages are not memory-mapped.
  • The idea traces back to the author's PhD thesis and an earlier tool, sqlelf, along with an arXiv paper on it that was not accepted for publication; the author says recent progress in large language models made revisiting the idea worthwhile.

Why it matters

ELF is the established Linux executable format, and virtually every tool that touches a binary, the kernel's loader, ld.so, binutils, readelf, LIEF, goblin, reimplements its own parser and serializer for the same terse, packed format. The author's argument is that ELF already behaves like a database, it just built its own primitives by hand instead of using one: a bloom filter standing in for an index, ad hoc sections standing in for tables, no enforced schema. SQLite already solves those problems in a stable, self-describing, well-tested form that every developer tool already knows how to read. Replacing ELF with an actual SQLite file collapses that duplicated parsing work: reading tools become SQL queries, and tools that modify a binary, such as strip or patchelf, become simple, safe database transactions instead of fragile manual offset surgery on a tightly packed format.

Who it affects

This is aimed at people who build or maintain low-level tooling around executables: authors of linkers, loaders, debuggers and binary-analysis tools, who today each reimplement their own ELF parser, and engineers curious about SQLite's extensibility beyond its usual role as an application data store. The Nix and NixOS community is a direct audience too: the author frames SELF's dependency-resolution ideas around Nix's own RUNPATH handling and ships the ELF-to-SELF converter as an opt-in NixOS build hook. It does not change anything for an ordinary user or application yet; this is a working research prototype, not a toolchain component shipped in any distribution.

How to use it

The code is posted on GitHub, both the earlier sqlelf tool for querying ELF files with SQL and the newer SELF prototype: a converter called elf2self, the self-exec interpreter, and the self-ld linker. Running a SELF binary needs a Linux system with the binfmt_misc kernel subsystem registered to recognize SQLite's file signature plus SELF's application ID, which the post shows configured as a NixOS build option; elf2self itself is opt-in per package on NixOS through a build hook. There is no compiler or linker integration yet: elf2self only converts an already-built ELF binary after the fact, so producing a SELF file today is a post-build conversion step, not a native build target, and extending gcc or ld to emit SELF directly is described only as a possible future direction.

How solid is it

This is a working, runnable prototype backed by concrete demonstrations rather than only a design sketch: the post shows a compiled binary being identified as a SQLite database, run directly, and queried with plain SQL for its dependencies, its segments and its stripped size. The dynamic-linking claim is demonstrated concretely too, deleting a shared library from disk and then resolving and running the dependent program purely through a SQL-backed audit library. Benchmarks span a real range of binaries, from a 15 KiB program up to a 42 MiB build of gdb linking 47 libraries, with specific size and latency numbers rather than vague claims. The author is also candid about the prototype's limits: self-ld, the from-scratch SQL linker, is explicitly called a proof of concept rather than a finished replacement for ld.so, and an earlier academic paper on the related sqlelf tool was not accepted for publication. The whole account is a single independent author's own self-reported work and benchmarks, built on ideas going back to a PhD thesis, and the captured text ends before the post's own concluding section.

Risks and caveats

The performance picture has real tradeoffs, not just overhead that stripping erases. Because SQLite's b-tree pages are not memory-mapped, two processes running the same SELF binary do not share text pages the way ordinary ELF binaries do under mmap; each has to copy its segment bytes out of the b-tree instead. Startup latency also scales with the number of linked libraries rather than with binary size, illustrated by a smaller curl binary starting more slowly than a larger, ELF-native git build because curl links more libraries. An unstripped SELF file runs roughly double the size of the equivalent ELF file, and stripping it back down is an extra explicit step, not the default. Structurally, self-exec itself has to remain an ordinary ELF binary, or it would recurse into an ELOOP error, and there is no compiler or linker that emits SELF natively yet, only a converter that operates on already-built ELF binaries. The post does not address security or trust implications, such as SQL-injection risk inside the dynamic linker's own queries or the trust boundary of an executable format that is also a writable database, and every mechanism it describes, binfmt_misc, the glibc audit interface, the NixOS registration, is Linux-specific, with no claim made about other platforms.

“ELF is already a database. It just implements many database primitives by hand, along with a surprising number of data structures for performance, like a bloom filter for symbol lookup.”

— the author of the blog post