Tinybird distills years of lessons from running ClickHouse at scale

Javi Santana, co-founder of Tinybird, a company built on top of ClickHouse, has published the first of a two-part account of running ClickHouse clusters in production, the second part already published and linked from the post. He says he has operated a ClickHouse cluster at Tinybird since version 18.4, almost six years by the time of writing, though the post's own headline instead states five years operating ClickHouse clusters, a gap the source does not explain. He adds that he wrote his first ClickHouse blog post, on geospatial analysis, almost eight years ago. The post opens with a disclaimer that Tinybird has no affiliation with ClickHouse, Inc., the database's sole maintainer and trademark owner; Tinybird describes itself instead as a contributor of new engines, performance improvements and distributed joins to the open-source project, and a user of it. The author credits someone named Alexey, no surname given in the post, with starting the ClickHouse project. His stated thesis: setting up a ClickHouse cluster is easy, and keeping it running is the hard part; the rest of the post works through where that difficulty actually shows up, in architecture, storage, upgrades, testing, cost planning and data ingestion.

On architecture, ClickHouse's standard design splits data into buckets, for example by hashing a user ID, assigns each bucket to a shard, and gives each shard one or more replicas. Santana says the standard approach is starting to feel like the wrong one for these kinds of systems, pointing to a broader shift toward separating compute from storage using cloud object storage, though he says he does not fully agree even as he concedes the cost and management advantages. Tinybird itself started years ago with no shards, only replicas scaled vertically on local SSDs, since cloud storage was not yet an option; it never added shards because re-sharding proved difficult and no customer's workload required it. The setup routes traffic through an HTTP load balancer rather than ClickHouse's native TCP protocol, a choice made to reuse standard, battle-tested tooling. To illustrate why this architecture gets expensive, Santana walks through a hypothetical, not a real Tinybird figure: a 300 terabyte table needing 1,000 queries per second, split across replicas that can each handle 100 queries per second, requires 10 replicas and therefore 3,000 terabytes of total storage if every replica holds a full copy. His recommendations: keep one replica dedicated purely to writes, a pattern he calls compute-compute separation, and isolate any workload that needs a stable p99 latency to its own replica kept under 40% load.

On storage, Santana argues that ClickHouse's open-source edition lags behind an industry standard set by Snowflake more than a decade ago of separating compute from storage via cloud object storage; he names StarRocks as further ahead than ClickHouse on this front. ClickHouse supports storing data locally, replicated between nodes through ZooKeeper, or in a central store like S3, either with each replica holding its own copy or through zero-copy replication, where replicas point at shared data instead of duplicating it. Santana says zero-copy replication was contributed by someone outside ClickHouse, Inc., that the company appears not to like it, for what he calls good reasons, since it is buggy, can lose data and can leave garbage behind in S3, and that ClickHouse, Inc. once planned to remove the feature before deciding not to. ClickHouse Cloud has its own storage layer instead, and it is not open source. Tinybird uses a modified version of zero-copy replication in a private fork, layered with local SSD caching and, for latency-sensitive customers, a hot-and-cold architecture combining local SSDs with S3, since S3 alone cannot match the latency SSDs provide. He also says ClickHouse's default way of writing to S3 is inefficient, generating heavy write-operation volume because the storage engine was designed around local storage, where writes are treated as free aside from IOPS, rather than tiered cloud storage. His overall recommendation is to use zero-copy replication for compute-storage separation while watching it closely, or to fall back to a hot-and-cold setup or plain SSDs for anyone not comfortable with the risk; for compression, he recommends ZSTD level 1 or 2 for the best balance of speed and compression, better than LZ4 in most cases, applied per column and periodically re-tested against other codecs.

On upgrades, Santana says Tinybird's first ClickHouse update took three hours of work after two weeks of preparation. Over time the team learned to upgrade without downtime, without dropping queries, and without data loss, and eventually without performance degradation, folded directly into its CI/CD pipeline, a progression that took four years in total; he says that, as far as he knows, no other company has reached the same point. His workflow: bring the new version up on one new replica, watch its logs closely, avoid schema changes on it, hold off on any new feature until the whole cluster matches, send it read traffic, then write traffic, before rolling the update out to the remaining replicas. This depends on ClickHouse's replication protocol staying backward-compatible, something Tinybird has invested in maintaining and testing in CI, occasionally catching problems upstream in time to submit a fix before a ClickHouse release ships. The problems that still occur, by his account: incompatible data-storage-format changes, rare but real, having hit Tinybird two to three times in the last two years, of which at least two were fixed; he says Tinybird notices these because its multi-tenant customer base exercises every possible combination of data type, and that a typical, non-multi-tenant deployment is unlikely to hit the same issue. Other recurring risks are SQL behavior changes from upstream bug fixes, which he checks against ClickHouse's own query log; performance regressions in either direction; and settings changes, which he says need close review release to release. His advice is to wait at least a month after any release before upgrading and to avoid experimental features without knowing the risk. Tinybird validates upgrades by running its test suite across mixed-version clusters in CI and by replaying every customer query against the candidate version once a day, fixing what breaks or contacting the affected customer, before automatically rolling out an update once everything passes.

On configuration, Santana says operating ClickHouse well requires reading its source code directly, something he says was not necessary running the hundreds of Postgres clusters he oversaw as CTO of an earlier company; Tinybird tracks setting changes with an automated system built for that purpose. His testing recommendation is to run CI against three ClickHouse builds at once, the version in production, ClickHouse's master branch, and the version being considered for upgrade, and to test exclusively against a real multi-machine cluster with ZooKeeper or ClickHouse Keeper rather than a single instance, since a lone node behaves differently: table creation failures and replication lag only show up on a cluster. On cost, he lists the line items to plan for: the replicas and shards themselves; at least three ZooKeeper replicas, kept on separate hardware from the database since he treats a ZooKeeper failure as effectively fatal to cluster operation; storage, whether local, S3 or both, factoring in that disks cannot be resized down and that S3 charges per operation; a small, highly available load balancer; and backup storage, whose cost depends on retention policy and data volume. As a sizing rule of thumb, he estimates a 32-core machine can healthily process up to about 5 gigabytes per second of uncompressed data. On staffing, a small cluster needs only a part-time person, but ingest rates above about 20,000 rows per second, combined with people actively pushing schema changes, tend to require a full-time engineer; following his basic operating rules, he says, can cut the hardware you need by 3 to 4 times, and good ClickHouse and data-infrastructure engineers are hard to find through generic hiring pipelines. He does not recommend running a cluster solo if it can be avoided, and says that sparing customers exactly this workload is why Tinybird exists, though he is explicit that Tinybird itself does not offer ClickHouse hosting: it solves the analytics problem instead, which he calls narrower than what a data warehouse solves and something Tinybird does much better, and he acknowledges that a reader who has made it this far into the post probably wants to do it themselves anyway.

The post's longest section is on ingestion, which Santana says every company running ClickHouse struggles with, typically losing data without realizing it, duplicating data, or taking the database down outright. Every insert creates a new data part; parts are merged in the background in a process he compares to compaction in Iceberg, and larger parts make reads faster but make merges slower and more resource-intensive, so the two pull against each other. Larger insert batches reduce the number of parts generated but add latency and memory pressure on the insert path; the failure chain he describes is a CPU or IO-heavy query competing for resources while inserts pile up and memory climbs, ending in an out-of-memory crash and, with it, data loss. His recommendations: batch inserts to land close to a single part per insert, batch by partition with staggered flush schedules where possible, run frequent inserts only on the tables that need them, use compact parts especially over S3 to cut write-operation costs and avoid S3 rate limits, route early writes through a small local hot disk to save on cloud costs, and tune the maximum part size carefully to avoid excess merges without oversizing parts. He calls table design inseparable from ingestion, since a bad partition key or excessive write volume can bring a cluster down on its own. Materialized views carry a related risk: one that is even slightly misconfigured and uses more memory than expected can trigger the same out-of-memory failure, which is why Tinybird built a system that automatically disconnects a materialized view when this happens. Other problems he lists: tables stuck in read-only mode, usually while a replica is still starting up, occasionally for no clear reason, fixed by dropping and recreating the table and letting replication resync it; too many parts generated at once, for example inserting three years of day-partitioned data in a single load, addressed by writing into one partition at a time instead; and a merge queue that cannot keep pace with inserts, addressed by increasing the thread pool, adding backpressure, or loosening queue-size limits. On backpressure specifically, he says some teams put Kafka in front of ClickHouse, which works but can get expensive; Tinybird built a custom system instead, citing its multi-tenant cost structure and the need to tune ingestion per table, and states plainly that running ClickHouse in production without some backpressure mechanism means any small incident turns into data loss. The post closes by naming what its published second part covers: operating the cluster and monitoring, including connecting Grafana for visualization, handling load, performance tweaks, and other common issues, none of which this post itself addresses.

Key facts

Javi Santana, Tinybird's co-founder, has run its ClickHouse clusters since version 18.4; the post's headline says five years of operating them, while the opening line instead puts the span at almost six.
Zero-copy replication, which lets ClickHouse replicas point at shared data in S3 instead of each copying it, was contributed by someone outside ClickHouse, Inc.; the author calls it buggy and capable of losing data or leaving garbage in S3, yet still recommends it over plain local-disk replicas for the cost savings.
Tinybird's first ClickHouse cluster update took three hours of work after two weeks of preparation; reaching zero-downtime, zero-data-loss upgrades built into CI/CD took four years, and the author says he knows of no other company that has done the same.
On ingestion, the author's core warning is that a stalled query plus piling-up inserts plus rising memory ends in an out-of-memory crash and data loss; his fix is batching inserts to land close to one data part per insert rather than many small ones.
Rules of thumb from the post: a 32-core machine handles about 5 gigabytes per second of uncompressed data, ingest rates above 20,000 rows per second usually call for a full-time engineer, and following the post's basic rules can cut the hardware needed by 3 to 4 times.

Why it matters

This is a named, on-the-record engineer's account of years spent keeping ClickHouse running in production, backed with specifics rather than general assurance. Its own thesis, stated at the top, is that setting up a ClickHouse cluster is easy and keeping it running is the hard part; what follows backs that claim with detail: a replication feature the author says ClickHouse, Inc. itself is uneasy about, an ingestion pipeline that can silently run out of memory and lose data, and an upgrade process that took four years to make safe. Santana writes from inside Tinybird, a company built specifically around operating ClickHouse for customers, so the account is also, by his own admission, part of the case for not running ClickHouse yourself.

Who it affects

Anyone running ClickHouse in production at meaningful scale, especially multi-tenant setups like Tinybird's own, where the post says incompatible data-storage-format changes surface precisely because customers exercise every combination of data type. It also affects ClickHouse, Inc., the project's sole maintainer, since the post says ClickHouse, Inc. appears not to like zero-copy replication, the feature Tinybird relies on for cost-effective cloud storage, and at one point planned to remove it before deciding not to. Teams deciding how to staff a ClickHouse deployment are a direct audience too: the post puts the threshold for needing a full-time engineer at ingest rates above about 20,000 rows per second, and says good ClickHouse and data-infrastructure engineers are hard to find through generic hiring pipelines. It also affects Tinybird's own customers and prospects indirectly, since the post doubles as an argument for using Tinybird's managed analytics layer instead of operating ClickHouse directly. The author scopes his own audience narrowly, saying the post is aimed at people who already handle ClickHouse clusters rather than a general readership.

How to use it

The post reads like a working checklist. On architecture: start with replicas only and add shards only once a single query needs to span more data than one machine can hold; use HTTP rather than ClickHouse's native TCP protocol if you want to reuse ordinary load-balancing and client tooling; dedicate one replica purely to writes, a pattern the post calls compute-compute separation; and if a workload needs a stable p99, isolate it to its own replica and keep that replica under 40% load. On storage, the post's default compression choice is ZSTD level 1 or 2, better than LZ4 in most cases, tested periodically against other codecs since results vary by data; for cloud storage, it recommends zero-copy replication over duplicating data per replica, but only for those prepared to watch it closely for the data-loss risks described below, falling back to a hot-and-cold local-SSD-plus-S3 setup or plain local SSDs otherwise. On upgrades, the workflow is to bring up the new version on one new replica first, watch its logs, avoid schema changes and new features until the whole cluster matches, move read traffic before write traffic, and update the remaining replicas last, waiting at least a month after any ClickHouse release before adopting it. On testing, run CI against the production version, ClickHouse's master branch and the target upgrade version together, and never validate against a single instance: a real cluster needs at least two machines plus ZooKeeper or ClickHouse Keeper, since single-node behavior does not match cluster behavior. On ingestion, the rule is to batch inserts so each one produces close to a single data part, batch by partition with staggered flush schedules, use compact parts especially over S3, and build some backpressure mechanism, whether that means putting Kafka in front of ClickHouse or, as Tinybird did, a custom system, since without one a small incident turns into data loss.

How solid is it

This is a single, first-person account from a named, on-the-record author, Javi Santana, published under his own byline as Tinybird's co-founder on Tinybird's company blog. It carries a small internal inconsistency: the post's own headline claims five years operating ClickHouse, while its opening line instead states almost six, since version 18.4; the source does not reconcile the two. The concrete numbers used to illustrate replica economics, a 300 terabyte table, 1,000 queries per second, 10 replicas, 3,000 terabytes of total storage, are explicitly a hypothetical the author constructs to make a point, not disclosed figures from Tinybird's real clusters; the post never gives an actual size, query rate or total storage figure for its own production systems, describing them only in general terms as petabyte-scale. Per the page's own metadata, the post was first published in April 2025 and last modified in January 2026, so it is not new material, and it is stated to be the first of a two-part series, with monitoring, load handling and performance tuning explicitly deferred to a second post this one does not cover. The claims throughout rest on the author's own operational experience and voice, told in the first person as a single practitioner's account rather than a benchmarked study.

Risks and caveats

Several of the risks are severe by the author's own account. Zero-copy replication, the feature Tinybird uses for cost-effective cloud storage, was contributed by someone outside ClickHouse, Inc., and the post says it is buggy, can lose data, and can leave garbage behind in S3; ClickHouse, Inc. itself is described as uneasy about the feature and as having at one point planned to remove it. The post treats a ZooKeeper failure as effectively fatal to a cluster's operation, which is why it recommends running ZooKeeper on hardware isolated from the database nodes. On ingestion, the chain the post warns about, a CPU or IO-heavy query, inserts piling up behind it, memory climbing, then an out-of-memory crash, ends explicitly in data loss, and materialized views carry their own version of the same risk since a slightly misconfigured one can OOM the server on its own. Incompatible data-storage-format changes across ClickHouse versions have hit Tinybird's own multi-tenant cluster two to three times in the last two years, though the author calls this unlikely for a typical, non-multi-tenant deployment. Much of the guidance here is the author's own heuristic rather than a benchmarked universal constant: the 40% load ceiling for p99-sensitive replicas, the month-long wait after a release, and the 20,000-rows-per-second staffing threshold are all offered as general advice rather than thresholds measured outside Tinybird's own environment. And the piece doubles as an argument for Tinybird's own product: it is published on the company's blog by its co-founder, who states plainly that sparing customers this operational load is why Tinybird exists, so the case against self-hosting ClickHouse is also a case for paying Tinybird instead.

“Setting up a cluster is easy, the hard part is keeping it running.”

— Javi Santana, Tinybird co-founder, in the post