Principles for fast Tokio applications

A blog post titled "Principles for Fast Tokio Applications," signed by an author identified only as Russell and published on the dial9 blog, sets out to collect practical rules for writing async Rust code that performs well on Tokio runtimes. The post grew out of a discussion at the RustConf Unconf about debugging and benchmarking async applications; the author frames it as a first draft of a living document, inviting readers to file issues or open pull requests, and plans to add a sample app demonstrating the issues alongside what a dial9 trace of them looks like.

The core argument is that there are few hard rules for Tokio performance, because the behavior of any given task depends on everything else running on the runtime at that moment, which is why so many problems only surface in production. Writing fast async applications means balancing fairness against batching, and contention against isolation. The author reports that almost every real application they have examined has poll durations well beyond the 10 to 100 microseconds that Alice Ryhl recommends in her post "What is Blocking?", but stresses that this alone does not mean there is a problem: fixing long polls that do not affect a metric you actually care about accomplishes nothing, so the first step is always to identify a real metric to improve. In the author's experience, the root cause is usually the application code itself, often in how components of a distributed system interact, rather than Tokio; the dial9 tracing tool is cited as demonstrating the absence of a Tokio problem about as often as it finds one. Among Tokio's own metrics, the schedule latency histogram, the time between a task becoming ready and Tokio actually polling it, is singled out as the most useful symptom to watch, even though it does not reveal the cause.

The post then works through a series of principles. On latency versus throughput: yielding more often improves fairness between connections. In a pipelining scenario such as Redis, a naive implementation reads all buffered data off a connection before yielding, so one client's pipelined batch can make every other client's requests wait; explicitly yielding after each request, for example by calling tokio::task::yield_now, cut latency by roughly 10 times in the author's example while leaving throughput largely unchanged, since the same total work still gets done. A P99 latency far above the P50, unusually long polls, or many tracing spans falling inside a single poll are given as signs of this problem.

On batching to amortize overhead: every runtime event, whether switching tasks, polling, or moving work between workers, carries a cost, so the more useful work packed into each one, the more efficient the application. The author calls out tokio::fs specifically, noting that without io_uring, Tokio runs every filesystem operation on its blocking pool, and each call to spawn_blocking has its own cost against a pool shared by the whole runtime; batching filesystem or other blocking work into the largest sensible segment, or moving it to a dedicated OS thread, is recommended. The same logic applies to spawning tasks: spawning is cheap per call, but each spawned task is separate overhead for the runtime to track, so turning a 10-microsecond unit of work into its own task is described as counterproductive.

On shared resources: the blocking pool and the global task queue are both runtime-wide, so they can become bottlenecks under load. The author reports seeing negative performance effects starting at roughly 50,000 blocking tasks per second on a 32-core host, while noting this figure will vary by workload. Tasks land on the global queue when local worker queues overflow or when work is scheduled from outside a runtime worker, such as a channel sender running on a non-Tokio thread; a consistently deep global queue is the tell.

On locking: blocking a worker on a contended mutex can stall an entire runtime, a risk the author illustrates with a metrics registry behind a lock, where a slow flush can leave every worker eventually stuck trying to record a metric on the same lock, making work stealing impossible because every worker is occupied. The guidance is to keep critical sections in async code extremely short, avoid RWLocks because they create contention on atomics even for reads, and reserve tokio::sync::Mutex, which is more expensive to lock and can hit subtler issues like FutureLock, for critical sections lasting multiple milliseconds.

On concurrency: Tokio will happily spawn more tasks than the rest of a system can absorb, and the author cites accidentally opening 3,000 concurrent connections to S3 from an unbounded task fan-out as a common failure mode; the fix is usually a plain semaphore rather than a fancier adaptive scheme.

On isolating Tokio from other work: under a heavily loaded operating system, the kernel can take 10 to 20 milliseconds or more to schedule a Tokio worker after it is woken, which is disastrous if the application targets single-digit-millisecond P99 latency. The author describes observing this during an incremental Java-to-Rust migration at Amazon, where the Rust process got faster the less work the co-located Java process did, an effect that grew stronger the more threads the other application used. Background Rust threads can cause the same problem: those used by tracing_appender, for instance, can sometimes run for more than 100 milliseconds without yielding the CPU, delaying a Tokio worker the kernel is trying to wake. The recommended fix in both cases is to use cgroups or similar mechanisms to pin Tokio workers and other work to separate cores, noting that Tokio rarely needs every core on a host.

A final section, on deliberately breaking these rules, argues that blocking the executor can be fine under light load, since Tokio's work stealing can absorb one worker being busy longer than usual; that breaks down once the runtime or the operating system is heavily loaded and workers cannot be unparked or work cannot be stolen quickly enough, which can delay core runtime maintenance like driving I/O. The author adds an explicit caveat that this does not apply inside constructs like tokio::join! or tokio::select!, where there is no work stealing within a single task, so blocking there stalls everything else on that task and can produce unexpected timeouts. The post then turns to using multiple runtimes to isolate latency-sensitive and lower-priority workloads by pinning them to dedicated cores, but the captured text breaks off mid-sentence at that point, so the remainder of that section is not available here.

Key facts

• Explicitly yielding after each request in a pipelined-read scenario like Redis cut latency by roughly 10 times in the author's example, while leaving throughput largely unchanged.
• The author reports negative performance effects from the shared blocking pool starting at roughly 50,000 blocking tasks per second on a 32-core host.
• Under a heavily loaded OS, the kernel can take 10 to 20 milliseconds or more to schedule a woken Tokio worker; background threads such as those used by tracing_appender can sometimes run over 100 milliseconds without yielding the CPU, making the same delay worse.
• An unbounded task fan-out accidentally opening 3,000 concurrent connections to S3 is cited as a common failure mode, fixed with a semaphore to bound concurrency.
• Alice Ryhl's post "What is Blocking?" recommends keeping poll durations to 10 to 100 microseconds, a bar the author says almost every real Tokio application they have examined exceeds, not always to the detriment of the metrics that matter.

Why it matters

Async Rust performance problems are notoriously hard to pin down because a task's behavior depends on everything else sharing the runtime at that moment, which is why issues so often appear only in production. This post tries to turn a RustConf Unconf conversation among practitioners into a reusable checklist rather than leaving that knowledge scattered across individual debugging sessions, explicitly framed as a first draft meant to be extended through issues and pull requests.

Who it affects

Developers building latency- or throughput-sensitive services on Tokio, particularly in distributed systems where problems emerge from how components interact rather than from Tokio itself. It is most directly useful to teams already instrumenting their applications with tools like dial9's tracing or tokio-metrics, since several of the diagnostic signals described, such as the schedule latency histogram or spans clustering inside one poll, depend on having that visibility in place.

How to use it

The post's recommendations: yield explicitly after pipelined reads to restore fairness between connections; batch filesystem and other blocking calls into the largest sensible segment instead of many small spawn_blocking calls, since Tokio routes filesystem work to a shared blocking pool without io_uring; avoid spawning a separate task for very small units of work; bound concurrency with a plain semaphore rather than letting task fan-out run unchecked; keep async critical sections extremely short, avoid RWLocks, and reserve tokio::sync::Mutex for sections that genuinely last multiple milliseconds; and, when Tokio shares a host with other heavy threads or processes, pin Tokio workers to their own CPU cores with cgroups, or split latency-sensitive and lower-priority work across separate runtimes.

How solid is it

The claims rest on one practitioner's own debugging experience plus a group discussion at a conference, not on a formal benchmark suite; figures like the 50,000-blocking-tasks-per-second threshold and the 10x latency improvement are presented as approximate, observed numbers from specific cases ("roughly," "I have seen") rather than controlled measurements, and the author repeatedly qualifies the advice with "it depends." The post is explicitly labeled a first draft still open to correction, and the version captured here cuts off mid-sentence in the closing section on using multiple runtimes, so that section's guidance is incomplete as retold.

Risks and caveats

Nearly every rule in the post comes with a stated exception: blocking the executor is described as sometimes fine, but only under light load, and it turns actively harmful once the runtime or the OS is saturated or once code sits inside a single-task construct like tokio::join! or tokio::select!, where there is no work stealing to fall back on. tokio::sync::Mutex is flagged as trading one problem (cheap but contended locking) for another, including exposure to a subtler issue the author calls FutureLock. Because the source text is cut off before the multiple-runtimes section concludes, any additional caveats the author raised there are not captured.

“Keep critical sections in async applications extremely short (e.g., a single hashmap update). RWLocks are almost never the right primitive to use as they still create contention on atomics, even for the read path.”

— Russell, dial9 blog