Glonce

Waymark compiles Python workflows to bytecode instead of replaying them

toolsThe Glonce editorsHacker News🔥44Glonce rating44Impact35Novelty45Relevance70Surprise25Players5Credibility65
Waymark compiles Python workflows to bytecode instead of replaying them

Waymark is an open-source library for durable background tasks: jobs that withstand server restarts, task crashes and long-running work. It is built on Postgres, with no other infrastructure to deploy, and has a Rust core. Python is the first supported language; the README says more are coming but does not name them. The project was shown on Show HN, and the material here is its GitHub README.

The central design choice is how it differs from replay-based engines such as Temporal or Vercel Workflow. The first time a workflow is run or scheduled, it is compiled to an intermediate representation (IR); in Python, the SDK parses the AST of the workflow's run() method. The IR captures the control flow (loops, conditionals, parallel branches) and is lowered to bytecode for a durable virtual machine. The Rust runtime executes that bytecode and snapshots the VM's state to Postgres as it goes, so after a crash a workflow resumes from where it was. The original workflow definition is never re-executed during recovery.

The README presents this as a practical benefit: if a workflow compiles, it will run as advertised. Non-deterministic standard-library calls such as time or uuid do not need workarounds; the compiler raises an error asking you to move them into an explicit action. A few patterns still slip past the compiler. They are listed as known issues and tracked in issue #772.

The programming model has two primitives. Actions are the distributed work: plain async functions (network calls, database queries, anything that can fail and be retried independently), sent to workers, with dependencies injected through Depends. Workflows are the durable control flow: plain async Python classes with a run() method that orchestrate actions but do not do heavy lifting themselves. The README's example fetches users, filters out inactive ones, then fans out welcome emails in parallel with asyncio.gather.

Workflows can do several things. By default an action runs once, like a normal function call; wrapping it in self.run_action(...) adds a retry policy and a timeout (the sample uses RetryPolicy(attempts=5, backoff_seconds=10) and a two-minute timeout, which are example values, not defaults). if/elif/else, for and while compile into the workflow program. asyncio.gather runs actions in parallel, and asyncio.sleep pauses a workflow durably across restarts, for a second or for a day. Workflows can run on a cron expression or fixed interval via schedule_workflow, with no extra infrastructure. The waymark-executor process also serves a built-in webapp showing every workflow instance, a timeline of each one's calls, and the load on the nodes. The docs also list an AI Agents page: a wrapper around pydantic-ai for declaring agents natively.

Getting started: all client and server binaries ship in one Python package, installed with uv add waymark. You set WAYMARK_DATABASE_URL to a Postgres database, optionally set WAYMARK_HTTP_ENABLED=true to get the webapp on localhost:24119, and run uv run waymark-executor. A workflow is then started from any Python process with WelcomeEmailWorkflow().run(user_ids). Nothing executes inline in the web server: the run is queued to Postgres and executed by the node.

The README's author gives the motivation in the first person. After trying most of the ecosystem over the last 3 years, the author believes background jobs should offer easy-to-write control flow as ordinary code, be simple to test locally and deploy remotely, and have reasonable defaults that scale to a reasonable request volume without tuning. Control flow should not be forced into a DAG definition with decorators or custom syntax. Existing robust queues, including ones like Temporal.io and RabbitMQ that scale to millions of requests a second, almost all need a dedicated task broker hosted next to the app. The README says that can get complex to tune for production, that cloud hosting of most of them is billed per event and can get very expensive, and that they tend to force you to fit the framework's conventions. It calls RabbitMQ battle tested over decades and Temporal well resourced, and both great choices for different scopes.

The authors set modest goals. They do not expect best-in-class load performance; they want it to scale most applications well past product market fit. Performance is described as a top priority, with a Rust core and continuous benchmarks on CI, but the README adds that there is only so much to do with Postgres as an ACID store. Once you tax Postgres you are probably at the scale where a more complicated architecture is warranted.

It recommends Waymark if you already use Postgres and a supported language, have async-heavy logic that must be durable and retryable (third-party API calls, database jobs), want identical behaviour locally and remotely, want job code to fit existing unit tests and static analysis, and care more about product market fit than scale. It advises against it for latency-sensitive jobs needing under 100ms acknowledgement and handling per task, for more than roughly 10k actions coordinated concurrently, and for teams that need to scale an existing task coordinator by another 10x.

The README carries an explicit warning: right now Waymark should not be used in any production application. The spec is changing quickly and backwards compatibility is not guaranteed before 1.0.0. The authors invite people to try it in a side project and to file an issue if a workflow that should compile does not. Building from source needs Node.js and npm, because the webapp is compiled into waymark-executor, plus make js-deps and cargo build --bin waymark-executor.

Key facts

  • Waymark is an open-source durable background-task library built on Postgres with a Rust core, needing no other infrastructure; Python is the first supported language.
  • Instead of replaying workflow code like Temporal or Vercel Workflow, it compiles a workflow's run() method into an IR, then bytecode for a durable VM whose state is snapshotted to Postgres; the original definition is never re-executed on recovery.
  • Non-deterministic calls such as time or uuid raise a compile-time error asking you to move them into an explicit action; a few patterns that slip past the compiler are tracked in issue #772.
  • The README says to avoid it for jobs needing under 100ms acknowledgement per task or for more than roughly 10k concurrently coordinated actions, and does not expect best-in-class load performance.
  • The authors say it should not be used in production yet: the spec is changing quickly and backwards compatibility is not guaranteed before 1.0.0.

Why it matters

Durable execution engines usually work by replaying workflow code against a recorded history, which is why workflow code has to be deterministic. Waymark tries another route: compile the control flow once and run it on a durable virtual machine whose state is stored in Postgres. It also aims at a gap the README describes, where robust queues need a dedicated broker and can get expensive in the cloud, by asking only for a Postgres database you probably already run. It is an early project, so the interest is in the design rather than in a proven product.

Who it affects

The README targets teams that already use Postgres and Python, have async-heavy work such as third-party API calls and database jobs that must survive crashes and be retried, and are focused on reaching product market fit rather than on large scale. It is not aimed at latency-sensitive jobs, very high concurrency, or teams outgrowing an existing task coordinator.

How to use it

Install with uv add waymark. Write actions as @action async functions and a workflow as a @workflow class with an async run() method. Set WAYMARK_DATABASE_URL to a Postgres database (and WAYMARK_HTTP_ENABLED=true if you want the webapp on http://localhost:24119), then start the node with uv run waymark-executor. Start a workflow from any Python process by calling the workflow class's run() with its arguments; the run is queued to Postgres and executed by the node. Retries and timeouts are added by wrapping a call in self.run_action(...), schedules use schedule_workflow with a cron expression or interval, and the docs list a pydantic-ai wrapper for AI agents. Treat it as something to try in a side project for now.

How solid is it

The only material is the project's own README, so every claim is the authors' description rather than independent testing. The README states that benchmarks run continuously on CI but gives no results, and no comparative numbers against Temporal or other engines. The project says plainly that it is not for production, that the spec is changing quickly, and that backwards compatibility is not guaranteed before 1.0.0. The compile-time guarantee ("if your workflow compiles, your workflow will run as advertised") is the authors' claim, qualified by a known-issues list for patterns the compiler still misses.

Risks and caveats

Not production-ready, by the authors' own statement. The compiler does not yet catch every problematic pattern (tracked in #772), so a workflow that looks valid may still misbehave. The authors say Postgres as an ACID backing store limits scale, and they list sub-100ms acknowledgement, more than roughly 10k concurrent actions and 10x growth beyond an existing coordinator as cases to avoid. Python is the only supported language today. Building from source requires Node.js and npm because the webapp is compiled into the executor.

“Right now you shouldn't use waymark in any production applications.”

— Waymark README