Memdoor workflow: Qwen3.8-27B team tops Fable 5 on 10 hard problems

The Memdoor repository includes an example workflow called team-solve: a multi-agent harness for programming problems. It has five roles (a planner, two ideators, a solver and a manager) plus a repair worker. Each starts from a fresh context and communicates with the others only through files on disk. A shell script that runs the sample tests decides success, not a model. The method is that of GVS5H (Lee, 2026), a paper whose title claims that five Qwen3.8-27B models match Fable 5 on LiveCodeBench Hard.

The repository's author rebuilt the authors' harness as a Memdoor workflow on the same model, Qwen3.8-27B, with a 128K reply cap and the authors' exact prompts and loop. They ran the first ten problems of the paper's LiveCodeBench Hard list, five passes each, and graded every program on every public and hidden test with the authors' corrected grader. The loop scored a mean of 9.8 of 10 over the five passes. The authors' own harness scored 9.6 and Fable 5 alone scored 8.0; both figures come from the authors' regraded result files, not from new runs. In the rebuild, 49 of 50 runs solved their problem. The 50 counted runs cost $35.79 (15.3M output tokens), or $38.77 including 16 attempts that were cut by a clock, stopped for a fix or paused. A typical run took a median of 40 minutes of model time and cost $0.39; the longest took 4 hours. All 50 runs are published in bench/results/parity-5pass/, with every request and reply, the program, the grader's full output, each call's cost and the host that served it.

The README says what is the same and what is different. A test file, lib/test_parity.py, runs the authors' multiagent_solve and the Memdoor workflow against one scripted model and compares each request; across four cases they were identical. The temperatures (plan 0.3, ideation 0.4, manager and worker 0.2), top_p 0.95, top_k 20, the 128K cap, up to 10 rounds, summarising a cut-off worker, and answering an empty reply with its reasoning all match. The host differs. The authors used their own vLLM in FP8; the rebuild went through OpenRouter, asking Modal, CoreWeave, Parasail and Wafer in that order until Modal slowed to 14 tokens/s, then using only the FP8 hosts. In total 203 calls were served by Modal, 27 by CoreWeave and 115 by Wafer, and three runs were moved mid-run with their finished steps kept.

Getting there took some work. The first version wrote its own task file around each problem, and the 27B model planned worse from it than from the authors' exact LiveCodeBench text: on arc196_c the plan scored 0 of 2 against 2 of 2. Four Memdoor changes let a workflow ask what the authors' harness asks: a task that answers with the model's reasoning when it wrote no text, top_p and top_k on a model line, mark_cutoff for a reply cut at the cap, and a call that runs on its own four-hour clock. The coder, a Memdoor agent, wrote the workflow files itself from a description of the method (2026-10-25); nobody hand-wrote them.

The write-up lists where the loop lost points, twice, in the same two gaps of the authors' loop. In pass 5 on abc399_e (32 of 44 hidden tests), round 1's program was right, but the manager invented a counterexample with a wrong expected answer, the team rewrote the program, and the new one passed the samples, which do not cover the case. In pass 4 on arc196_c and arc196_d, a program that passed every hidden test was overwritten by a later round because the manager did not say done. Both got a correct program back before the loop ended (arc196_d at round 8, arc196_c through the final call), at $3.44 and $4.24, the two most expensive runs. The README says both gaps can be fixed without changing the method, by checking a manager's counterexample against a brute force on small inputs and by keeping the last program that passed the samples. Neither fix is in the reported numbers.

The README is explicit about limits. It covers ten problems with five passes each, not the authors' 100. The contests date from 29 March to 6 April 2025, and the model may have seen them in training, so contamination cannot be ruled out. Fable 5's column is a single call per problem from the authors' files; it was not run in this harness. The loop sees only the public samples, while the score is the hidden tests.

The repository also describes a small sample run: counting substrings of a string over {a, b, c} with equal counts of each letter, up to 200,000 characters. It finished 8 of 8 steps with 5 of 5 samples passing. The solver took ideator A's prefix-state method, which is O(N), and a brute force on 300 random strings found no mismatch. It runs in 0.04 s at N = 200,000. That run exposed three bugs. One ideator's provider dropped the stream mid-answer and the run died because the coder had set max_retries: 0; agent steps now carry max_retries: 1, and the run was resumed with --partition. The coder's ideator prompts named the sample's methods and would have steered other problems the wrong way, so they now ask A for the most direct provable algorithm and B for one from a different family of technique (2026-10-28). And early attempts failed every step in 0 s with "no agent named coder", because a run looked its agent up only in its own workspace; that was fixed in the gateway the same day (v1.692.0).

Key facts

Rebuilt with the authors' prompts on Qwen3.8-27B, the loop scored a mean of 9.8 of 10 over five passes on the first ten LiveCodeBench Hard problems, against 9.6 for the authors' own harness and 8.0 for Fable 5 alone.
49 of 50 runs solved; the 50 counted runs cost $35.79 (15.3M output tokens), or $38.77 with 16 cut, stopped or paused attempts. The median run took 40 minutes and $0.39.
The harness has a planner, two ideators, a solver, a manager and a repair worker, each with a fresh context, talking only through files; a shell script running the samples decides success.
Caveats from the README: ten problems, not the authors' 100; contamination cannot be ruled out; Fable 5's 8.0 is a single call from the authors' files, not run in this harness; the host was OpenRouter, not the authors' vLLM in FP8.
The loop lost points in two known gaps (a manager's wrong counterexample, and a passing program overwritten by a later round); fixes are described but not in the numbers.

Why it matters

The GVS5H paper claims that a team of small open models, working through files and a script-based check, can match a much larger model on hard competitive programming problems. This repository is an outside attempt to reproduce that claim with every run published. The reproduction matched the authors' harness (9.8 against 9.6 of 10) and landed above Fable 5 alone (8.0). It also reports what it took: using the authors' exact problem text rather than a rewritten task file made a visible difference to the 27B model's planning on arc196_c.

Who it affects

People building multi-agent coding harnesses, and anyone checking claims that a 27B model with a good loop can match a larger one. Memdoor users get a ready workflow to try on their own problems; the README says it runs on any model the user pins, including local ones through Ollama, llama.cpp, vLLM or LM Studio.

How to use it

Install Memdoor with the curl script from memdoor.ai, then run memdoor connect with a provider (openrouter, anthropic, openai, gemini or chatgpt), or memdoor connect local for no key. Clone the repository and copy examples/workflows/team-solve into ~/.memdoor/workflows/, or into a project's .memdoor/workflows/ (the project's copy wins). Copy the sample folder, which holds task.md and tests/N.in with tests/N.out, into a new directory and run memdoor workflow run team-solve --wait. It exits 0 when every sample passes. For your own problem, replace task.md and add paired test files. The program is solution.py, read from stdin and compared to the samples with diff; it needs Python 3 and bash. The steps name no model, so whatever /model is pinned to answers. For scale, the README puts the rebuild's median run at 40 minutes and $0.39 on OpenRouter.

How solid is it

Better documented than most. All 50 runs are published with requests, replies, programs, grader output, per-call cost and host, and a parity test shows the authors' code and the workflow send identical requests in four cases. But the sample is ten problems with five passes each, not the authors' 100, and the README gives no significance or variance for 9.8 against 9.6. Fable 5's 8.0 is a single call per problem from the authors' files, so the comparison is a team against a single call, not like for like. The authors' regraded files supply the 9.6 and the 8.0.

Risks and caveats

The README says plainly that the contests date from 29 March to 6 April 2025 and the model may have seen them in training, so contamination cannot be ruled out. The host differs from the authors' FP8 vLLM: the rebuild used OpenRouter, where Modal does not publish its precision and served 203 calls before slowing to 14 tokens/s. The loop has known failure modes, an invented counterexample from the manager and a passing program overwritten by a later round, and neither proposed fix is in the reported numbers. Runs can be long and costly: the longest took 4 hours and the two priciest cost $3.44 and $4.24. The loop sees only the public samples, so a wrong program that passes them can still fail hidden tests.

“fresh context, read the files, do not rely on chat”

— Text that every agent prompt in the Memdoor team-solve workflow says