Glonce

Octopus trains a 4B model as many agents sharing one KV cache

researchThe Glonce editorsarXiv🔥61Glonce rating61Impact45Novelty65Relevance100Surprise35Players10Credibility70

A new arXiv paper introduces Octopus, a single-model multi-agent reinforcement learning (RL) framework. The authors start from a contrast. Multi-agent LLM systems explore in parallel by spawning sub-agents with isolated context windows, while single-agent systems append every search and verification step to one growing trajectory. According to the authors, existing multi-agent frameworks have three costs: they duplicate prefix KV caches across branches, they exchange findings by re-prefilling text summaries, and they flatten or drop cross-branch dependencies during RL.

In Octopus, all threads share one backbone and coordinate over a GPU-resident directed acyclic graph (DAG) of KV states. Five primitives drive it: fork, read_thread, wait_threads, rewind and commit.

Three co-designed mechanisms make it work. The first is Zero-Copy Radix Forking, which spawns branches as O(1) cursor views over shared physical KV pages. The second is Attention-Only KV Mounting, which mounts a sibling thread's terminal KV blocks directly into full-attention layers and bypasses linear-recurrent states (Gated DeltaNet / Mamba-2). The authors say this cuts off-task hijacking by 21x (from 57.9% to 2.7%) and retains 92.2% of full-text mutual information, at zero extra KV allocation. The third is a Packed DAG-GRPO Trainer, which packs all episode branches into one block-sparse attention pass and assigns counterfactual branch credit through leave-one-branch-out marginal rewards.

The model is trained end-to-end with GRPO on Qwen3.5-4B over a sparse BM25 corpus. Octopus reaches 49.6% accuracy on BrowseComp-Plus. That is 17.5 points above single-agent GRPO at 143K context and 12.5 points above text-summary multi-agent RL. On OpenResearcher it reaches 51.2%. The authors say it outperforms dense-retrieval agents up to 250x larger, while raising training throughput 3.3x and cutting peak KV memory 2.8x.

Key facts

  • Octopus is a single-model multi-agent RL framework: all threads share one backbone and coordinate over a GPU-resident DAG of KV states through five primitives (fork, read_thread, wait_threads, rewind, commit).
  • Trained with GRPO on Qwen3.5-4B over a sparse BM25 corpus, it reaches 49.6% on BrowseComp-Plus, 17.5 points above single-agent GRPO at 143K context and 12.5 points above text-summary multi-agent RL.
  • It scores 51.2% on OpenResearcher and, per the authors, outperforms dense-retrieval agents up to 250x larger.
  • Attention-Only KV Mounting cuts off-task hijacking 21x (57.9% to 2.7%) while retaining 92.2% of full-text mutual information at zero extra KV allocation.
  • Training throughput rises 3.3x and peak KV memory falls 2.8x.

Why it matters

Parallel sub-agents are a common way to handle long research tasks, but the authors argue current frameworks pay for it: duplicated prefix KV caches, findings passed as re-prefilled text summaries, and cross-branch dependencies that get flattened or dropped during RL. Octopus tries to remove all three by keeping branches inside one shared KV graph. The reported result is a 4B model reaching 49.6% on BrowseComp-Plus and beating dense-retrieval agents the authors say are up to 250x larger.

Who it affects

Researchers working on multi-agent LLM systems and agentic RL, especially those training search and deep-research agents with GRPO. Teams that care about KV memory and training throughput in long-context agent training are the other natural audience.

How to use it

The text gives no code or model release, so there is nothing to run yet. The design is the takeaway: branch with cheap cursor views over shared KV pages, mount a sibling's terminal KV blocks into full-attention layers only, and train all branches of an episode in one block-sparse attention pass with leave-one-branch-out rewards. The setup reported is Qwen3.5-4B, GRPO and a sparse BM25 corpus.

How solid is it

These are the authors' own results from an arXiv preprint. The headline numbers are 49.6% on BrowseComp-Plus and 51.2% on OpenResearcher. Gains on BrowseComp-Plus are given only as point gaps over single-agent GRPO (17.5 points, at 143K context) and text-summary multi-agent RL (12.5 points); absolute baseline accuracies are not given. It does not say which dense-retrieval agents make up the 250x-larger comparison, and it does not say which baseline the 3.3x throughput and 2.8x memory figures are measured against.

Risks and caveats

The claims are unreplicated and no code release is mentioned. The 250x comparison is unnamed, so its strength is hard to judge. The throughput and memory gains lack a stated baseline. Hardware, training time and cost are not stated. Evidence is on one 4B backbone, and the mounting mechanism is described for models that mix full-attention layers with linear-recurrent ones (Gated DeltaNet / Mamba-2).