Glonce

Agent Memory Distillation lifts small LLM agents by up to 27.2%p on tool use

agentsThe Glonce editorsHugging Face🔥73Glonce rating73Impact65Novelty70Relevance98Surprise62Players35Credibility82
Agent Memory Distillation lifts small LLM agents by up to 27.2%p on tool use

Researchers propose Agent Memory Distillation (AMD), a training-free framework for improving small language model agents by transferring structured knowledge from a large teacher agent into a small student agent through hierarchical memory. The motivation is that small models struggle to generate enough successful task trajectories on their own to learn from, while memory systems that could help them have mostly gone untested at small scale. AMD builds three complementary memory types out of the teacher's successful trajectories: workflow memory, which encodes task-level strategies; subtask memory, which supplies concrete behavioral examples at an intermediate level of granularity; and function memory, which captures per-function calling conventions and common pitfalls. Workflow and subtask memories are injected into the student proactively at the start of each task, while function memory is retrieved reactively only when a tool-calling error occurs. The authors evaluate AMD on three tool-use benchmarks, AppWorld, BFCL V3 and ToolSandbox, using four student models in the 4B-8B parameter range with GPT-5-mini serving as the teacher. AMD delivers average accuracy gains of 27.2 percentage points on AppWorld, 11.2 percentage points on BFCL V3 and 3.4 percentage points on ToolSandbox, and it consistently outperforms existing memory-based baselines. Further analysis in the paper finds that subtask memory contributes the largest share of the gains, that teacher effectiveness depends on both the teacher's own capability and its compatibility with the student, and that the 4B-sized students benefit the most from AMD among the four sizes tested.

Key facts

  • AMD is training-free: it transfers a large teacher agent's knowledge to a small student agent through structured memory rather than fine-tuning.
  • It builds three memory types from the teacher's successful trajectories: workflow (task-level strategy), subtask (concrete behavioral examples) and function (per-function calling conventions and pitfalls).
  • Workflow and subtask memory are injected proactively at task start; function memory is retrieved reactively only after a tool-calling error.
  • Tested on four 4B-8B parameter student models with GPT-5-mini as teacher, AMD gains 27.2 percentage points on AppWorld, 11.2 on BFCL V3 and 3.4 on ToolSandbox over baselines.
  • Subtask memory drives the largest share of the improvement, and 4B-sized students benefit the most among the sizes tested.

Why it matters

Small language models are cheaper and faster to run as agents, but they rarely generate enough successful task trajectories on their own to learn effective tool-use behavior. AMD sidesteps that by borrowing a large teacher agent's successful trajectories and turning them into structured memory a small student can consult, without any additional training pass. That makes small agents more capable of using tools reliably at a fraction of the cost of running or fine-tuning a larger model.

Who it affects

The work targets builders of small LLM agents (4B-8B parameters in this study) that need to call external tools and functions, and anyone weighing a large capable model against a small cheap one for agentic tasks. GPT-5-mini plays the teacher role in the experiments; the student models are not individually named beyond their 4B-8B parameter range.

How to use it

AMD is applied at inference time: workflow and subtask memories, distilled from a teacher's successful runs, are injected into the small agent's context before a task starts, giving it task-level strategy and concrete behavioral examples to follow. Function memory, covering calling conventions and common pitfalls for specific tools, is pulled in reactively only when the agent hits a tool-calling error, rather than being loaded upfront. No training or fine-tuning step is required to apply it.

How solid is it

The method is evaluated on three separate tool-use benchmarks, AppWorld, BFCL V3 and ToolSandbox, across four different student models, and is reported to consistently outperform existing memory-based baselines in each case, with the largest gain (27.2 percentage points) on AppWorld and the smallest (3.4 percentage points) on ToolSandbox. Further analysis in the paper isolates subtask memory as the biggest contributor to the gains and finds the benefit is not uniform: it depends on the specific pairing of teacher and student, and it is largest for 4B-sized students.

Risks and caveats

The abstract does not report absolute accuracy figures, only the average accuracy gains in percentage points over baselines, so the resulting absolute performance level on each benchmark is not stated here. It also does not give the specific names of the four student models beyond their 4B-8B parameter range, nor any timescale or cost figures for building or querying the memory, nor the authors or their institutions.