Glonce

Activity Frames turns screen activity into agent memory, 86x smaller

researchThe Glonce editorsHugging Face🔥80Glonce rating80Impact68Novelty62Relevance100Surprise40Players72Credibility82
Activity Frames turns screen activity into agent memory, 86x smaller

Computer-use agents today burn full frontier-model inference tokens re-deriving routines a user has already performed, because an agent's memory records what the user said, not what the user did. To close that gap, the authors built Activity Frames, a deterministic, zero-model pipeline that compiles passively captured screen activity into structured agent memory. It segments a local capture stream into typed activity frames, bounded episodes carrying the application, site, timing, input volume, and evidence pointers back to the raw rows. With no model in the loop, the output is byte-identical, cacheable, and mechanically auditable.

The authors evaluated the pipeline on one professional's single-user corpus of 128,756 frames spanning 51 active days. The compiler reduces a day of raw capture to a prompt-ready context block 86x smaller in 68 ms. An agent reading that compiled block answers questions about the day at 98.4% accuracy (Wilson 95% CI 91.7-99.7%) against an independent oracle, versus 66-80% for an LLM summary of the same capture; a mid-tier model reading the block matches the performance of a frontier model on the same task.

The compiler also doubles as a demand-side cost instrument. Reading passive, pre-delegation human activity rather than agent rollouts, it supplies two parameters the authors say agent-cost models assume but had not previously measured: the Routine Overhead Ratio R and the routine recurrence h. The authors report first values of R, described as a modeled upper bound, at 60-343x, and a delegable recurrence of 9.0% in-sample and 7.7% out-of-sample, implying a realistic all-fleet token ceiling near 8%. A compiled routine can replay deterministically with the model removed from the loop, which the authors demonstrated live at zero model tokens on a guard-matched hit. The schema, compiler, and evaluation harness are released as open.

Key facts

  • A deterministic, zero-model pipeline called Activity Frames compiles passively captured screen activity into typed 'activity frames' for agent memory, with no model in the loop.
  • Evaluated on one professional's single-user corpus of 128,756 frames over 51 active days.
  • Compresses a day's raw capture into a prompt-ready context block 86x smaller in 68 ms.
  • An agent reading the compiled block scores 98.4% accuracy (Wilson 95% CI 91.7-99.7%) versus an independent oracle, against 66-80% for an LLM summary of the same capture; a mid-tier model matches a frontier model reading the same block.
  • First reported values for two agent-cost parameters: Routine Overhead Ratio R at 60-343x (a modeled upper bound) and delegable recurrence at 9.0% in-sample / 7.7% out-of-sample, implying an all-fleet token ceiling near 8%.

Why it matters

Computer-use agents currently burn full frontier-model inference re-deriving routines a user has already performed, because agent memory today records what the user said rather than what the user did. Activity Frames closes that gap with a deterministic pipeline, no model in the loop, that turns raw screen capture into a compact, cacheable, mechanically auditable record an agent can read directly. It also gives agent-cost modeling something the authors say has been missing until now: measured values for the Routine Overhead Ratio and delegable recurrence, rather than assumed placeholders.

Who it affects

The work targets builders of computer-use and desktop agents that need a persistent memory of a user's routines, and anyone modeling the token economics of agent fleets, since the compiler supplies the R and h parameters those cost models previously had to assume rather than measure.

How to use it

The authors say the schema, compiler, and evaluation harness are released as open, so teams can inspect the activity-frame format and run the pipeline against their own capture streams. No pricing or licensing terms are given.

How solid is it

The headline figures come from a single professional's capture corpus, 128,756 frames across 51 active days, evaluated against an independent oracle, with a reported Wilson 95% confidence interval of 91.7-99.7% around the 98.4% accuracy figure. The authors explicitly describe the 60-343x Routine Overhead Ratio as a modeled upper bound rather than a direct measurement.

Risks and caveats

The evaluation corpus is a single user; the abstract does not name the professional's occupation, employer, or industry, nor the specific mid-tier or frontier models compared, nor which applications, sites, or platforms the capture covers, so it is unclear how the 86x compression and 98.4% accuracy generalize to other work patterns or capture setups. Continuous screen capture also raises its own privacy considerations for whoever the recording covers.