Frozen random CNNs compress Pong-playing RL agents to 3 neurons
Researchers report that deep reinforcement learning agents trained with a frozen, randomly initialized CNN feature extractor, rather than one trained end to end, spontaneously develop extremely sparse fully connected representations, even though nothing in the training objective pushes for sparsity. The first fully connected layer, FC1, maps 3,136 inputs down to 64 units. On deterministic Pong, agents route task-relevant information through as few as 1 to 3 of those 64 neurons; on stochastic Pong the count rises to 5 to 11. Agents with a fully trainable CNN, run under matched conditions, activate 55 to 64 neurons in the same layer, dozens of times more.
Four findings anchor the paper. First, FC1 sparsity scales with task complexity: 1 to 11 active neurons for Pong, 19 to 26 for Breakout, and about 42 for Space Invaders; scaling the network's width shows this tracks the actual structure of the task rather than a fixed share of available capacity. Second, sparsity varies run to run even on the identical game: three Pong training runs with the same setup but different random seeds end up using 5, 7, and 11 active neurons. The 5-neuron run plateaus at a reward of +14, while the 7- and 11-neuron runs reach expert-level performance of +18.4 and +18.7, suggesting the usable dimensionality of the frozen random projection sets a ceiling on how well the agent can do. Third, ablation experiments confirm the active neurons are not incidental: removing them crashes performance, a result that holds across two separate PPO implementations and four games. Fourth, the sparsity pattern forms early: the active set of neurons locks in by 15 to 30 million training steps, while reward does not turn positive until 35 to 105 million steps later, well after the bottleneck is already fixed.
A complementary comparison on Breakout shows frozen and trainable CNNs can reach similar reward levels through structurally different bottlenecks. Frozen-CNN agents rely on 17 to 25 active neurons, with a participation ratio of about 10 to 14, while trainable-CNN agents spread activity across 51 neurons, with a participation ratio of about 3.6, meaning the trainable network compresses its representation into even fewer effective dimensions despite comparable performance.
The authors argue the broader implication is methodological: whenever a task's raw input dimensionality vastly exceeds its true, intrinsic dimensionality, running gradient descent on top of a frozen random projection can reveal the effective rank of the underlying problem, without building any explicit sparsity machinery into the model.
Key facts
- FC1 (3,136 to 64 neurons) uses just 1 to 3 active neurons on deterministic Pong and 5 to 11 on stochastic Pong, versus 55 to 64 active neurons for a fully trainable CNN under matched conditions
- Sparsity scales with game complexity: 1 to 11 active neurons for Pong, 19 to 26 for Breakout, about 42 for Space Invaders
- Three identical Pong runs, differing only by random seed, use 5, 7, and 11 active neurons; the 5-neuron run plateaus at +14 reward while the others reach expert-level +18.4 and +18.7
- Ablating the active neurons crashes performance across two PPO implementations and four games
- The active neuron set locks in by 15 to 30 million training steps, well before reward turns positive 35 to 105 million steps later
Why it matters
The result is a basic-science finding about representation learning in deep reinforcement learning: a minimal, sparse representation can emerge on its own from a frozen random projection paired with a task-driven fully connected layer, with no sparsity regularizer and no architecture designed to encourage it. It suggests the number of neurons an agent chooses to use is a readable signal of a task's effective dimensionality, obtainable without training the vision backbone at all.
Who it affects
Researchers working on representation learning and interpretability in deep reinforcement learning, and anyone relying on frozen or pretrained feature extractors instead of training the vision backbone jointly with the policy, an increasingly common choice for compute efficiency. The finding also speaks to the broader study of the intrinsic dimensionality, or effective rank, of decision-making problems.
How to use it
The practical takeaway is a cheap diagnostic: pair a frozen random CNN with a standard PPO agent, count how many FC1 neurons stay active once training converges, and read that count as a proxy for the task's intrinsic complexity, with no architecture changes required. The seed-to-seed variation the paper reports is a caution against relying on a single run: the lowest-neuron-count seed also underperformed, so multiple seeds are needed before treating the neuron count as a reliable estimate.
How solid is it
The findings are checked across two PPO implementations and four Atari environments (deterministic and stochastic Pong, Breakout, Space Invaders), backed by explicit ablation experiments that remove the active neurons to test necessity, and a training-step sweep that times the sparsity pattern against the reward curve. That is a reasonably multi-angle test, though the scope stays within a handful of classic Atari games rather than a broader or more modern benchmark suite.
Risks and caveats
The text does not state how many seeds were run beyond the three Pong examples discussed, nor any formal statistical test of significance. The phenomenon is documented specifically for frozen random CNN feature extractors combined with PPO on Atari; whether it generalizes to other algorithms, other environments, or continuous-control tasks is untested here. No author names or affiliations are given in the available material, so the research group's track record cannot be assessed from this record alone.
“We report a striking phenomenon: deep reinforcement learning agents trained with frozen, randomly initialized CNN feature extractors spontaneously develop extremely sparse fully-connected representations, without any sparsity-inducing objective.”
— the paper's abstract