Glonce

Karpathy has Opus 5 render Lord of the Rings in three.js

modelsThe Glonce editorsHacker News🔥72Glonce rating72Impact50Novelty55Relevance85Surprise65Players75Credibility85
Karpathy has Opus 5 render Lord of the Rings in three.js

Karpathy describes an experiment that moves past the familiar "draw an SVG of a pelican on a bicycle" test for LLMs. He fed Opus 5 the first paragraph of The Lord of the Rings, gave it a 1M token budget (about $10), and asked for a three.js render of the passage. Opus 5 worked for about two hours and produced 5500 lines of code that procedurally rendered the story. Karpathy calls the result kind of janky but fun, and notes it is striking that the model has to place and orchestrate polygon assets in (x, y, z) coordinates and write the animation code itself, let alone that it produces anything usable at all. He frames this as a new class of test: no one in their right mind would ever spend the time to write something this custom by hand, but an LLM has all the stamina and patience in the world, so tasks move from "no one would ever do this" to "sure, why not, it's about free." Karpathy says he expects many more examples like it, and is excited about the prospect of hyper custom worlds a person could be dropped into, for instance as a spectator or a character inside the Lord of the Rings scene he generated, describing it as an ephemeral GTA of X on demand. His last point is a limitation rather than a result: the worlds-and-games domain exposes a weakness in current LLMs, because they cannot efficiently and natively perceive video or play games within them. To check its own output, Opus 5 had to slowly and painstakingly take screenshots at different points, and it still made mistakes that left some of the render janky. Karpathy calls this an example of raw multimodal and gameplay capability that is still quite lacking. He links to Simon Willison's earlier writeup on the pelican-on-a-bicycle test for background, and says he uploaded the source so the render is playable in the browser and forkable at karpathy.ai/lotr-movie/.

Key facts

  • Karpathy gave Opus 5 the first paragraph of The Lord of the Rings, a 1M token budget (about $10), and asked for a three.js render of the passage.
  • Opus 5 worked for about two hours and wrote 5500 lines of code that procedurally rendered the story.
  • Karpathy frames it as a successor to the pelican-on-a-bicycle test, checking whether an LLM can place and animate 3D assets on its own rather than draw a static image.
  • He describes it as a step toward hyper custom worlds a person could enter as a spectator or character, calling it an ephemeral GTA of X on demand.
  • Karpathy says the exercise exposes a weakness: Opus 5 cannot natively perceive video or gameplay, so it had to slowly take screenshots to check its own work and still made mistakes.

Why it matters

The pelican-on-a-bicycle test has become a shorthand way to probe an LLM's raw generation ability with a single static image. Karpathy's experiment generalizes that idea into three dimensions and time: instead of one SVG, the model has to place 3D assets in a coordinate space, write code that animates them, and sustain that effort over an entire scene. That a model can be handed a paragraph of prose and a token budget and turn it into 5500 lines of working procedural rendering code, unsupervised, over two hours, is itself the finding: it marks a jump in what counts as a reasonable thing to even attempt asking an LLM to do.

Who it affects

The experiment is aimed at people who evaluate or build with large language models, particularly anyone thinking about generative worldbuilding or game content. Karpathy's own framing points at game and simulation designers: he imagines dropping a player into a custom-generated scene as a spectator or as one of the characters, on demand, rather than through hand-built content.

How to use it

Karpathy uploaded the source code for the render so it is playable directly in the browser and forkable, at karpathy.ai/lotr-movie/. He also points to Simon Willison's earlier writeup on the pelican-on-a-bicycle test for background on the benchmark this experiment extends.

How solid is it

This is a first-person account from Karpathy himself, posted as a single tweet and picked up on Hacker News, not a controlled or independently reproduced benchmark. The only figures given, the token budget, cost, run time and line count, come from Karpathy's own description of the one run; there is no third-party verification of the code or the render's behavior beyond his own account of it being janky in places.

Risks and caveats

Karpathy is explicit that the result is janky and that Opus 5 made mistakes while trying to check its own output. He identifies a specific limitation behind those mistakes: LLMs still cannot efficiently and natively perceive video or play games within them, so auditing a generated 3D scene requires the slow, indirect workaround of taking screenshots at various points. He calls this a case of raw multimodal and gameplay capability that remains quite lacking, so the demo should be read as a glimpse of a direction rather than a finished or reliable capability.

“No one in their right mind would ever spend the time to write something this custom but LLMs have all the stamina and patience in the world.”

— Karpathy