Glonce

big-pickle stealth model scores 50.8% on SWE Atlas, trailing only Claude

modelsThe Glonce editorsHacker News🔥53Glonce rating53Impact60Novelty65Relevance85Surprise55Players50Credibility70
big-pickle stealth model scores 50.8% on SWE Atlas, trailing only Claude

A self-reported evaluation run on 2026-08-11 tested big-pickle, a free stealth model served through OpenCode Zen whose maker is officially unconfirmed, on Scale AI's SWE Atlas Codebase QnA benchmark, using the mini-swe-agent agent scaffold. Across all 124 Codebase QnA tasks, big-pickle resolved 50.8% of them (63 out of 124) in a single trial. Measured against the official SWE Atlas QnA leaderboard as it stood on 2026-07-28, that score outscores every other entry within the Mini-SWE-Agent scaffold class, the apples-to-apples comparison group, and also beats the GPT models run on the Codex scaffold. Only two Claude models, running on their own native Claude Code scaffold, score higher on the leaderboard overall.

The run followed Scale's published protocol closely. It used all 124 Codebase QnA tasks from Scale's own scaleapi/SWE-Atlas repository unmodified, including Scale's shipped agent configuration file and its step limit of 250 steps. It ran on Harbor version 0.18.0 with Modal sandboxes, as the SWE-Atlas documentation specifies, using mini-swe-agent pinned to version 2.4.6, the same minimal, bash-only scaffold Scale uses to evaluate non-first-party models on its leaderboard. Scoring used the benchmark's own unmodified, rubric-based verifier, under which a task counts as resolved only if every scored must-have rubric passes, and judging was done with claude-opus-4-5-20251101, the exact judge model Scale specifies, reached through Anthropic's own OpenAI-compatible endpoint.

For the full run, big-pickle itself consumed 674 million input tokens and 4.3 million output tokens at no charge, since the model is free during its current stealth period. The surrounding infrastructure was not free: the write-up estimates about $70 of Modal compute at the reduced sandbox resources actually used, and roughly two to three times that if the sandboxes had matched the benchmark's declared 16 CPU and 16 GB specification, plus about $25 of Anthropic API usage for judging.

The write-up is explicit about where the result is weaker than it looks. It reflects a single trial per task rather than the three trials Scale's official protocol runs and averages; at 124 tasks, that puts the single-trial standard error at about ±4.5 points, close to the leaderboard's own reported ±5-point error bars. Sandbox resources were also cut to 4 CPU and 8 GB of memory, below the 16 CPU and 16 GB the tasks declare, to fit a personal budget; slower hardware can only cost an agent points, through command timeouts or memory kills, not add them, and a scan of all 124 trajectories found neither: no command hit the 900-second timeout ceiling or the memory limit. Two of the 63 resolved tasks also had rubrics the judge model could not parse after 8 retries, 5 of 11 rubrics on one task and 1 rubric on the other; the benchmark's verifier excludes unscored rubrics from the pass calculation by design, but scoring those two as failures instead gives a stricter lower bound of 61 out of 124 tasks, or 49.2%, a figure the write-up says still beats every other entry in the Mini-SWE-Agent scaffold class.

Scale AI did not run or verify this evaluation itself; it is entirely self-reported, though the full per-task verifier logs are published in the repository for independent audit, and the run is reproducible from the included configuration plus the public SWE-Atlas and Harbor repositories. big-pickle's own identity remains officially unconfirmed: leaked provider errors and API response signatures suggest it is currently served by DeepSeek infrastructure, though the write-up treats that as a signal rather than a confirmed fact, and it notes the model behind the alias can change without notice, so the 50.8% score is only a snapshot of whatever answered on 2026-08-11. OpenCode Zen states that prompts sent to big-pickle during its free period may be used to improve the model, and running this benchmark necessarily sent SWE Atlas's already-public, canary-marked task content to that endpoint. The evaluation configuration and results published in the repository are MIT-licensed; the SWE Atlas benchmark itself is Apache-2.0-licensed by Scale AI.

Key facts

  • big-pickle, a free stealth model served through OpenCode Zen whose maker is unconfirmed, resolved 50.8% of tasks (63 of 124) on Scale AI's SWE Atlas Codebase QnA benchmark in a self-reported, single-trial run dated 2026-08-11.
  • Within the Mini-SWE-Agent scaffold class, the run outscored every other entry on the official leaderboard and also beat the GPT models run on the Codex scaffold; only two Claude models running on their own Claude Code scaffold scored higher.
  • The run matched Scale's published protocol closely: the unmodified 124-task set, Harbor version 0.18.0 on Modal sandboxes, mini-swe-agent pinned to 2.4.6, and Scale's own specified judge model, claude-opus-4-5-20251101.
  • The write-up flags real caveats: a single trial instead of Scale's three-trial protocol, which puts the standard error at about ±4.5 points, and sandbox resources cut to 4 CPU and 8 GB from the declared 16 CPU and 16 GB to fit a personal budget.
  • Running big-pickle for the full evaluation cost nothing (674 million input and 4.3 million output tokens, free during its stealth period), against roughly $70 of Modal compute and $25 of Anthropic API costs for judging.

Why it matters

A model nobody has officially named, and that currently costs nothing to call, has posted a result that only two Claude models beat on a serious code-understanding leaderboard, while also beating every other entry that shares its agent scaffold and the GPT models run on the Codex scaffold. If that number holds up under scrutiny, it points to a coding-capable model at or near the frontier being served for free under a placeholder name, with no company having confirmed ownership of the result yet. A strong, cheap and unlabelled model landing near the top of a leaderboard is unusual enough to be worth tracking on its own, independent of whose infrastructure eventually turns out to be behind it.

Who it affects

Anyone evaluating or building AI coding agents has a new data point to weigh, provisional as it is. Developers routing real work through OpenCode Zen while big-pickle is free are, per OpenCode Zen's own stated terms, feeding those prompts back for model improvement. Scale AI is affected as a third party: an outside, unverified run is now circulating next to its official SWE Atlas leaderboard numbers, built from the benchmark's own public, canary-marked task set. And whichever infrastructure actually operates big-pickle, DeepSeek according to the leaked provider errors and API signatures the write-up cites, though that remains unconfirmed, is the real subject of this result once its identity is settled.

How to use it

big-pickle is reachable today at no charge through OpenCode Zen's OpenAI-compatible endpoint; no end date or post-stealth price has been announced. OpenCode Zen's own terms say prompts sent to the model during this free period may be used to improve it. The evaluation itself is also built to be reused: its configuration, run scripts and full per-task verifier logs are published under an MIT licence in the linked repository, and the write-up gives exact steps to reproduce the run, cloning Scale's SWE-Atlas repository and Harbor version 0.18.0, then running the included smoke test before the full 124-task script.

How solid is it

The number comes with real slack attached, and the write-up says so itself. Scale AI did not run or verify it: this is one person's self-reported result, at a single trial per task rather than the three trials the official protocol averages, which puts the single-trial standard error at about ±4.5 points on a sample of 124 tasks, close to the leaderboard's own reported ±5-point error bars. Sandbox resources were also cut to 4 CPU and 8 GB of memory, below the 16 CPU and 16 GB the tasks declare, to fit a personal budget; slower hardware can only cost an agent points, through command timeouts or memory kills, never add them, and a scan of all 124 trajectories found neither. Two of the 63 resolved tasks had rubrics the judge could not parse after 8 retries; scoring those two as failures instead of excluding them drops the total to a stricter 61 out of 124, or 49.2%, which the write-up notes is still ahead of every other entry in the Mini-SWE-Agent scaffold class. Working in the result's favor: the task set, harness version and judge model all match Scale's own specification exactly, and the full verifier logs plus run configuration are public for independent audit.

Risks and caveats

big-pickle's identity is officially unconfirmed. Leaked provider errors and API response signatures point toward DeepSeek infrastructure, but the write-up treats that as a signal rather than a confirmed fact, and the model behind the alias can change without notice, so the 50.8% figure is only a snapshot of whatever answered on 2026-08-11, not a durable rating. Running the benchmark also necessarily sent SWE Atlas's already-public, canary-marked task content to OpenCode Zen's endpoint, where, by OpenCode Zen's own stated terms, free-period prompts may be used to improve the model, a benchmark-integrity question for whatever comes after big-pickle. No end date or price has been announced for the free period, and no company has stepped forward to confirm or claim credit for the result.