Glonce

GPT-5.6 Sol jumps to 46.2 mAP on Roboflow's vision benchmark, up from 13.8

modelsThe Glonce editorsHacker News🔥79Glonce rating79Impact78Novelty62Relevance98Surprise48Players85Credibility70
GPT-5.6 Sol jumps to 46.2 mAP on Roboflow's vision benchmark, up from 13.8

OpenAI released the GPT-5.6 lineup last week, comprising three models named Sol, Terra and Luna, with the launch stream focused on computer use and UI agents. Roboflow, a computer-vision company, ran the three models through its upcoming VLM benchmark, which tests object detection, object counting, OCR and targeted data extraction, and has not yet been publicly released. The company found Sol to be the best vision model OpenAI has released to date, with Terra and Luna weaker than Sol but both improved over GPT-5.5.

The clearest jump was in object detection: GPT-5.5 scored 13.8 mAP@50 in Roboflow's benchmark, while Sol reached 46.2, with Terra and Luna close behind at 44.7 and 43.3. Sol handled document layout detection well (titles, paragraphs, tables, images, signatures) and coped with dense scenes of many similar, closely packed objects, though it occasionally returned bounding boxes with little or no overlap with the actual objects, arranging them instead in unnaturally regular rows or grids. Roboflow reported these failures to OpenAI, whose team confirmed that Sol grows less stable on images around 2,000 by 2,000 pixels or larger, particularly at lower reasoning effort; higher reasoning effort helps stability but raises token use, latency and cost, and Roboflow suggests resizing or cropping large images before sending them to the API as a workaround. Roboflow also noted that prompting GPT-5.6 for absolute XYXY pixel coordinates works best; using the YXYX-normalized-to-1000 format that suits Gemini 3.5 Flash instead cost GPT-5.6 around 15 mAP points.

Object counting improved across the lineup: Sol scored 73.0% versus 64.9% for GPT-5.5, with Terra at 67.6% and Luna, the cheapest model, at 66.2%, still ahead of the old GPT-5.5 baseline. Sol correctly counted heavily overlapping metal brackets and bullet holes limited to specific scoring zones, but struggled on blister packs (counting empty slots versus sealed pills) and miscounted the abnormal candy example, though Roboflow could not tell whether Sol miscounted or misidentified the category.

OCR stayed roughly flat: Sol scored a 90.7% mean similarity, just 0.5 points behind GPT-5.5's 91.2%, with Terra and Luna at 88.8% and 88.4%. Targeted text extraction slipped more: Sol scored 82.5% against GPT-5.5's 87.6%, with Luna at 81.4% and Terra at 79.4%. Sol read handwritten notes well, extracted a tire size printed along a curved, dirty tire surface, and pulled a live score off a hockey broadcast in the requested format, but failed to read a small, vertical, low-contrast expiration date on a blister pack affected by reflections.

The vision gains come at a cost. Sol averaged close to 10 seconds per image in the benchmark, versus around 6 seconds for Terra and just over 5 for Luna. Per-image cost was roughly 2.5 cents for Sol (the second most expensive model in Roboflow's tests after Claude Fable 5), about 1 cent for Terra, and under 0.5 cents for Luna. Gemini 3.5 Flash, at 0.8 cents per image, remained cheaper than Sol while still leading Roboflow's detection and counting results outright, making it, in Roboflow's assessment, the stronger practical choice for high-volume detection and counting work despite GPT-5.6's progress.

Key facts

  • GPT-5.6 Sol scored 46.2 mAP@50 on object detection in Roboflow's benchmark, up from 13.8 for GPT-5.5; Terra and Luna scored 44.7 and 43.3.
  • Object counting rose to 73.0% for Sol (from 64.9% for GPT-5.5), with Terra at 67.6% and Luna at 66.2%.
  • OpenAI confirmed to Roboflow that Sol becomes less stable on images around 2,000x2,000 pixels or larger, especially at lower reasoning effort.
  • Sol costs roughly 2.5 cents per image versus 0.8 cents for Gemini 3.5 Flash, which still leads on raw detection and counting scores.
  • Using the wrong bounding-box coordinate format (YXYX normalized to 0-1000 instead of absolute XYXY pixels) cut GPT-5.6 detection performance by about 15 mAP points.

Why it matters

Vision has been OpenAI's weak spot relative to rivals, and GPT-5.5's 13.8 mAP@50 on Roboflow's detection benchmark was described as a major weakness. Sol's jump to 46.2 shows a real capability shift rather than a marginal update, and it arrives alongside a launch that leaned on computer use and UI agents, both of which depend on models correctly seeing and locating what is on screen.

Who it affects

Developers building agents, document-processing pipelines, or any visual-reasoning application on OpenAI's API gain a materially stronger detection and counting model in Sol, with Terra and Luna as cheaper, faster, slightly weaker alternatives in the same family. Teams already using Gemini 3.5 Flash for high-volume detection or counting work have less reason to switch, since it remains both cheaper and higher-scoring on those specific tasks in Roboflow's tests.

How to use it

Roboflow recommends prompting GPT-5.6 models for absolute XYXY coordinates in image pixels rather than the normalized YXYX format that suits Gemini 3.5 Flash, since using the wrong format cost about 15 mAP points. For large images, above roughly 2,000 by 2,000 pixels, resizing or cropping before sending them to the API is suggested to avoid the instability OpenAI confirmed at that scale, especially at lower reasoning effort; raising reasoning effort also helps but adds token use, latency and cost. All three models can be tested against each other and against Claude Fable 5 and Gemini 3.5 Flash in Roboflow's Playground.

How solid is it

The results come from Roboflow's own upcoming VLM benchmark, not yet publicly released, covering detection, counting, OCR and data extraction; the company did not disclose dataset size or scoring methodology beyond the task categories. Roboflow is a vision-tooling vendor with a direct commercial interest in this space, and it disclosed the large-image instability finding to OpenAI, which confirmed it, lending that specific claim independent corroboration. No official OpenAI pricing for GPT-5.6 is cited, only Roboflow's own measured per-image costs within its benchmark.

Risks and caveats

Sol still misplaced bounding boxes into unnatural, regularly spaced layouts in some cases, particularly on large images, and failed on some visually simple tasks, including reading a small, low-contrast, reflection-affected expiration date and correctly counting an unusual candy arrangement. Detection and counting gains come with roughly triple the cost of Gemini 3.5 Flash and close to double the latency of Luna, whose own speed is close to Gemini 3.5 Flash's; Gemini 3.5 Flash still outperforms Sol on both benchmarks in absolute terms, so the choice between them depends on whether an application values accuracy or cost and speed more.

“Sol is clearly the best vision model OpenAI has released so far.”

— Roboflow