Glonce

Import AI 476: AI visual reasoning trails humans, and new open models trail China's

researchThe Glonce editorsImport AI🔥52Glonce rating52Impact35Novelty40Relevance95Surprise35Players35Credibility55
Import AI 476: AI visual reasoning trails humans, and new open models trail China's

Import AI issue 476 is a newsletter digest of several items. The lead is Humanity's Sixth Sense (HSS), a benchmark built and released by the AI startups Elorian and Scale AI. It is described as measuring how a model interprets and reasons through information a visual scene implies but does not explicitly show. Example tasks include a photo of a bookcase with the question of whether it can fit more books, and a photo of horses pulling a plough with the question of whether the chains bear the plough's drag weight. HSS has 522 questions covering 288 images and 234 short video clips, 17.6 hours of video in total, in four categories: temporal and causal dynamics, physical and spatial logic, social understanding, and abstract and contextual inference.

Humans score an aggregate pass@1 of 93.1 (image 94.1, video 91.9, time 97, space 94.9). The best AI system, GPT 6 Astra, gets 53.6 (image 55.6, video 51.1, time 58.2, space 53.6). Chinese open-weight models do worse still: GLM-5.3-flash scores 26.8 and Kimi K3 scores 25.5. The benchmark's authors report that the two most common single mistakes were missing a detail that decides the answer (21% of failures) and misidentifying an object, person, or role (20%). The newsletter author's reading is that the more interesting point is how well frontier LLMs do at all, since many people have argued that true visual reasoning needs spatially aware 2D and 3D world models, and the results are something of a counter to that argument.

The second item summarises an AI Frontiers article on AI and space. The newsletter expects a space-economy boom in the 2030s as space-based power, computation and manufacturing come online, helped by greater orbital lift from platforms like Starship. The article's ways AI unlocks space: spacecraft can operate autonomously; they can be redesigned for hosting AIs rather than humans (a speck of debris that could puncture the ISS fatally might leave an uncrewed, unpressurized vessel largely functional); and space-based assembly and later manufacturing will let AI build large, complex things. It then asks whether first movers gain compounding advantages: an industrial flywheel of infrastructure begetting more energy and manufacturing capacity at lower cost, strong domain awareness that may let a space-based power deny others access to space, and self-replicating probes expanding into deep space. The newsletter author agrees with the article's framing and suspects space will move from an academic topic to one of practical relevance sooner than we think, with some solar system activity dominated by autonomous machines.

Third is TasteVal, a benchmark from P-Zero Research that tests research taste, defined as the ability to pick interesting problems, design experiments and interpret results. TasteVal measures the experimental component: given a fixed problem, how well a model iteratively designs experiments and draws conclusions. It has 8 tasks representative of frontier AI development, spanning pretraining data curation, pretraining and fine-tuning language models, preference modeling and robustness to adversarial prompts. The human baseline is 24 experts, at least two per task, who have recently worked at organizations including OpenAI, Google DeepMind, NVIDIA and Microsoft Research. The authors report that the experimental research taste of frontier models has doubled every 3.0 months since December 2025 (95% CI 1.7-5.0 months), and that the best model, Opus 5.5, exceeds the human baseline with a compute multiplier of 2.30x (95% CI 1.15-4.37x) at roughly 1/30 of the expert baseliners' average cost per attempt. Growth appeared much slower in prior years, with the slope changing around the end of 2025 and start of 2026. The author, who earlier argued that creativity was a key variable for recursive self-improvement (RSI), now thinks full RSI is more likely to become possible in 2027, where before he thought 2028.

Fourth, the startup Reflection has revealed Beam, a 501 billion parameter MoE LLM with 23b active parameters, promoted on token efficiency. The newsletter notes that its benchmark scores sit behind Chinese models from Zhipu (GLM), Moonshot (Kimi) and DeepSeek. Beam was pre-trained on 6,144 GB300 NVL72 GPUs for four weeks, then an RL run used about 10.5K of the same GPUs for four more weeks. Pre-training used 23.8 trillion tokens from the web and proprietary licensed datasets. Scores against GLM-5.3: DeepSWE 1.1 44.4 vs 61, GPQA Diamond 90.5 vs 91.7, MCP Atlas 78.7 vs 84.2. Reflection says Beam reaches scores comparable to GLM-5.2 on advanced reasoning benchmarks with 3 to 4 times less inference compute, helped by one million coding, agentic and STEM RL environments, which it believes is one of the largest RL runs by any open lab to date. In early October the company wrote that it would release the weights, technical report, model card and developer artifacts later in the month. The author argues the US has fallen off the open-weight frontier, apart from Google's small Gemma models, largely because Meta stopped making good LLaMA models, and that Reflection is a chance to return.

Fifth, Mistral, described as Europe's leading AI developer, has announced Mistral Large 4 (ML4), an open-weight model it plans to release by the end of the month. Its pitch is strength at cyber tasks with fewer refusals than proprietary Anthropic and OpenAI models; Mistral says it pairs top-tier cyber performance with open weights and self-deployment. ML4 was trained on 3,800 NVIDIA Grace Blackwell GPUs in Mistral's European datacenters. Example scores: Artificial Analysis DeepSWE 1.1 at 62 versus 68 for Kimi K3 and 61 for GLM-5.3; Terminal-Bench 4.0 at 28.3 versus 41.9 for GLM-5.3; Finance Agent V2 at 54.7 versus 55.8 for GLM-5.3. In an internal preference test against GLM-5.3, ML4 was preferred in CAD and STEM and performed on par or close in finance and coding. The newsletter's impression is that ML4 will be the strongest Western open-weight model on release but less powerful than Chinese ones, and that competing with China will be very difficult.

Key facts

  • On Humanity's Sixth Sense (522 questions, 288 images, 234 video clips), humans score an aggregate pass@1 of 93.1 while the best AI system, GPT 6 Astra, scores 53.6; GLM-5.3-flash gets 26.8 and Kimi K3 gets 25.5.
  • P-Zero Research's TasteVal reports frontier models' experimental research taste doubling every 3.0 months since December 2025; Opus 5.5 beats the 24-expert human baseline with a 2.30x compute multiplier at roughly 1/30 of the cost per attempt.
  • The newsletter author now thinks full recursive self-improvement is more likely to become possible in 2027, where before he thought 2028.
  • Reflection's Beam (501 billion parameters, 23b active) trails GLM-5.3 on DeepSWE 1.1 (44.4 vs 61), GPQA Diamond (90.5 vs 91.7) and MCP Atlas (78.7 vs 84.2); weights are planned for later in October.
  • Mistral Large 4 (trained on 3,800 NVIDIA Grace Blackwell GPUs) scores 62 on DeepSWE 1.1 against 68 for Kimi K3 and 61 for GLM-5.3, with open weights planned by the end of the month.

Why it matters

The digest puts one gap and one acceleration side by side. On visual reasoning, the best model still sits about 39.5 points of pass@1 below people on a test of what a scene implies but does not show. On research taste, TasteVal reports models passing a human expert baseline and improving quickly, which is the variable the author links to recursive self-improvement; it moved his own estimate from 2028 to 2027. The open-weight items add the competitive picture: Reflection's Beam and Mistral Large 4 are both described as behind Chinese models, and the author frames the US as having fallen off the open-weight frontier.

Who it affects

Researchers building or evaluating multimodal models get a new visual-reasoning test from Elorian and Scale AI. Labs and people tracking automated AI research get TasteVal's doubling-time estimate. Teams that want open-weight models they can run themselves get two new candidates, Beam from Reflection and Mistral Large 4, the latter pitched at organizations that want to run security work under their own policies. Policy and strategy readers are the audience for the AI Frontiers argument on space and concentration of power.

How to use it

Beam's weights, technical report, model card and developer artifacts were promised by Reflection for later in October, and Mistral plans to release the Mistral Large 4 weights by the end of the month; neither is described as released yet. The Humanity's Sixth Sense write-up and paper (Elorian, Scale Labs), the TasteVal write-up (P-Zero Research), the AI Frontiers article, Reflection's Beam blog post and Mistral's announcement are all linked from the newsletter for anyone who wants the full numbers.

How solid is it

This is a newsletter digest, so the benchmark figures and company claims are secondhand summaries of the linked primary sources. The scores quoted for Beam and Mistral Large 4 come from the companies' announcements; the DeepSWE line for Mistral is labelled Artificial Analysis, and the Mistral preference result is an internal test. TasteVal's headline numbers carry wide confidence intervals (1.7 to 5.0 months for the doubling time, 1.15 to 4.37x for the compute multiplier). The space section rests on one article and is phrased in terms of what might or may be possible.

Risks and caveats

Beam's token-efficiency claim compares against GLM-5.2, while the score list compares against GLM-5.3, so the two are not the same yardstick. The Mistral Large 4 verdict that it will be the strongest Western open-weight model is the author's impression ('it seems like'), not a measured result. The RSI date is the author's personal estimate drawn from a benchmark of experimental taste, not a forecast from the benchmark's authors. The space-dominance scenarios are speculative, and the author's own musings about autonomous machines in deep space are opinion.

“The experimental research taste of frontier models has doubled every 3.0 months since December 2025 (95% CI 1.7-5.0 months).”

— P-Zero Research, TasteVal, as quoted in Import AI 476