Glonce

Reflection releases Beam, a 501B open MoE model with 1M context

modelsThe Glonce editorsHugging Face🔥74Glonce rating74Impact70Novelty70Relevance100Surprise45Players60Credibility82
Reflection releases Beam, a 501B open MoE model with 1M context

Hugging Face, working with the Reflection team, has published a launch post for Beam, an open-weight model on the Hub as reflection/Beam-1.0-501B-A23B. The post describes it as focused on coding and agentic use cases, and aimed at teams that want a frontier-scale model on their own hardware, infrastructure or cloud and want to customize it. It ships with its own open source Rust-based harness, which the post says delivers the best performance from the model. It is released under Apache 2.0.

The headline specs: 501B parameters in total with 23B active per token, text in and text out, and a context window of 1M tokens. Checkpoints come in BF16, FP8 and NVFP4. The model supports five reasoning values ("low", "medium" as the default, "high", "xhigh" and "max"), reasoning that can stay in context across turns (preserve_thinking), and tool calling with a dedicated grammar. A speculative decoding drafter built with DFlash2 is included, and full-parameter RL is supported with Miles. Hugging Face says it worked with Reflection on inference, SFT and RL recipes.

Architecture, as the post lays it out. The decoder has 52 layers with a hidden size of 5,120. Most layers use sliding-window attention over the previous 512 tokens; every sixth layer and the last one attend to the full context, giving 43 sliding-window layers and 9 global ones. Sliding-window layers use RoPE; global layers use no positional encoding, and instead scale each query by a factor that grows slowly with the number of visible tokens, which keeps the softmax from flattening as context grows. Attention is grouped-query with 48 query heads and 8 key-value heads, RMSNorm on queries and keys, and a learned sigmoid gate on every head channel.

The first 3 layers are dense SwiGLU MLPs; the other 49 are Mixture-of-Experts blocks with 256 experts each, each expert a SwiGLU MLP with intermediate size 2,560 (about 39M parameters). Experts hold 98.4% of all parameters. The router follows DeepSeek-V3's group-limited scheme: it scores the 256 experts with a sigmoid, splits them into 8 groups of 32, keeps the 4 best groups, then picks the top 8 experts. Unlike DeepSeek-V3 it has no shared expert, and expert outputs are combined with normalized router weights and scaled by 2.5. Only 8 experts run per token, so deployment needs the memory of a 501B model but runs at the speed of a 23B one.

The residual connections are called the most distinctive part. Instead of adding each sublayer output with weight 1, Beam counts the input embedding as step 1 and at step k adds the sublayer output with weight 1/sqrt(k) while keeping the existing stream with weight sqrt((k-1)/k). The stream therefore stays at constant scale, and after all 104 additions every sublayer output and the input embedding carry the same weight in the final hidden state. Sublayers are normalized before and after, and the final logits pass through a tanh softcap at ±30 computed in FP32.

The hybrid attention also keeps the KV cache small. If all 52 layers kept every token, the cache would grow by 208 KiB per token; because only the 9 global layers keep the whole sequence, it grows by 36 KiB per token, roughly a sixth of that (per sequence, BF16, transformers dynamic cache). The post says benchmark results across precision levels are provided by the Reflection team.

Hardware. The BF16 checkpoint is about 1 TB in safetensors. The post calls NVFP4 the most practical way to serve the model: it retains most of the quality of BF16 and shrinks the weights to about 280 GB, but runs natively only on Blackwell GPUs, not on older ones like the H200. FP8 checkpoints suit GPUs older than Blackwell. Two B200s (180 GB HBM3e each, 360 GB) can hold the weights with little room for activations and KV cache, so the post advises four B200s (720 GB). Alternatives it names are a full 8xB200 node, two B300s at 288 GB each, or even 8xRTX Pro 6000. Without Blackwell hardware, it suggests Inference Providers (serverless) to test the model.

Running it. In transformers (version 5.20.1 or later) the simplest path is the text-generation pipeline; AutoModelForCausalLM and AutoTokenizer work too. The tokenizer carries a response template, and parse_response splits raw output into reasoning, answer and tool calls without regex; for tool calls you pass the tool schemas so arguments are cast to their declared types. For multi-GPU use, the config defines an expert-parallel plan: each expert sits on one device, while attention, dense layers, embeddings and LM head (8B parameters, 16 GB in BF16) are replicated everywhere. On 16 H200s each GPU holds 78 GB of weights; 8 GPUs would need 139 GB each, which does not fit a 141 GB H200. The device count must divide 256. For vLLM, one command serves the BF16 checkpoint with tensor parallelism across 16 GPUs (two 8x H200 nodes) behind an OpenAI-compatible API; adding DFlash2 speculative decoding lets a small drafter propose up to 15 tokens ahead for Beam to check in one forward pass. The post also recommends DFlash2 for SGLang, which also has day-0 support.

Contest. Reflection will give out $10,000 in Hub credits to the best projects built with Beam: a Spaces demo, a quantization for particular hardware, a synthetic dataset, an RL environment, a benchmark, or something else. Builders share a space, model or dataset repo on the model's repo page, and Hugging Face collects the links, selects the best and sends credits to Hub accounts.

Key facts

  • Beam (reflection/Beam-1.0-501B-A23B) is a text-only sparse MoE with 501B total and 23B active parameters per token, 52 layers, 256 experts per MoE layer and 8 routed per token.
  • It has a 1M-token context window, Apache 2.0 licensing, BF16, FP8 and NVFP4 checkpoints, and five reasoning-effort levels with "medium" as default.
  • Hybrid attention (43 sliding-window layers over 512 tokens, 9 global) keeps KV cache growth to 36 KiB per token versus 208 KiB if every layer were global.
  • Day-0 support in transformers, vLLM and SGLang; the post recommends NVFP4 (about 280 GB versus about 1 TB in BF16) on four B200s (720 GB), and FP8 for pre-Blackwell GPUs.
  • Reflection is giving $10,000 in Hub credits to the best projects built with Beam.

Why it matters

Beam is an Apache 2.0 open-weight model at 501B parameters with a 1M-token context, pitched at coding and agentic work and at teams that want to run a frontier-scale model on their own hardware or cloud and customize it. It launches with inference, SFT and RL recipes and day-0 support in three major libraries, plus its own Rust-based harness. Architecturally it mixes sliding-window and global attention, a 256-expert router modeled on DeepSeek-V3 without a shared expert, and an unusual residual-stream scheme that keeps the stream at constant scale.

Who it affects

Teams that want to self-host and post-train a large open model for coding and agent workloads, including those with Blackwell GPUs (B200, B300, RTX Pro 6000) or Hopper-class clusters such as 16 H200s. Open-source tinkerers can also enter the Hub contest with demos, quantizations, datasets, RL environments or benchmarks.

How to use it

Install transformers 5.20.1 or later and load reflection/Beam-1.0-501B-A23B with the text-generation pipeline or AutoModelForCausalLM; use tokenizer.parse_response (with tool schemas for tool calls) to split reasoning, answer and tool calls. For serving, the post recommends NVFP4 on Blackwell, planning on four B200s (720 GB), or two B300s, or 8xRTX Pro 6000, and FP8 on older GPUs. vLLM can serve the BF16 checkpoint across 16 GPUs with --tensor-parallel-size 16, and DFlash2 speculative decoding can be added through --speculative-config. Without suitable hardware, the post points to Inference Providers (serverless). The licence is Apache 2.0. To enter the contest, share a space, model or dataset repo on the model's repo page.

How solid is it

This is a launch post published by Hugging Face with the Reflection team, so the specifications and claims come from the release's own side. The post says benchmark results across precision levels are provided by the Reflection team. Claims such as the Rust harness delivering the best performance and NVFP4 retaining most of BF16 quality are the post authors' own statements. The architecture numbers (layer counts, expert counts, cache sizes, memory per GPU) are given in detail and are internally consistent: 43 sliding-window plus 9 global layers make 52, and 3 dense plus 49 MoE layers make 52.

Risks and caveats

Memory is the main constraint: all 256 experts stay resident, so the model needs the memory of a 501B model even though it computes like a 23B one. Two B200s hold the weights with little room left for activations and KV cache. NVFP4 runs natively only on Blackwell, so older GPUs like the H200 need FP8 or BF16, and BF16 needs 16 GPUs in the vLLM example. The DFlash2 drafter is named reflection/dflash2-drafter-preview. The NVFP4 checkpoint keeps some parameters in BF16 for accuracy and, per the post, retains most, not all, of BF16 quality. The post gives no DFlash2 speedup figures.

“deployment needs the memory of a 501B model but runs at the speed of a 23B one”

— Hugging Face blog post on Beam