Glonce

Needle 2 packs a 45M-parameter tool-calling model into a 14MB binary

modelsThe Glonce editorsHacker News🔥79Glonce rating79Impact72Novelty72Relevance95Surprise58Players42Credibility85

Needle 2 is a 45M-parameter open model, released under Apache 2.0 with weights on Hugging Face, built for tool calling, device use and structured extraction. The whole model is a single 14MB binary that runs a full session in 28MB of RAM, trained end to end against 2-bit Cactus Quants compression (CQ2-bit) and baked into its own inference engine rather than quantized after the fact.

On tool-calling and mobile device-use benchmarks, Needle 2 trades wins with other small models: FunctionGemma 270M, LFM2.5 230M and Apple FM, while running at 5x to 70x smaller size and 2 bits against their f16 precision. Measured speeds: 500 tokens/sec decode on a Raspberry Pi 5, 400 to 1,500 tokens/sec on VR headsets such as Meta Quest 3S and Apple Vision Pro, and 300 to 700 tokens/sec on sub-$200 phones such as the Samsung A-Series. Peak session RAM sits around 28MB, which is enough to run on microcontrollers like the ESP32-S3.

The stated bet behind the project: edge hardware, not PCs, is where most AI will run. The team cites over 21 billion connected IoT devices against roughly 1.5 billion PCs, and estimates that roughly four in five edge devices, from budget phones to Raspberry Pis, microcontrollers, wearables, small robots and connected-home gear, cost under $200 with no GPU or NPU and only a few hundred MB of RAM. Their framing: turning on a light or extracting a field from a document does not need world knowledge or open-ended prose, only mapping a sentence onto a function's typed parameters, which is why 45M parameters can suffice where chat needs billions. A byte-level grammar compiled from the declared function schemas constrains every generated token, so all 45M parameters go toward picking the right function and grounding its arguments rather than syntax. Every response also carries a confidence score; below a set threshold the system can re-ask or escalate to the cloud instead of guessing, and off-topic requests return an empty call.

Architecturally, Needle 2 is a 27-layer, 512-wide network. A "Hadamard MLP" replaces the usual dense projections with a fixed Walsh transform plus learned diagonals to cut channel-mixing parameters; an "engram" moves world knowledge out of the weights into hashed n-gram tables read a few rows per token; and multi-lane residual streams give the network the routing flexibility of a wider model at a fraction of the compute. Attention uses a 256-token sliding window that bounds the KV cache regardless of session length, while the system prompt and tool declarations are pinned as permanent sinks so they can never be evicted from context. The 2-bit weights never fully decompress into RAM; the engine expands codes inside vector registers and runs an int8 arithmetic path end to end, and the compiled grammar lets it skip up to 98% of the vocabulary-projection compute on structural output tokens. The result ships as one dependency-free C++ binary that probes the CPU at startup and self-selects a kernel (SDOT, NEON, AVX2, RISC-V vectors, WASM SIMD or scalar), running unmodified from Cortex-M microcontrollers through x86 to WebAssembly.

On compute efficiency, the team estimates a conventional transformer at Needle's width and depth would cost 164 MFLOPs per token, and one shrunk to Needle's parameter count would still cost 87; Needle itself spends 70 MFLOPs per token while holding a fifth of its parameters as gathered memory that costs no arithmetic at all, for 7x to 85x fewer FLOPs per token than the models it is benchmarked against. Needle 2 was pretrained on a proprietary 115B-token corpus and post-trained on 38B tokens with compact reasoning traces; for scale, baseline LFM2.5-230M was pretrained on 19 trillion tokens, roughly 120x Needle's total training data, and the two still trade wins in evaluation.

Evaluation ran on five public function-calling benchmarks: Google's Mobile Actions, DroidCall, the Seal-Tools in-domain and out-of-domain tests, and BFCL v4 single-turn, scored by strict exact match on function name, call order and every argument value. Needle 2's numbers were measured end to end through the shipped production engine (CQ2-bit weights, tool retrieval on, 256-token sliding window); baselines ran their released checkpoints at full context under vLLM at f16, and Apple FM ran on-device. The authors flag two deliberate asymmetries: keeping baselines at f16 favors the baselines, since post-hoc 2-bit quantization would collapse models never trained for it, while testing Needle only on the narrow scope it was trained for favors Needle, since every baseline is a general model also carrying chat and world knowledge. On Python simple calls, Needle 2 lands within a point of FunctionGemma, a model six times larger built specifically for this task, and holds a 93.4 well-formed rate across all 3,641 evaluation rows. The gap concentrates in categories outside Needle's training distribution: Java, JavaScript SDK calls and parallel multi-call categories that sit inside BFCL's general-purpose and enterprise API surface.

The model is already in production. Wearable maker Pebble, described in the writeup as "the pioneer of the modern wearable industry," runs Needle 2 locally inside its Index 01 app to turn spoken requests into actions on the screenless Index Ring without depending on a network connection. Pebble's own account: it runs Needle locally instead of relying on the cloud, and the model's footprint is tiny while its performance never lets the app down. A public Playground lets developers test Needle across wearable, robot, smart-home, phone and automotive scenarios, and a companion repo and Python package let teams fine-tune the model on their own Mac or PC in minutes to a few hours to match a specific product's tool vocabulary.

Key facts

  • Needle 2 is a 45M-parameter open model (Apache 2.0, weights on Hugging Face) shipped as a single 14MB binary that runs a full session in 28MB of RAM.
  • It trades wins on tool-calling benchmarks with FunctionGemma 270M, LFM2.5 230M and Apple FM while being 5x to 70x smaller and running at 2 bits against their f16.
  • Measured throughput: 500 tokens/sec decode on a Raspberry Pi 5, 400 to 1,500 tokens/sec on VR headsets, 300 to 700 tokens/sec on sub-$200 phones.
  • Trained end to end against 2-bit Cactus Quants rather than quantized after the fact, Needle spends about 70 MFLOPs per token versus 164 for a conventional transformer of its size, 7x to 85x fewer FLOPs per token than the compared models.
  • Wearable maker Pebble already runs Needle 2 locally in its screenless Index 01 app to execute voice commands offline.

Why it matters

The pitch is that edge AI has meant Macs and PCs while the actual edge is cheap: the team cites over 21 billion connected IoT devices against roughly 1.5 billion PCs, and estimates roughly four in five edge devices, phones, Raspberry Pis, microcontrollers, wearables, small robots and connected-home gear, cost under $200 with no GPU or NPU. Their core claim is that tool calling and structured extraction do not need world knowledge or open-ended prose, only mapping a sentence onto a function's typed parameters, which is why a 45M-parameter model can compete with models many times its size on this narrow task. The other novelty is quantization: instead of shrinking a trained model to 2 bits afterward, which the authors say breaks small models, Needle 2 trains against Cactus Quants from the start across weights, activations and the KV cache, so the deployed 2-bit model is the model that was trained.

Who it affects

Builders of agentic wearables, robots, smart-home devices and automotive systems who need on-device tool calling without a GPU or NPU; teams doing structured extraction from documents on constrained hardware; and the open-source ML community, since weights are on Hugging Face under Apache 2.0. Pebble is the concrete example already shipping it in a consumer product.

How to use it

Needle 2 is free and open under Apache 2.0, with weights on Hugging Face and a repo to run it. A public Playground lets developers test it across wearable, robot, smart-home, phone and automotive scenarios before integrating. A companion Python package lets teams fine-tune the model on their own Mac or PC in minutes to a few hours to teach it a specific product's tool vocabulary. The single C++ binary runs unmodified from Cortex-M microcontrollers (ESP32-S3, ESP32-P4 with 32MB of PSRAM, STM32H7, NXP i.MX RT) through x86 to WebAssembly, with no separate install step. No price is stated for the model itself; the only dollar figures in the writeup describe target device cost, not Needle.

How solid is it

Evaluation ran on five public function-calling benchmarks, Mobile Actions, DroidCall, Seal-Tools in- and out-of-domain, and BFCL v4 single-turn, scored by strict exact match on function name, call order and every argument value, and Needle 2's numbers were measured through the exact shipped production engine rather than a relaxed benchmark configuration. The authors disclose two opposing biases in their own comparison instead of hiding them: keeping baselines at full f16 precision favors the baselines, while testing Needle only within its narrow training scope favors Needle. Within that scope, Needle 2 lands within a point of FunctionGemma, a model six times larger built for the same task, and holds a 93.4 well-formed rate across all 3,641 evaluation rows. It is also already running in a shipping consumer product, Pebble's Index 01, not just in benchmarks.

Risks and caveats

Needle 2 was trained on a narrow corpus of consumer device actions (smart home, mobile, wearables, TV, car) plus structured extraction, and explicitly was not trained for general-purpose function calling; BFCL's enterprise API surface and its Java and JavaScript SDK categories fall entirely outside that training distribution, and the accuracy gap concentrates there and in parallel multi-call categories. Beyond the headline well-formed rate and the single Python-calls comparison to FunctionGemma, this retelling does not carry individual scores for the other four benchmarks; the source points to its own results tables for them, which were not captured in the extracted text. The authors themselves flag that keeping baselines at f16 rather than also quantizing them skews the comparison in Needle's favor. And the low-confidence escalation path to the cloud means the "private, instant, and free" description holds for routine requests, not for every request the device receives.

“We run Cactus Needle locally in the app, instead of relying on the cloud. The model's footprint is tiny and the performance never lets us down.”

— Pebble, on running Needle 2 in its Index 01 app