Glonce

Hugging Face releases AnySpeechRecognizer and AnyDecisionModel for Apple platforms

toolsThe Glonce editorsHugging Face🔥61Glonce rating61Impact50Novelty60Relevance95Surprise30Players40Credibility80
Hugging Face releases AnySpeechRecognizer and AnyDecisionModel for Apple platforms

Hugging Face has announced two Swift packages that extend its AnyLanguageModel approach, familiar Swift APIs with a choice of model, to speech and decisions. The pitch: a language model can do a lot, but for a support call you want speech recognition first and a routing decision second, and there are models built for each that are better, faster and cheaper. Both packages can download open models from the Hugging Face Hub and run them on device.

AnySpeechRecognizer mirrors Apple's SpeechAnalyzer API, introduced with macOS 26 and iOS 26. That API takes modules such as SpeechTranscriber and SpeechDetector, accepts a file or a stream of audio buffers, and returns results through async sequences; AssetInventory installs the models. The authors call it well designed, but note that the only engine behind it is Apple's. The package mirrors SpeechAnalyzer, SpeechTranscriber, DictationTranscriber, SpeechDetector and AssetInventory, with Apple's Speech framework as the default engine, so code that uses the shared API needs only a changed import. To switch engines, you change how the transcriber is created, for example by passing ParakeetSpeechModel(variant: .v3) or QwenSpeechModel. Developers can add their own engine by implementing SpeechRecognitionModel and SpeechRecognitionExecutor.

The engines on offer are: Parakeet v3 (NVIDIA's Parakeet TDT 0.6B v3), which recognizes 25 European languages and works out the language itself; Parakeet Ultra, Moondream's post-trained version of v3, which the authors say was more accurate in most of their measurements; and Parakeet Unified, NVIDIA's English-only Parakeet. All three run on the package's own Core ML runtime, with file and live input, volatile results, word timing and word confidence. Qwen3-ASR, Alibaba's model in 0.6B and 1.7B sizes, recognizes 30 languages and 22 Chinese dialects, including Mandarin, Cantonese, Japanese and Korean, which Parakeet does not cover. It detects the language or accepts one. It returns one final result per segment of up to 40 seconds, with no volatile results, word timing or confidence, runs on the GPU and is slower than Parakeet, so it suits files better than live input.

Hugging Face converted the models to Core ML and published them on the Hub. Each repository records its source checkpoint and a checksum for every file, and the package pins a fixed revision. The Parakeet repositories add an 8-second encoder for live input that cuts live analysis time by about a third, at the cost of a somewhat higher word error rate. Conversion workflows are included, most following FluidInference's mobius scripts, and each conversion is checked against the original model. Downloads happen only when the app asks: about 600 MB per Parakeet, about 590 MB more for the live encoder, 0.9 to 2.4 GB for Qwen3-ASR and about 250 MB for Sortformer. Files go into the standard Hugging Face cache.

Apple's Speech framework has no speaker diarization module, so the package adds SpeakerDiarizer, which runs NVIDIA's Streaming Sortformer on Core ML alongside an Apple or Parakeet transcriber on the same timeline. It tracks up to four speakers, even when they talk over each other, and a SpeakerTimeline attributes each timed word to a speaker. Sortformer v2 is the default; v2.1, trained by NVIDIA with more meeting data, was more accurate on meetings in the authors' measurements.

On performance, the authors measured with the package's hear command-line tool on an M4 Max Mac Studio running macOS 27.0.1. Parakeet transcribes an hour of audio in about 20 to 40 seconds from a file, or about 1.5 to 2.5 minutes live, using about 100 MB of memory once Core ML has compiled the model. Qwen3-ASR is much slower and needs a few gigabytes of memory. On AMI meeting recordings, where people talk over each other, every engine tested had a word error rate above 23%. The authors stress that the numbers come from one machine and a few datasets, and point to hear evaluate and the repository's natural-speech-eval.py to reproduce or rerun them on your own audio. They have tested only on macOS 27, not on an iPhone, and would like to add more engines, including Core AI on macOS 27 and iOS 27.

AnyDecisionModel has no Apple framework to map onto. The authors say decision models are not exactly new; what is new is the name, a lot of hype and a crop of models trained to make decisions, which they think is where things are going. Apps often prompt a language model to decide things (is this urgent, which team should handle this ticket) and parse the generated text, which gives one answer without the probabilities of the alternatives. AnyDecisionModel instead answers yes-or-no, multiple-choice and rating-scale questions with typed results and probabilities, deriving each answer from the model's probabilities rather than generated text. Backends are a model on your device with MLX, Core AI or llama.cpp, or a decision model from Jev or Ollama.

In the example, a support ticket about a double charge and a logout bug becomes the state of a DecisionSession built on MLXDecisionModel with mlx-community/Qwen3-4B-Instruct-2507-4bit. The code asks for a probability (urgent is a Double), a choice from an enum conforming to Choosable (a Choice with the selected case, a probability per option and a confidence value, usable in an exhaustive switch) and a score over levels such as Calm, Frustrated and Very angry (a Score with the expected level, per-level probabilities and a confidence). A session holds a single state, text or JSON; questions are independent, and several can be passed to decide(_:) as a batch. The MLX backend builds a prompt with a one-token label per allowed answer and reads each label's probability from one forward pass.

The demo is Emoji Magnet, a macOS app with a text field above a pile of 343 emoji. Type a phrase such as 'things you wear' and matching emoji rise out of the pile; refine it to 'things you wear in winter' and non-matching ones fall back. There are no built-in keyword rules: when you pause typing, the model gets one yes-or-no question per emoji. With the hosted Jev backend, which answers all 343 questions in one request, the emoji settle in about half a second per phrase. Without a TYPESAFE_AI_API_KEY, the app runs Qwen3 4B locally with the MLX backend, which uses prefix caching to evaluate the state once and then answers 16 questions per forward pass. On an M4 Max a group of 16 takes about 0.45 seconds after the model loads, and a full pass over all 343 emoji takes about 10 seconds, with each group's results shown as they arrive. The app is in the AnyDecisionModel-Examples repository.

The package also supports models with a trained decision head. Cloudflare's Clef models turn a state and a schema of typed questions into decisions, returning probabilities for all allowed options across the questions in one forward pass. Clef-Flash is the 9B variant, and MLXHeadDecisionModel (with modelID mlx-community/clef-flash-4bit and head: .jointSchema) runs it on a Mac with MLX, using the same sessions and questions. The 4-bit model is a 6.2 GB download, and its model card recommends a Mac with at least 16 GB of memory. Clef can read images and video, but this backend accepts only text and JSON states.

The authors also warn that raw probabilities from language models are often overconfident. The local backends accept a Calibration, which applies a temperature to every answer and a bias to yes-or-no answers, and separate datasets should be used to fit the calibration and to evaluate the results. Language models can also favor an option because of its position in a list.

Key facts

  • AnySpeechRecognizer mirrors Apple's SpeechAnalyzer API (macOS 26 / iOS 26) with a choice of engine: Apple's default, NVIDIA Parakeet v3, Moondream's Parakeet Ultra, Parakeet Unified, or Alibaba's Qwen3-ASR, all on Core ML; for code on the shared API, changing the import is enough.
  • It adds speaker diarization via NVIDIA's Streaming Sortformer, tracking up to four speakers, which Apple's Speech framework does not offer.
  • On an M4 Max Mac Studio, Parakeet transcribes an hour of audio in about 20 to 40 seconds from a file; every engine tested had a word error rate above 23% on AMI meeting recordings.
  • AnyDecisionModel answers yes/no, multiple-choice and rating questions from model probabilities, using MLX, Core AI, llama.cpp, Jev or Ollama, and also runs Cloudflare's 9B Clef-Flash with a trained decision head.
  • The Emoji Magnet demo asks 343 yes-or-no questions per phrase: about half a second with hosted Jev, about 10 seconds locally with Qwen3 4B on MLX.

Why it matters

Apple's SpeechAnalyzer is a well-designed API, but the only engine behind it is Apple's. AnySpeechRecognizer keeps the API and lets developers swap in open models such as Parakeet and Qwen3-ASR; Qwen3-ASR covers languages Parakeet does not, including Mandarin, Cantonese, Japanese and Korean. It also adds diarization that Apple's framework lacks. AnyDecisionModel gives Swift apps typed yes/no, choice and score answers with probabilities instead of parsing generated text, and is built ahead of what the authors see as a trend toward models trained to make decisions.

Who it affects

Swift developers on Apple platforms who transcribe audio or need classification-style decisions in an app: support tools, meeting transcription, ticket routing, sentiment scoring. Teams already using SpeechAnalyzer, Foundation Models or AnyLanguageModel get a familiar pattern, since the model-and-session design is similar.

How to use it

For speech, replace import Speech with import AnySpeechRecognizer; Apple's engine stays the default. To use Parakeet or Qwen3-ASR, change only how the SpeechTranscriber is created, for example with ParakeetSpeechModel(variant: .v3) or QwenSpeechModel(variant: .large, language: ...). Add a SpeakerDiarizer with SortformerDiarizationModel() to the analyzer's modules for speaker turns. Models download from the Hub only when your app asks, from about 250 MB (Sortformer) up to 2.4 GB (Qwen3-ASR). For decisions, build a DecisionSession from a model such as MLXDecisionModel and call probability(of:), choice(:from:) or score(:levels:), or pass a batch to decide(_:). Try the Emoji Magnet demo from the AnyDecisionModel-Examples repository; the README for AnySpeechRecognizer has a complete example. Use hear evaluate on your own audio.

How solid is it

This is a first-party announcement by Hugging Face, and the Core ML conversions are published on the Hub with source checkpoints, per-file checksums and pinned revisions, and each is checked against the original model. The performance figures come from the authors' own tests on one M4 Max Mac Studio running macOS 27.0.1 and a few datasets, and they say to measure on your own hardware and audio. The claim that Parakeet Ultra was more accurate in most measurements is the authors' own.

Risks and caveats

Testing so far covered only macOS 27, not an iPhone. Meetings are hard: every engine tested had a word error rate above 23% on AMI recordings. Qwen3-ASR has no volatile results, word timing or confidence, is slower and needs a few gigabytes of memory, so it fits files better than live use. The 8-second live encoder is about a third faster but has a somewhat higher word error rate. Clef-Flash is a 6.2 GB download with a recommended 16 GB of memory, and its backend takes only text and JSON, not images or video. Raw language model probabilities are often overconfident and need calibration fitted on a separate dataset from the one used for evaluation, and models can favor an option because of its position in a list.

“It's a well-designed API, but the only engine behind it is Apple's.”

— Hugging Face blog post, on Apple's SpeechAnalyzer