Glonce

Jev-style logprob trick extended to vision models with Gemma 4 12B

toolsThe Glonce editorsHacker News🔥60Glonce rating60Impact40Novelty48Relevance95Surprise28Players12Credibility72
Jev-style logprob trick extended to vision models with Gemma 4 12B

A blogger describes getting interested in Jev, a request format for LLMs, after noticing self-hostable projects built around it such as OpenJev and SemIf. Reading about them introduced the author to a technique for reading an LLM's token log probabilities, something the author calls new to them personally even though it is documented elsewhere, such as in OpenAI's logprobs cookbook. The basic idea: write a prompt that states a situation and offers lettered options, for example asking which team should handle a returned order and offering billing, shipping, or returns as choices A, B, and C, then send it as a Chat Completions request with max_completion_tokens set to 1 and top_logprobs set to 20. The API returns not just the single letter the model picked but also its log probabilities for the alternative tokens, so a strict single-token answer becomes a full probability distribution over the possible options. The author notes that forcing a single output token avoids a lengthy answer and is quick to generate, though processing the input still takes time, and that a shared state prefix across multiple questions can be KV-cached if the backend supports it.

Jev's documented request format currently covers only text or JSON state, so the author added a custom 'attachments' field to carry base64-encoded images and tried the same logprob-scoring trick on vision models. The result is a standalone Python script that captures webcam frames, encodes them as base64 JPEGs, and scores each frame against three questions: whether a person is visible, whether the setting is indoors or outdoors, and how bright the scene is, printing a running table of the answers. Running it locally with Gemma 4 12B on an RTX 3090 gets around 1 frame per second with three questions per frame. Running the same script against OpenAI's gpt-6-luna instead gets around 0.2 frames per second, a gap the author attributes, presumably, to not having made any effort to avoid the cost of opening a separate connection through OpenAI's system for each question on each frame.

The script has to speak two different dialects depending on backend: against OpenAI it hits the Responses endpoint with top_logprobs set to 20, while against a llama.cpp-compatible backend it hits the Chat Completions endpoint with top_logprobs set to 1024. It enforces between 2 and 20 criteria per question, converts the returned log probabilities into normalized probabilities through a softmax-style calculation, and raises an error if the API silently omits a non-negligible option's score rather than returning it. The author concludes that specialized computer vision models are surely far more efficient at this kind of task, but says what appeals to them here is the flexibility: a condition can be changed just by describing it in plain text rather than retraining a model.

Key facts

  • The technique forces an LLM to answer with a single token (max_completion_tokens: 1) and reads its top_logprobs to turn a lettered multiple-choice question into a full probability distribution over the options.
  • The author added a custom 'attachments' field, not part of Jev's documented request format, to send base64-encoded images and apply the technique to vision models.
  • A standalone Python script scores webcam frames against three questions each (person visible, indoor or outdoor setting, brightness), printing results as a live table.
  • Local scoring with Gemma 4 12B on an RTX 3090 ran at about 1 frame per second; the same script against OpenAI's gpt-6-luna ran at about 0.2 frames per second, a gap the author attributes to per-question connection overhead on OpenAI's side.
  • The script requires 2 to 20 criteria per question and validates that the API did not silently omit a non-negligible option's log probability before computing normalized scores.

Why it matters

Reading an LLM's token log probabilities instead of just its top answer is a known but under-used trick for turning a chat model into a cheap, flexible classifier. Extending it with an image-carrying 'attachments' field shows the same trick can score visual conditions, letting a developer define what counts as 'a person is visible' or 'the scene is bright' in plain text rather than training a dedicated model.

Who it affects

Developers experimenting with local or self-hosted LLMs (via llama.cpp-compatible backends) or with OpenAI's API who want a lightweight way to extract structured, probabilistic answers from a model, and anyone already using or building around the Jev request format and its related self-hostable projects OpenJev and SemIf.

How to use it

The published script is run directly (uv run webcam.py) against a local llama.cpp-style endpoint with the Gemma 4 12B model by default, or pointed at OpenAI's API with an alternate URL and the gpt-6-luna model, reading the API key from the OPENAI_API_KEY environment variable. Requests differ by backend: OpenAI's Responses endpoint is queried with top_logprobs at 20, while the Chat Completions endpoint used for llama.cpp-style backends is queried with top_logprobs at 1024, and each question must offer between 2 and 20 criteria.

How solid is it

The account comes from the author's own blog post and includes the full, runnable Python source, including the exact request bodies and the throughput figures for both backends, which supports the specifics. The crawled page cuts off partway through the script's remaining setup code, so any material that followed is not covered here.

Risks and caveats

Throughput is modest either way, about 1 frame per second locally and about 0.2 frames per second against OpenAI, and the author flags the OpenAI slowdown as presumably caused by not avoiding a separate connection per question per frame rather than a fundamental limit. The author also states plainly that specialized computer vision models are surely much more efficient at this kind of task, and the 'attachments' extension is the author's own addition, not part of Jev's documented request format.

“Specialized computer vision models surely are much more efficient, but what I like here is the flexibility: change a condition by describing it in plain text.”

— the blog's author