Glonce

rag-staleness-check flags stale, orphaned and duplicate RAG vectors

toolsThe Glonce editorsHacker News🔥51Glonce rating51Impact45Novelty40Relevance85Surprise25Players15Credibility75
rag-staleness-check flags stale, orphaned and duplicate RAG vectors

A developer using the handle rimironenko posted rag-staleness-check on Show HN, an open-source, read-only command-line tool that audits a single already-indexed pgvector, Qdrant or Chroma vector database and reports four kinds of problems. Stale chunks are indexed pieces whose source document has since changed, detected by comparing a manifest of per-document last_modified timestamps against a per-row last-modified value the engine itself stores. Orphaned chunks are indexed pieces whose source document no longer appears in the manifest at all. Near-duplicate chunks are flagged by cosine similarity at or above a default threshold of 0.98, with an optional exact-hash pass added if a per-chunk content hash is stored. A fourth check, called retrievable-after-delete, takes a list of ids the user believes were deleted and checks whether each vector is still fetchable by id at all, which points to storage-layer persistence, and whether it still surfaces in a top-k search result, which the tool calls a functional leak.

The tool is described as the free, single-engine slice of a separate, private, paid product called RAGproof, referenced as the source of a fuller write-up and a multi-engine, ledger-verified methodology. Compared to that paid audit, the free tool does not include orchestration across multiple engines at once, a GDPR Article-17 reporting pack with signed evidence and a verifiable-erasure report, deeper engine-specific checks such as pgvector dead-tuple and VACUUM detail, Qdrant optimizer-threshold detail or Chroma on-disk HNSW growth, or a precision and recall harness scored against a ground-truth ledger the paid product derives from git history across all three engines simultaneously. The free tool itself reports counts and pairs rather than precision or recall, since there is no ground truth available client-side to score its own findings against.

Installation is via pip install rag-staleness-check for pgvector support only, or with the [qdrant], [chroma] or [all] extras added; it can also be run without installing, through pipx run. It requires Python 3.10 or newer. A run is configured with the target engine, connection details such as a DSN for pgvector or a collection name for Qdrant and Chroma, column or payload-field mappings for the document id, last-modified value and content hash, a JSON manifest describing which documents should currently exist, and optionally a JSON array of ids believed deleted. Output is a scorecard printed to stdout plus a full findings.json file; any check missing a required input, such as no manifest, no per-row metadata configured, or no deleted-ids list, is reported as skipped with a stated reason rather than as a misleadingly clean zero.

The read-only guarantee is enforced two ways, according to the documentation. A static test parses the real syntax tree of every file in the package and fails the build if it finds any write, delete or upsert-shaped method call, or any SQL write statement passed to execute() or executemany(); the docs note this is not a substring search, since that would false-positive on the project's own README and docstrings. At runtime, for pgvector specifically, every connection opens with SET default_transaction_read_only = on, and a helper function re-checks SHOW default_transaction_read_only before each query runs, refusing to proceed if the answer is not "on"; this is meant as defense against a connection pooler such as PgBouncer silently dropping that session-level setting. Qdrant and Chroma give no client-exposed way to check whether a session is read-only, so for those two engines enforcement relies on the static test plus the user's own deployment-side scoping, such as connecting with a read-only-scoped API key.

The deleted-vector check is tied explicitly to EDPB Guidelines 05/2019, which the documentation cites as requiring that erasure be verifiable and irreversible, so that suppressing a record from search results is not enough on its own if the underlying vector is still present; the findings.json output includes a retrievable.framing field that states this distinction on every run. The tool has no default telemetry. A --share-anonymous-scorecard flag is an explicit opt-in that would print an anonymized payload of aggregate counts and engine type only, never DSNs, hostnames, doc_ids or chunk_ids, though no backend is configured yet, so nothing is actually sent over the network regardless of the flag; a DO_NOT_TRACK environment variable forces the flag off even if it is passed. Separately, the tool disables chromadb's own built-in, default-on, PostHog-based client telemetry whenever it connects to Chroma.

Several limitations are documented directly by the author. On chromadb version 0.6.3 specifically, the telemetry-disabling setting takes effect internally, but a startup event still throws a harmless-looking error, "capture() takes 1 positional argument but 3 were given," before any request is made. The duplicate-detection exact-hash pass silently falls back to cosine-only matching if no content hash is configured, a condition flagged through a warning field rather than left silent. A chunk with no retrievable vector is excluded from the cosine-ANN pass rather than counted as not a duplicate. Chroma's snapshot_stats() reports only a live count, with no on-disk index-size measurement, unlike a development build of the paid audit that used a docker exec du command against a known local container, an approach the author says has no equivalent against a real third-party deployment. Neither qdrant-client nor chromadb is pinned to an exact version, since the tool connects to a third party's already-running server whose version is outside its control. The project is released under the Apache-2.0 license, independent of any other RAGproof project's license, and a minhash-based surface-text duplicate-detection mode is listed as not yet implemented.

Key facts

  • rag-staleness-check is an open-source, read-only CLI that audits a single pgvector, Qdrant or Chroma index for stale, orphaned and near-duplicate chunks, and for vectors that remain retrievable after deletion.
  • Duplicate detection uses a default cosine-similarity threshold of 0.98, with an optional exact-hash pass if a per-chunk content hash is stored.
  • It requires Python 3.10 or newer and installs via pip or pipx, with optional [qdrant], [chroma] or [all] extras.
  • The read-only guarantee is enforced by a static syntax-tree test plus, for pgvector, a runtime check of SHOW default_transaction_read_only before every query.
  • It is the free single-engine slice of a separate paid product, RAGproof, which adds multi-engine orchestration, a GDPR Article-17 reporting pack, and a git-history-derived ground-truth ledger for precision and recall scoring.

Why it matters

RAG systems accumulate silent decay: source documents get edited or deleted, but their old embeddings can stay indexed and keep surfacing in retrieval, with no built-in signal that anything is wrong. rag-staleness-check lets a team check for that decay directly against its own running index rather than assuming an ingestion pipeline stayed in sync with source documents. The retrievable-after-delete check targets a specific gap: a system can suppress a record from search results while the underlying vector, and the information it encodes, is still physically stored and fetchable by id.

Who it affects

Anyone running pgvector, Qdrant or Chroma as the vector store behind a production RAG pipeline, especially teams handling documents subject to deletion requests; the tool's documentation cites GDPR erasure guidance directly. It is built for self-serve, single-engine use. The documentation states that multi-engine orchestration and a GDPR Article-17 signed-evidence reporting pack are reserved for the separate, paid RAGproof audit rather than included here.

How to use it

Install with pip install rag-staleness-check for pgvector-only support, or add the [qdrant], [chroma] or [all] extras; it can also run without installing, via pipx run rag-staleness-check. Python 3.10 or newer is required. A run is configured with the engine, connection details (a DSN for pgvector, a collection name for Qdrant or Chroma), column or payload-field mappings for document id, last-modified value and content hash, a JSON manifest of documents that should currently exist, and optionally a JSON array of ids believed deleted. Output is a scorecard on stdout plus a full findings.json file; a check missing required input is marked skipped with a stated reason instead of silently reporting a clean 0%. No price is stated anywhere in the source, either for this tool, which is free, or for the paid RAGproof audit.

How solid is it

The read-only claim rests on two documented mechanisms: a static test that parses the real syntax tree of every source file and fails the build on any write, delete or upsert-shaped call, or any SQL write statement passed to execute() or executemany(); and, for pgvector specifically, a runtime check that re-verifies SHOW default_transaction_read_only before every query, guarding against a connection pooler silently dropping the session-level read-only setting. Qdrant and Chroma have no equivalent client-exposed read-only check, so enforcement there depends on the static test plus the user's own read-only-scoped credentials. The tool reports counts and pairs rather than precision or recall, since there is no client-side ground truth to measure accuracy against; that scoring is reserved for the paid, git-history-derived ledger in the private RAGproof audit. The source gives no adoption numbers, benchmarks or performance figures for the tool itself.

Risks and caveats

Several gaps are documented by the author directly. On chromadb 0.6.3, a telemetry-disabling setting takes effect internally but a startup event still throws a harmless-looking error before any request is made. The duplicate-detection exact-hash pass silently falls back to cosine-only matching if no content hash is configured, flagged with a warning field rather than left silent. A chunk with no retrievable vector is excluded from the cosine-ANN pass rather than counted as not a duplicate, which can understate the duplicate count. Chroma's snapshot_stats() reports only a live count, with no on-disk index-size measurement. Neither qdrant-client nor chromadb is pinned to an exact version, since the tool connects to a third party's already-running server whose version it does not control. A minhash-based surface-text duplicate-detection mode is listed as not yet implemented.

“erasure has to be verifiable and irreversible - suppressing a record from search results isn't enough on its own if the underlying vector is still there”

— EDPB Guidelines 05/2019, as cited in the tool's documentation