AISPA audit finds system prompt gaps across 88 AI products
Researchers introduce AISPA, short for Artificial Intelligence System Prompt Assurance, a user-centric framework for systematically auditing the system prompts that developers write to govern how AI applications behave. System prompts are used throughout commercial AI products but are rarely disclosed to the public or to regulators, which the authors say creates a serious trust and accountability gap. AISPA examines specific parts of a system prompt and scores them against eight dimensions that matter to users, then classifies each instruction as either protective of users or problematic. The team applied the framework to 3,249 instructions drawn from system prompts in 88 commercial AI products. Four findings stand out. First, system prompt design varies widely by product and developer: some organizations average over 60 protective instructions per product, while others average fewer than 5. Second, protective instructions are common but shallow: 98.9% of the audited products contain at least one, yet only 24% cover all eight dimensions of the AISPA taxonomy. Third, system prompts have grown steadily longer and more protective of users over time, which the authors read as user protection becoming a more visible concern in commercial prompt design. Fourth, despite that trend, problematic instructions remain widespread: about 40% of products contain at least one instruction that works against user interests, and protective and problematic instructions frequently sit side by side in the same prompt. The paper does not name which of the 88 products or organizations it audited, nor does it give a timeframe for the trend it describes.
Key facts
- AISPA audits system prompts along eight dimensions that matter to users, classifying each instruction as protective or problematic.
- The study reviewed 3,249 instructions from system prompts in 88 commercial AI products.
- Protective instructions per product average over 60 at some organizations and fewer than 5 at others.
- 98.9% of products contain at least one protective instruction, but only 24% cover all eight taxonomy dimensions.
- About 40% of products contain at least one instruction that works against user interests, often alongside protective ones.
Why it matters
System prompts quietly set the rules an AI application follows, yet they are almost never shown to the people using the product or to the regulators overseeing it. AISPA is the first framework the authors describe for auditing this hidden layer at scale, turning a question that was previously anecdotal, whether commercial AI products protect or work against their users, into something measured across thousands of instructions and dozens of products.
Who it affects
The audit speaks to developers who write system prompts, the companies that ship AI products built on them, and the users of those products whose interests the instructions can either protect or override. It also speaks to regulators, since the paper frames the lack of disclosure as an accountability gap that oversight bodies currently cannot see into.
How to use it
The paper's eight-dimension taxonomy gives developers a concrete checklist for self-auditing their own system prompts before release, rather than relying on ad hoc review. The same taxonomy could serve as a basis for external or regulatory audits, since it defines what counts as a protective instruction and what counts as a problematic one in reproducible terms.
How solid is it
The audit is grounded in a fairly large sample: 3,249 instructions across 88 commercial AI products, with the classification method laid out along the eight AISPA dimensions. What the text does not supply is which products or organizations were audited, over what time period the growth in prompt length and protectiveness was observed, or who the authors are, so the findings currently rest on the aggregate numbers rather than named, checkable cases.
Risks and caveats
The headline protective-instruction numbers coexist with a less reassuring one: roughly 40% of the audited products carry at least one instruction that works against user interests, and the paper notes protective and problematic instructions frequently sit in the same prompt, so the presence of user-protective language does not rule out harmful instructions alongside it. The anonymized product set also means the findings cannot be traced back to specific companies or checked against a particular product's published behavior.
“Fourth, despite this progress, problematic instructions remain pervasive: roughly 40% of products contain at least one instruction that works against user interests, and protective and problematic instructions frequently coexist within the same prompt.”
— AISPA audit findings