Researchers build synthetic customer twins to stress-test bank chatbots
A new paper tackles a problem regulated industries keep running into: LLM chatbots are spreading through customer service in banking, but there is no cheap, scalable way to check they behave safely before launch. The authors' fix has two parts. First, they build synthetic customer agents, digital twins grounded in real transactional and conversational data, that can be generated automatically and tuned to act out different customer profiles and interaction styles. Testing shows these synthetic agents track real customers closely in meaning and tone, rarely hallucinate, and can reproduce specific personality traits on command when the researchers dial them in deliberately. Second, the authors wrap those synthetic agents in a validation framework that layers three checks together: an LLM acting as an automated judge, human expert testers, and adversarial probing designed to provoke failures. Running this framework across a range of scripted scenarios, varying customer emotional state, demographic group, and language used, the chatbots held up robustly. The authors then took the whole approach out of the lab and applied it to a live, customer-facing chatbot at a large UK bank (unnamed in the paper), presenting it as a workable route for financial institutions to meet regulatory compliance bars for chatbot deployment.
Key facts
- The paper proposes synthetic customer agents (SCAs), AI digital twins of real bank customers built from actual transactional and conversational data, that can be generated automatically and behaviourally tuned.
- SCAs tested closely align with real customers semantically, show low hallucination rates, and reproduce targeted personality traits under controlled intervention.
- The validation framework layers three methods: automated LLM-as-a-Judge scoring, human expert testing, and adversarial probing, run across scenarios that vary customer emotion, demographic group, and language.
- The method was applied to validate a live customer-facing chatbot at an unnamed leading UK bank, which the authors present as a scalable path to regulatory compliance.
Why it matters
Banks and other regulated firms are rolling out LLM chatbots for customer service, but manual testing for safety and compliance does not scale. This paper is a direct attempt to close that gap: instead of relying on scripted test cases or slow human-only review, it proposes generating large populations of realistic synthetic customers to probe a chatbot automatically before and after it ships.
Who it affects
Banks and other regulated financial institutions deploying customer-facing LLM chatbots, and by extension the compliance and risk teams that have to sign off on those deployments. It also affects the customers on the other end of these chatbots, whose real transaction and conversation data feeds the synthetic twins used to test the systems.
How to use it
The framework combines automatically generated synthetic customer agents with a three-layer check: an LLM-as-a-Judge scoring pass, testing by human experts, and adversarial probing meant to surface failure modes. Institutions would run their chatbot against scenario sets that vary customer emotional state, demographic group, and language before and during deployment, exactly as the authors did on a live chatbot at a UK bank.
How solid is it
This is a single case study on one unnamed bank's chatbot, described by the paper's own authors, with no independent replication and no named institution to check the claims against. The paper reports strong results, high alignment with real customers, low hallucination, robust performance across scenarios, but these are self-reported outcomes from an arXiv preprint that has not gone through peer review.
Risks and caveats
Synthetic customers built from historical data may not capture genuinely novel or adversarial customer behaviour that only shows up after launch, so passing this kind of validation is not a guarantee against real-world failure. There is also a governance risk: a framework a bank runs and grades itself against could be used to demonstrate compliance on paper without independent oversight, and the paper names neither the bank nor the specific chatbot involved, which limits any outside verification of the claims.