Microsoft builds synthetic enterprise tenants to evaluate Copilot

Microsoft Research has described how it tests Microsoft Copilot without using real customer data. The problem, in the authors' words, is that evaluating enterprise AI needs test data with the full complexity of real organizations (reporting structures, project histories, communication patterns, domain workflows), yet real enterprise data contains sensitive information that cannot be used for evaluation under any circumstances.

Their answer is the Industry Evaluation Data Generation Platform, built on three pillars: synthetic enterprise environments with realistic users, meetings, emails, documents and org structures; automated query-generation pipelines for rapid, reproducible creation of evaluation datasets; and structured quality frameworks to check that the synthetic data is good enough. The stated aim is to answer two questions: is Copilot doing what users need, and can that be proved without touching their data.

Each simulated workspace is a "tenant", an isolated organizational environment with its own users, data and configurations. Rather than small hand-made test fixtures, the team generates full-scale tenants. They are grounded in public sources: O*NET occupational data from the U.S. Department of Labor sets roles, responsibilities and the mix of work activities, and shapes industry workflows such as clinical trial reviews or portfolio updates; online job postings supply role titles, reporting hierarchies and departmental makeup; SEC filings inform organizational scale, business units and financial context. Microsoft's legal and policy organization reviewed and approved all of these sources. The environments are not static snapshots. They simulate organizations that change over time, with meeting histories, project timelines, email threads that develop over weeks and organizational changes. That matters because many Copilot queries are time aware, such as "What did the team decide last Thursday?" or "What's changed since the last quarterly review?"

The team generated three environments: a pharmaceutical company (regulatory submission workflows, clinical trial discussions, cross-functional safety review meetings), a capital markets firm (deal pipelines, compliance communications, client portfolio reviews) and an IT/software company (sprint planning, code review discussions, product launch coordination). The point is to test Copilot on the questions real users in those fields ask, rather than generic cases that may miss domain-specific failure modes.

Because writing thousands of grounded, industry-specific queries by hand is prohibitively expensive and slow, a six-stage automated pipeline produces them. It starts from a large library of common workplace tasks, narrows them to what is relevant for an industry, and adapts them with domain language and context. AI then generates queries grounded in the synthetic environment's data, rewrites them to sound more natural and human, and validates each for accuracy and uniqueness.

Quality is checked with an internal semantic evaluation framework. The post lists structural fidelity (realistic reporting hierarchies, team sizes, cross-functional relationships), contextual relevance (whether queries and workflows represent actual enterprise usage), behavioral realism (human-like communication and collaboration) and content safety (no harmful, offensive or personally identifiable material, and no impersonation of real people). It introduces these as "three dimensions" but lists four.

The authors say end-to-end automation addresses four problems of manual work: reproducibility, scale across industries and configuration variants, iteration speed (manual creation takes days) and cross-team consistency of formats and quality. Dataset creation moved from a multi-day manual process to automated runs. They estimate savings of approximately $100,000 per evaluation dataset and $300,000 in total project savings, and say the more important result is that any team can regenerate a dataset from the same configuration and get identical results.

On privacy, the post says no real customer data is used at any stage; everything from org structures to email content and meeting transcripts is synthetically generated. Safeguards include privacy-preserving aggregation where real-world statistical distributions are referenced, legal review of public sources, and provenance and lineage tracking. As a result, teams across Microsoft can use the datasets freely for experimentation, benchmarking and regression testing without privacy review overhead or data handling restrictions.

Reported results: the composite quality metric, which measures how closely synthetic data mirrors the complexity and patterns of real enterprise environments, improved 39% compared with the initial generation approach. Some industry-specific environments scored above 0.9 out of 1.0. The gains came partly from incorporating realistic industry signals and partly from automating quality improvements. Each environment now has hundreds of users with complete communication histories across the three verticals. The platform has been published to an internal shared catalog and established as the first standardized evaluation baseline for Microsoft Copilot.

Key facts

Microsoft Research's Industry Evaluation Data Generation Platform builds fully synthetic Copilot tenants, so Copilot can be tested with no real customer data at any stage.
Three environments were generated (pharmaceutical, capital markets, IT/software), each with hundreds of users and full communication histories, grounded in O*NET data, online job postings and SEC filings.
A six-stage automated pipeline turns generic workplace tasks into industry-specific, validated test queries, and any team can regenerate a dataset from the same configuration with identical results.
Microsoft reports a 39% improvement in its composite quality metric versus the initial approach, with some environments scoring above 0.9 out of 1.0.
The authors estimate savings of approximately $100,000 per evaluation dataset and $300,000 in total project savings.

Why it matters

Testing an AI assistant on enterprise work runs into a hard limit: the realistic data it needs is the data that cannot be used. Microsoft's approach is to build lifelike but fully synthetic companies, with meetings, emails and org changes unfolding over weeks, so that time-aware questions such as "what changed since the last quarterly review?" can be tested at all. The post also claims the industry-specific setups help find issues that generic benchmarks would miss, and that the platform is now the first standardized evaluation baseline for Copilot.

Who it affects

Mainly teams inside Microsoft that build and test Copilot: the datasets sit in an internal shared catalog and can be reused across teams. Beyond that, the method is a reference for anyone evaluating enterprise AI assistants who faces the same conflict between realistic test data and privacy. The three industries covered so far are pharmaceuticals, capital markets and IT/software.

How to use it

The source describes an internal Microsoft platform and gives no indication that the data or the platform is released publicly. What outsiders can take is the recipe: ground synthetic organizations in public sources (occupational data such as O*NET, job postings, SEC filings), make the environments evolve over time, generate queries through a staged pipeline that adapts, grounds, humanizes and validates them, and score the result on structure, context, behavior and safety. Inside Microsoft, teams can use the datasets for experimentation, benchmarking and regression testing without privacy review.

How solid is it

This is a first-party description by Microsoft Research of its own system, and the figures are self-reported. The composite quality metric is defined only in a parenthetical; the 39% baseline and the actual score values are not given, only that some environments exceeded 0.9 out of 1.0. The method behind the $100,000 per dataset and $300,000 total savings is not given, and the post labels them estimates. No figures on Copilot accuracy, and no examples of issues the platform found, appear in the visible text.

Risks and caveats

The 39% gain is relative to Microsoft's own earlier generation approach, measured with an internal scoring system, so it says little about how closely the data matches any particular real company. Only three environments exist, and only some reached scores above 0.9. The claim that the platform surfaces problems generic benchmarks miss is general and comes without examples. The privacy argument rests on the data being synthetic and safety-screened, as described by its builders. The post also says the quality framework has three dimensions but lists four.

“Is Microsoft Copilot actually doing what our users need it to do, and can we prove it without ever touching their data?”

— Microsoft Research blog post