If agents run the tests, who prepares the data?

An agent changes the order service, identifies affected workflows, writes a regression test, and fixes a failure. Much of the verification happens before the pull request opens.

Now ask the agent to test what happens when a customer crosses their credit limit while an earlier order is still being processed.

The customer must exist, with the correct credit exposure and order state across systems. Another test must not change those records halfway through the run.

As agents take on more verification work, they need a dependable way to create these test conditions. That is why the debate about replacing continuous integration matters to test data.

The debate, beyond the headline

In late August, Paul Stack described a workflow that verifies and repairs changes before a pull request opens, then passes structured evidence to CI. His trial covered 35 pull requests alongside the existing pipeline, a sample too small to establish industry-wide reliability. Paul Stack, August 26, 2026

In his September 1 response, Martin Fowler pointed out that continuous integration is an engineering practice, and testing changes before sharing them has always been part of it. Coordinating changes into a working mainline still requires verification. Martin Fowler’s response

Stack’s September 3 follow-up acknowledged that separating verification from merge coordination predates today’s agents. What changes is who handles implementation, checking, and repair. An agent can increasingly operate that cycle, while independent infrastructure enforces integration and release requirements. The Feedback Loop Is Moving Out of CI

What changes in the next two years?

Over the next two years, agents will likely run more checks, propose tests, and investigate failures during development. Some duplicated pipeline work may disappear when earlier results are trustworthy and still apply to the integrated change.

Enterprise teams are less likely to let an implementing agent choose every check and approve its own release. An agent can misunderstand a requirement and build both code and tests around that mistake. Anthropic’s experiments describe problems with self-evaluation and the value of a separately calibrated evaluator. Anthropic’s harness research

Two changes can pass separately and fail together. GitHub’s merge queue addresses this by checking a proposed change with the latest target branch and preceding queued changes. Moving tests earlier does not remove that dependency. GitHub’s merge queue documentation

Through September 2028, we expect this shared responsibility to persist. Adoption should move fastest in applications with clear requirements, useful instrumentation, and reproducible environments. Complex enterprise landscapes have more dependencies to resolve.

The bottleneck moves to business state

A familiar pipeline runs a known suite against prepared data. An adaptive workflow may uncover a new scenario while investigating a change. Waiting for a refresh or finding suitable records manually can consume the time saved by generating code.

The useful request becomes more specific than “generate 10,000 customers.” An agent might ask for customers just below, exactly at, and just above a credit threshold, each with the relevant outstanding orders.

Valid field formats are only a starting point. A customer record can satisfy its schema yet be unsuitable because the account is blocked, order history is missing, or its status differs across systems.

In an illustrative SAP-connected credit-limit test, a data service would select an approved scenario, prepare related customer and order records, deliver them through supported interfaces, and confirm their state. The test would check the outcome against an independently maintained business rule.

A failure at the exact threshold should be preserved and replayed after a fix. If an order could not be created, the result should report incomplete setup. These failures require different responses.

A few generated values or a fixture library may suffice for a unit test. A specialist platform earns its place by preparing connected business states, enforcing access rules, delivering into test systems, and explaining what happened.

More agents may mean more data requests, but not larger datasets. Small sets can cover relevant boundaries, histories, and combinations. A million ordinary records can still miss the condition that exposes a defect.

A failing test needs a memory

Repeating a prompt does not guarantee the same data. Even a fixed random seed is insufficient if the generator, schema, reference data, clock, or environment changes. Replay needs versioned inputs and retained data, or a controlled way to reconstruct the state.

Separate checkouts do not automatically isolate test data. Shared SAP systems may require reserved records, separate tenants or clients, unique identifiers, or coordinated access to scarce scenarios. Cleanup must account for business objects that need cancellation or reversal rather than deletion.

What this means for DataMaker

DataMaker already exposes tools through MCP for coding agents. Its Python scenarios support related records, conditional logic, and delivery across targets, connecting data preparation to agent workflows and established automation. DataMaker MCP, DataMaker scenarios

We see four capabilities that DataMaker should develop further.

These are product direction principles, with priorities guided by customer workflows. A data receipt documents the prepared state; the test runner must still establish what executed and whether it behaved as expected.

DataMaker’s opportunity is to become a dependable source of test conditions for QA engineers, pipelines, and coding agents. Each needs the right business state, safe access, and a way to reproduce what happened.

As agents take on more of the verification loop, reliable access to those conditions will determine how much autonomy teams can use with confidence.