The Agentic Trading Evidence Ladder: Simulation, Replay, Paper, Shadow, and Live

An agentic trading result is incomplete until it says where the agent acted, what could actually happen, and what remained simulated.

That sounds obvious. In practice, terms such as backtest, paper, shadow, and live are often compressed into a single story of progress. A profitable replay becomes evidence that an agent can trade. A paper account is described as live because it consumed live prices. A live pilot is treated as proof of general strategy quality even when it covered one venue, one regime, and a tightly capped action set.

The evidence ladder prevents those category errors. It separates five environments:

1. Simulation: a constructed market and constructed consequences.
2. Replay: recorded historical events delivered to an agent under a controlled clock.
3. Paper: current market data and simulated order handling.
4. Shadow: current production inputs and full decisions, with orders prevented from reaching the venue.
5. Live: real orders, real execution, and real economic consequences.

We treat the five levels as separate evidence environments, each answering a different question. Simulation can provide cleaner counterfactuals than a live deployment. Replay can make a regression exactly reproducible. Shadow can expose production drift without putting capital at risk. Live trading can reveal execution and operational failures that no offline environment contains, but a narrow live run may say little about causality or transfer.

If you first need the system definition, begin with what an agentic trading system is. The short version is that the research subject is the model plus its harness: market data, memory, tools, and mandate compiler. It also covers the policy checks, execution path, portfolio state, monitoring, and human control surface.

Evidence boundary: The level belongs to a particular claim rather than permanently to a product or research team. One system can have simulated strategy evidence, replayed safety evidence, shadow operational evidence, and live settlement evidence at the same time. Label each result separately.

The Ladder at a Glance

The practical question is therefore “Which uncertainty did each test remove?”

Level 1: Simulation

In simulation, the evaluator constructs the environment. Prices may come from a stochastic process or a multi-agent market. Counterparties may be scripted. Fills, fees, latency, information, and liquidation are rules written by the experimenter. Capital is not exposed.

That artificiality is a feature when the objective is control. A simulation can repeat the same state while changing one prompt sentence, risk limit, memory policy, or tool permission. It can generate market gaps, stale data, contradictory instructions, venue failures, or manipulation attempts at frequencies that would be impractical or irresponsible to wait for in production. We often reach for this level first when the question is whether an intervention caused a behavioral change inside the specified model of the world.

Simulation evidence becomes weak when its assumptions disappear from the claim. The U.S. Commodity Futures Trading Commission's advisory on hypothetical trading systems highlights several recurring omissions: bid-ask spreads, commissions, liquidity, execution uncertainty, market impact, and the benefit of hindsight. Those omissions define what the return means.

A useful simulation report should disclose the simulator itself:

• the market generator or participant model;
• whether the evaluated agent can affect subsequent state;
• the fill, fee, latency, borrow, liquidation, and rejection rules.

It should also disclose the evaluation protocol:

• the information available at each decision time;
• the number of configurations tried before selecting the reported one;
• the random seeds and uncertainty interval;
• the exact action and loss constraints.

The right claim is narrow: “Under this simulator, the policy respected the configured position cap in the reported stress episodes.” A broader claim, “The agent is safe,” goes beyond what the simulation shows.

Level 2: Replay

Replay replaces synthetic market history with a recorded sequence. A test harness releases quotes, trades, news, or order-book updates according to their historical availability and asks the agent to act without seeing the future.

Replay is especially useful for deterministic regression. If a prompt change causes a tool-selection failure at event 18,432, the evaluator can rerun the same history with the old and new systems. It can test multiple models against identical state transitions and preserve a complete trace from input through reasoning, policy validation, and intended action.

The defining limitation is counterfactual silence. In a pure replay, the future tape has already happened. An agent's intended order leaves the recorded book untouched: no information revealed, no liquidity consumed, no change in what another participant does next. A fill model can estimate consequences, but it remains another model layered over history.

Replay is also vulnerable to research overfitting. Bailey, Borwein, López de Prado, and Zhu's peer-reviewed paper on the probability of backtest overfitting addresses the selection problem created when many strategy variants are tested and the apparent winner is reported. A held-out date range is helpful, but it does not erase repeated researcher decisions made after looking at earlier results.

Minimum replay documentation should include data provenance, symbol and venue coverage, timestamp semantics, and corporate-action handling. It should also pin the as-of availability time for every feature, the train/validation/test boundaries, attempted variants, fill assumptions, and costs. If news published at 10:00 reached the historical vendor at 10:03, the agent receives it at 10:03.

Replay can be stronger than paper for a causal prompt or policy comparison because the input path is repeatable. It is weaker for production liveness because the evaluator controls the clock.

Level 3: Paper

Paper trading runs against the present. The agent wakes on a real schedule, consumes a current market feed, calls production-like APIs, maintains state across hours or days, and submits orders to a simulator rather than a live venue.

This level tests problems that replay compresses away: expired credentials, reconnects, delayed jobs, and duplicate events. It also surfaces partial service outages, clock drift, stale state, and market-hours logic, along with whether the agent continues to follow its mandate after hundreds of turns. Paper is valuable operational evidence even when its return is economically uninformative.

The fill boundary must remain prominent. Alpaca's current paper-trading documentation describes its environment as a real-time simulation and lists omissions including market impact, information leakage, latency slippage, non-marketable limit-order queue position, price improvement, regulatory fees, and dividends. It also notes that paper platforms can differ in fill, liquidity, return, and data assumptions.

Those limitations generalize. A paper agent can demonstrate that it emitted syntactically valid orders for 30 days. It cannot demonstrate that those orders would have occupied the same queue position, received the same partial fills, or produced the same subsequent market.

A credible paper report separates at least four denominators:

• decision cycles attempted;
• decisions successfully produced;
• orders accepted by the paper interface;
• simulated fills produced by that interface.

Collapsing them into “trade success rate” hides where the system failed.

Level 4: Shadow

Shadow testing places the candidate in the production information path while preventing its actions from changing production.

The pattern exists beyond finance. Amazon SageMaker's production-validation documentation defines a shadow variant as receiving replicated production requests while its responses are logged for comparison and withheld from the caller. For trading, the safety invariant is stricter: market data, portfolio snapshots, news, and mandates may reach the candidate, but every order-capable output must be intercepted before routing.

Shadow evidence can answer:

• Does the system remain available at production cadence?
• Do state and memory stay consistent across long horizons?
• How frequently does it disagree with the incumbent policy, and which decisions would risk checks reject?
• Do model, feature, or market-regime shifts change behavior?
• Can operators reconstruct every intended action from the trace?

It still cannot validate fills, market impact, or realized return. A “shadow P&L” is counterfactual performance computed from intended orders and a fill model. Live inputs alone never make it live.

The no-order invariant should itself be tested and monitored. A shadow service with credentials or a network path that can accidentally reach a venue has not fully removed execution risk. Record all abstentions, timeouts, validation failures, and intercepted orders; logging only attractive hypothetical trades recreates survivorship bias.

Level 5: Live

Live evidence begins when the agent's order can reach a venue and create a real fill, fee, position, obligation, or loss. “Live data” is different from live trading, and an operator manually copying selected recommendations is human execution, one step removed.

Live operation reveals the remaining path: authentication, order admission, venue rejection, partial fills, queue dynamics, and slippage. It continues through fees, portfolio reconciliation, liquidation, and downtime, with market feedback closing the loop. It also exposes human and organizational behavior around overrides, incident response, and changing risk appetite.

Because errors can compound quickly, live evidence requires more than a profit chart. For U.S. broker-dealers with market access, the SEC's guidance on the Market Access Rule describes controls intended to reject orders that breach capital, credit, price, or size parameters and requires regular review of those controls. Separately, FINRA's guidance for member firms on algorithmic-trading supervision recommends independent quality assurance, retrievable code versions, rapid-disable mechanisms, limited pilot deployment where feasible, heightened monitoring, and continuing review after production. These sources are scoped to the firms and activities they govern; they are useful control references, not a universal agent rulebook.

Those controls make bounded observation safer and more interpretable; they say nothing about strategy quality.

A live result should identify the venue, instruments, dates, capital at risk, leverage, and permitted actions. Model and harness version belong in the same disclosure. It should also report the execution record: order count, fill count, rejected-order denominator, and fees. Slippage method, outages, operator interventions, risk caps, and halt events complete that record. Results should be reported both before and after costs. Strategy performance, settlement reliability, and risk-control effectiveness are three different claims.

In our own DX Terminal Pro deployment, we label the 21-day real-capital observations, the controlled pre-launch interventions, and the separate internal execution evaluation as different evidence classes rather than combining them into one “live” result.

Live evidence is not automatically the strongest kind. A five-trade live pilot has high execution realism and very low statistical power. A 30-day live run during one market regime can support a bounded operational claim while leaving cross-regime transfer unanswered. A profitable deployment without a concurrent control cannot identify whether the agent, market beta, luck, or selection produced the difference.

Evidence Is a Vector, Not a Badge

The five levels change several properties at once:

That is why later evidence should add to earlier evidence rather than replace it. Every live system still needs deterministic unit tests, stress simulations, frozen replays, paper integration tests, and shadow comparisons when material components change.

NIST's AI Risk Management Framework Core treats risk management as continuous across the AI lifecycle and calls for post-deployment monitoring, incident response, recovery, and change management. For an agentic trader, we treat promotion as a change in which uncertainties and consequences are now real, never a graduation.

That trace-first, evidence-labeled discipline is the default we are building toward: DXRG is building reusable infrastructure for evaluating, constraining, and eventually deploying market agents under explicit permissions and evidence gates.

A Minimum Evidence Bundle

Before interpreting a result, ask for:

1. A frozen subject. Commit, model, prompt, tools, memory policy, risk policy, and configuration.
2. A named evidence level per claim. Each claim gets its own label.
3. A complete data boundary. Venue, instruments, dates, timestamps, missingness, and information availability.
4. An end-to-end trace. Mandate, state, model output, tool call, validation, order, fill, and portfolio effect where applicable.
5. Explicit denominators. Attempts, valid decisions, submitted orders, accepted orders, fills, accounts, and time at risk.

The remaining items make results comparable and auditable:

6. Costs and constraints. Fees, slippage, latency, borrow, liquidity, action limits, leverage, and capital caps.
7. Controls and uncertainty. Baseline, concurrent control where possible, confidence interval, attempted variants, and contamination.
8. Failure reporting. Timeouts, rejected orders, policy violations, incidents, overrides, and halted runs.
9. A transfer boundary. What changed across models, venues, assets, users, and regimes, and what stayed fixed.

For a fuller scorecard covering behavioral validity, execution integrity, risk, and operational reliability, use the companion guide on how to evaluate an AI trading agent. For the control layer that should constrain any promotion, use the agentic-trading guardrail matrix.

Promote Only to Answer the Next Question

The five levels are useful when advancement is tied to unresolved risk:

• Move from simulation to replay when synthetic tests pass and the next question concerns real historical states.
• Move from replay to paper when leakage, overfitting, and cost assumptions are documented and the next question concerns wall-clock operation.
• Move from paper to shadow when the system is stable and the next question concerns production inputs, scale, and drift.

Live exposure follows the same logic:

• Move from shadow to a constrained live pilot only after the no-order invariant, monitoring, risk limits, reconciliation, and rapid shutdown have been independently tested.
• Increase live exposure only when the prior window is interpretable, incidents are closed, and the new exposure answers a named question.

Sometimes the correct move is downward. A live incident should become a deterministic regression, a replay case, and a simulated stress family before the fix returns to shadow or live operation.

The most trustworthy agentic trading claim is rarely the largest one. It is the one whose subject, environment, denominator, consequences, and limits are so clear that another evaluator knows exactly what was learned, and exactly what remains unknown.

---

This framework is for evaluating evidence, not recommending a strategy or financial product. Nothing here is financial advice.