How to Evaluate an AI Trading Agent: The DXRG Benchmark Card

To evaluate an AI trading agent, define exactly what system is being tested, reconstruct what information it could see at each decision, and charge realistic execution costs. Then compare it with relevant baselines, repeat the run, and preserve the trace from instruction to outcome. A return chart by itself is not enough. The evaluator must be able to tell whether an apparent result came from the agent, the market regime, data leakage, an execution shortcut, or the surrounding software.

This guide turns that principle into a practical benchmark card, the one we use at DXRG when reviewing agentic-trading claims. It is designed for researchers, product teams, investors, risk owners, and anyone reviewing a claim about an autonomous or semi-autonomous market system.

Evidence boundary. This is an evaluation framework, not a report of investment performance. Passing the checklist makes a result easier to interpret and reproduce; profitability, safety, and deployment suitability remain separate questions.

Download the versioned card as JSON or CSV. For a controlled component swap, use our harness-transfer test method. For terminology, start with What Is Agentic Trading?. For the difference between backtest, replay, paper, shadow, and live evidence, use the Agentic Trading Evidence Ladder.

The Short Evaluation Standard

A credible agent evaluation should let an independent reader answer twelve questions:

1. What exact system was evaluated?
2. What decisions could it make without a human?
3. What data was genuinely available at decision time?
4. What actions and tools were permitted?

The first four pin down the system under test and what it may do. The next four follow the result from order to attribution:

5. How did model output become an order or transaction?
6. Which fees, slippage, delays, failures, and market-impact assumptions applied?
7. What baselines faced the same information and constraints?
8. How much of the result can be attributed to the agent?

The last four test whether the result survives repetition and inspection:

9. Was the result stable across periods, assets, prompts, and repeated runs?
10. What permissions, limits, logs, and interruption controls existed?
11. What class of evidence supports the claim?
12. Which artifacts allow inspection or reproduction?

If several answers are missing, the correct conclusion is not necessarily that the agent failed. It is that the claim is under-specified.

1. Freeze the Evaluation Subject

“The agent” is rarely one object. A trading system may include a model, system prompt, memory store, retrieval policy, and data adapters. It may also include a portfolio-state compiler, tool definitions, validation rules, an order router, and an execution venue. Together these parts form what we call the harness around the model. Changing any of those can change behavior.

Record a stable identifier for every material part:

• model provider, model name, and dated version;
• system prompt and strategy template hashes;
• tool schemas and permitted parameters;
• memory and retrieval policy.

The execution environment needs the same treatment:

• market-data sources and transformations;
• risk, validation, and execution code;
• deployment configuration and randomness settings;
• asset universe, venue, dates, and starting portfolio.

Do not label a result “GPT,” “Claude,” “Qwen,” or “an LLM agent” when the surrounding harness did most of the work. Conversely, do not assign every failure to the model when an adapter supplied stale data or an order router changed the requested action. The subject should be the complete decision system, with component-level traces when attribution matters.

2. Map Decision Rights

Autonomy is not a binary label. Write down which decisions the system can make and which require human approval.

Each step up in decision rights delegates another class of decision. An AI research assistant summarizes filings. An agent with asset-selection authority does more. An agent that proposes orders does more still, and one that can submit them holds a further grant. Beyond submission sits the execution agent operating inside strict size and venue limits, and beyond that the system that can change its own strategy, add tools, transfer funds, or expand its permissions.

A useful decision-rights map covers:

FINRA's discussion of AI agents distinguishes agents that support tasks from agents that can execute trades, while NIST's draft concept work on software-agent identity and authority proposes identifying an agent and constraining what it is authorized to do. The NIST material describes a potential project, not a final standard. For evaluation, those distinctions belong in the benchmark specification rather than in marketing copy.

3. Reconstruct the Information Set

Point-in-time integrity is the first major defense against false skill. At every decision timestamp, preserve the data the system could actually access, which may differ from the data that later became available.

Check for these data and label problems:

• revised fundamentals, restated filings, or corrected datasets;
• end-of-period values used before they were published;
• survivorship-biased asset universes;
• delisted assets omitted from history;
• future news embedded in summaries or model memory.

Leakage also hides in caches and model memory:

• labels calculated with future prices;
• cached web pages newer than the simulated decision;
• historical events memorized during model pretraining;
• prompts or examples that disclose the later outcome.

The last two are especially important for language-model agents. A backtest can appear temporally clean while a pretrained model recognizes the event or period. The May 2026 KTD-Fin preprint was built to examine temporal-data contamination and market/style exposure in financial-agent evaluation. Its broader lesson is simple: an evaluator needs controls that distinguish market reasoning from recognition of remembered history.

Useful controls include shifted or synthetic scenarios, renamed assets, counterfactual event details, held-out recent periods, and comparisons against an agent deprived of the potentially contaminated context. None is perfect alone. Together they reveal whether the system responds to the supplied state or recalls a familiar trajectory.

4. Define the Action Space

List every action the system may emit and every transformation applied before execution.

For a simple study, the action space might be buy, sell, or hold at a fixed interval. A production-like system may select order type, price, quantity, time-in-force, and venue. It may also set cancellation policy, leverage, collateral, and hedges. Tool calls can also fail validation, time out, return partial data, or be retried.

The benchmark must say:

• whether invalid actions are rejected, repaired, or silently coerced;
• whether a human or deterministic layer modifies the agent's proposal;
• whether repeated tool calls are allowed;
• how abstention is treated.

It must also say what happens downstream of a proposal:

• how unavailable tools and stale data are represented;
• whether the agent sees validation errors and can revise;
• which policy-valid action is ultimately submitted.

This is where attractive language can diverge from actual behavior. An agent may explain a cautious trade and emit an aggressive quantity. Evaluation should score the final action and the transformation path alongside the prose.

5. Model Execution Honestly

A trading result is jointly produced by decisions and execution. The test should make the execution clock explicit: when information became available, when the agent received the state, and how long inference and validation took. It should also record when an order became eligible for submission, how the fill price was determined, and what happened to unfilled or partially filled orders.

Charge all material costs: commissions, spread, market impact, and gas. Borrow, funding, protocol fees, and failed-transaction costs apply where the venue has them. Use venue rules and order-book or quote data when the claim requires that level of realism. A next-bar close or frictionless mid-price can be a useful simplified assumption, but it must be labeled as one.

The May 2026 survey preprint Agentic Trading: When LLM Agents Meet Financial Markets reviewed 77 studies and identified protocol incomparability as a central problem. Among the closed-loop empirical studies its authors audited, extractable time-consistent splits and explicit transaction-cost treatment were rare. That finding does not invalidate every reported experiment; it explains why benchmark specifications must make timing and costs inspectable.

6. Choose Baselines That Answer Different Questions

One baseline cannot establish every kind of value. Use several when possible:

• Do nothing or hold: Did activity add value after costs?
• Simple rule: Did the agent beat a transparent momentum, mean-reversion, or allocation rule?
• Traditional model: Did agentic orchestration add value beyond a non-language-model forecast?
• Stronger information baseline: Could a simple system using the same data reproduce the result?

A second set isolates the agent architecture itself:

• Same model without tools or memory: Did the agent architecture add value?
• Same strategy with deterministic execution: Did runtime discretion add value?
• Human-approved version: What changed when autonomy was introduced?

All baselines should face the same universe, timestamps, costs, capital limits, and execution opportunity. Comparing an agent with rich intraday data against a daily benchmark confounds information advantage with decision quality.

7. Separate Attribution From Outcome

Profit and loss is an outcome, not an attribution method.

A positive result may reflect rising market exposure, a concentrated factor bet, favorable asset selection, data leakage, or luck. Report risk and behavior alongside returns: exposure, turnover, concentration, and drawdown. Tail loss, holding time, capacity, and sensitivity to fees belong in the same report. Compare those properties with the baselines.

Then preserve the decision trace:

mandate → observed state → retrieved evidence → model output
→ proposed action → validation result → submitted action
→ venue response → portfolio state

The trace lets an evaluator locate the cause of an outcome. Was the mandate ambiguous? Was the market state stale? Did retrieval omit a source? Did the model choose an invalid action? Did a risk rule change the size? Did settlement fail? Without that chain, “the agent made money” and “the agent lost money” are both too coarse to teach much.

The first-party example is our own DX Terminal Pro: it shows why the instruction-to-settlement trace matters. Its historical live observations, controlled pre-launch tests, and separate internal harness evaluation are labeled separately rather than collapsed into one performance claim.

8. Repeat the Run

Language-model systems can vary across runs even when the market path is identical. Sampling settings are only one source of variation; model serving, tool results, prompt ordering, retrieval, and concurrency can also matter.

Repeat enough times to estimate a distribution rather than showcase one path. Report:

• number of independent runs and random seeds;
• median, dispersion, and worst-case outcomes;
• action agreement or flip rate at identical states;
• sensitivity to prompt wording and order.

Then slice the results:

• sensitivity to model or harness version;
• results by period, regime, asset, and venue;
• failures and abstentions alongside successful actions.

Keep one untouched holdout for the final evaluation. If a scenario repeatedly informs prompt or harness changes, it has become development data even if it was originally called a test.

9. Test Runtime Controls as Behavior

A policy document does not prove that a control works. Exercise the control under normal and adversarial conditions.

At minimum, test the limits and permissions:

• per-order, per-asset, and total exposure limits;
• leverage and loss limits;
• venue and asset allowlists;
• withdrawal and transfer restrictions.

Then test behavior under adverse conditions:

• stale-data and disconnected-market behavior;
• malformed or conflicting instructions;
• mandate conflicts and malformed state inputs;
• tool failure, duplicate submission, and retry behavior.

Finally, walk through interruption and audit:

• pause, kill, liquidation, and recovery procedures;
• immutable or tamper-evident logs;
• authority changes and credential rotation.

NIST's announced AI Agent Standards Initiative includes trustworthy agent interoperability and security in its planned scope. It is an initiative, not a final agent standard. For market agents, the practical evaluation question is whether identity, authority, and interruption controls survive contact with the actual tool chain.

10. Assign the Right Evidence Class

Name what the result actually demonstrates. A backtest supports a historical simulation claim. A replay can test sequential behavior against recorded events. Paper trading can test live data and operational timing while fills stay simulated. Shadow evaluation can compare proposed actions with a production path while withholding execution authority. Bounded historical live evidence shows what happened under a particular deployed configuration.

Each level demonstrates something different, and together they form no single quality ranking: a carefully controlled replay may answer a causal question better than a noisy live episode, while live operation exposes failures a replay cannot reproduce.

Use the Agentic Trading Evidence Ladder to label each result without inflating it.

11. Publish the Artifacts

A useful report should provide or identify what ran:

• benchmark-card version;
• code or immutable commit;
• prompt and tool-schema hashes;
• asset universe and date range.

It should identify how the result was measured:

• data lineage and availability rules;
• cost and fill model;
• baseline specifications;
• run-level outputs or sufficient aggregates;
• confidence intervals or distribution summaries.

And it should leave enough behind for inspection:

• trace schema and representative traces;
• excluded data and failed runs;
• limitations, conflicts, and correction history.

Keep private data and credentials private. Reproducibility leaves user information, secret keys, proprietary positions, and exploitable control surfaces out of the public package. It does require enough structure to understand what was done and to detect when two evaluations are not comparable.

The DXRG Benchmark Card

The downloadable benchmark card is deliberately compact and includes a fillable values object; we version it so two evaluations of the same system can be compared card to card. Replace each null with the evaluated configuration's disclosure, or fill the value column in the CSV version. Issue a new card version when the subject, data, action space, execution assumptions, or primary metric changes.

The card has twelve sections:

The CSV contains the same required fields in flat form for registries and audits.

Audit a Benchmark Before You Use Its Score

At DXRG, we start a benchmark audit by writing the claim we want the score to support. We inspect construct fit and the complete evaluation subject first. We then review the protocol, point-in-time data, execution model, attribution, uncertainty, and artifact trail. A strong score can answer a narrow question well. It needs another layer of evidence before it can support a statement about a different market, harness, or deployment.

We published that review path as the Trading-Agent Benchmark Audit Registry in JSON and CSV. Version 1.2 freezes two completed source reviews and keeps the CLQT row UNRUN, with its finding, review date, and verdict fields empty.

The source plan uses the initial public draft of NIST AI 800-2 for benchmark fit, protocol, and reporting practices, plus NIST AI 800-3 for the distinction between a fixed benchmark result and a generalized estimate. Trading-specific review targets include the Agentic Trading survey, Agent Market Arena, KTD-Fin, and CLQT. Those sources test different mechanisms and use different environments, so the registry keeps their findings separate.

For the first completed review, we froze Agent Market Arena at arXiv:2510.11695v2, revised October 30, 2025. We first inventoried the paper's objectives, evaluated subjects, inputs, and actions. We checked its dates, metrics, baselines, and artifacts in the same pass. We then reopened every required field at an exact section, table, or appendix locator. Seven audit units were partially disclosed. Repeated-run uncertainty was undisclosed.

The paper reports an August 1 to September 30, 2025 study covering BTC, ETH, TSLA, and BMRN. It names four agent frameworks and five model families, gives them common daily inputs, and maps their outputs to simulated BUY, SELL, or HOLD signals. The frozen source leaves material reconstruction gaps in immutable system versions, exact point-in-time controls, cost and order mechanics, matched attribution, repeated-run uncertainty, and downloadable traces. We therefore treat it as reported benchmark evidence inside that window. Stronger claims about realistic execution profitability, causal agent-versus-model attribution, reproducibility, or transfer need additional evidence.

For the second review, we froze KTD-Fin at arXiv:2605.28359v1, submitted May 27, 2026. We followed the anchor model from its prompt through six read-only tools into a typed JSON action and the next-open simulator. A second inventory covered the model panel, dates, masking, repeated runs, baselines, and metrics. We then checked each field against the 20-page paper and its arXiv source package. All eight audit units were partially disclosed.

The paper reports a price-only CSI300 simulation from January 1, 2024 through April 10, 2026. On each trading day, the model researches through the tool loop and submits a typed action. The simulator applies next-day-open execution, T+1 sell availability, fees, structured retries, and a hold fallback. The paper also reports multi-seed runs, a 200-sample de-anonymization probe for each attacker model, and a Barra-style return decomposition. The mask hides tickers and calendar identifiers across prompts and tools. It does not establish that every future-information path is closed.

The frozen manuscript does not provide an artifact locator for the environment, scripts, or evaluation dataset it says were released. Provider snapshots, point-in-time constituent lineage, raw runs, and complete traces also remain unavailable. The paper gives two different starting years for Qlib baseline training, 2008 in the setup and 2015 in the introduction. We therefore treat KTD-Fin as a reported masked historical simulation with material disclosure limits. DXRG did not rerun the benchmark, validate its data or results, certify the mask, or infer live performance or transfer from the paper.

Our DX Terminal Pro record provides the practical reason for the trace unit. A settled transaction can still contain an earlier failure in mandate compilation, state assembly, validation, or retry handling. Following the linked mandate-to-settlement trace lets a reviewer locate that boundary before a market outcome gets assigned to the model.

How to Read a Claim in Sixty Seconds

When time is short, use this order:

1. Find the evidence class.
2. Find the exact dates, assets, venue, and costs.
3. Check whether the information set was point-in-time.
4. Identify what the agent controlled versus what deterministic software controlled.

Then stress the result itself:

5. Look for same-information baselines and repeated runs.
6. Inspect the worst case, not only the average.
7. Confirm that an instruction-to-outcome trace exists.
8. Read the limitations before the headline result.

If the report cannot answer those questions, treat the number as a lead for further investigation rather than proof of skill.

What Good Evaluation Produces

The purpose of a benchmark is to make results comparable enough that failures can be found, fixes can be tested, and authority can expand only when evidence supports it. A leaderboard is a byproduct, never the goal.

A strong evaluation may conclude that an agent should remain in replay, paper, or shadow mode. That is still a useful result. It reveals what the system can do, which failure modes remain, and what evidence would justify the next boundary.

The standard is therefore not “Did the chart go up?” It is: Can we reconstruct what the system knew, what it was allowed to do, what it actually did, why the result occurred, and whether the evidence supports the claim being made?

---

Version 1.5. Published July 18, 2026; refreshed August 18, 2026. Corrections: poof@dxrg.ai. This article is research and educational material, not financial advice.