Strip private details while preserving the failure

By DX Research Group · · Data and learning flywheels

A proposed transformation contract keeps the relationships needed to reproduce an owner correction.

Redacting a support report can remove the very relationship that caused an agent failure. Replacing every number with a blank protects detail but leaves a curator unable to tell whether a limit was exceeded. We propose producing a minimal synthetic fixture from a contributed trace, with an explicit account of which relationships survive and which questions the fixture can answer.

The controls paper companion links mandate, rendered context, typed action and validation. That structure suggests what to preserve: the instruction relevant to the failure, the state it acted on and the decision under review. A reproducible contribution rarely needs the owner's complete account history. Our proposed pipeline would accept only explicitly authorized material and keep the original restricted while reviewing the transformed artifact.

Preserve an inequality, remove a balance

In an illustrative report, an owner has $8,400 equity and a $420 entry cap. The agent proposes $510. A synthetic fixture can use $2,000 equity, a $100 cap and a $121.43 proposal, preserving the approximate ratios: the cap is 5% of equity and the proposal exceeds it by about 21.43%. The fixture tests interpretation of a relative limit without publishing the original balance.

That transform would be unsuitable for a bug involving an absolute $500 threshold. In that case scaling crosses the threshold and changes the failure. The curator must declare whether the mechanism depends on ratios, absolute values, precision or an external instrument rule before choosing a transformation. A fixture with rounded values should carry its tolerance, so an arithmetic discrepancy does not become a new label.

Addresses should become unrelated placeholders rather than reversible abbreviations. Free text needs review for distinctive strategy phrases, named counterparties and embedded credentials. Timestamps can reveal identity through public trade matching, so a time-sensitive fixture should retain elapsed intervals while using a synthetic origin when calendar time has no causal role. Removing an identifier alone offers little protection if the remaining event sequence identifies the same account.

Reproduce the diagnosis after transformation

We would run the original authorized example and transformed fixture through the same frozen evaluator. The curator compares the failure classification and relevant policy result, recording any changed behavior. If transformation destroys the mechanism, the artifact remains unsuitable for that evaluation even if it looks tidy. It can be revised or retained in a more restricted research setting within its permission scope.

The acceptance report should state retained relationships, removed fields and reconstruction limits. It should also record whether a second reviewer could recognize a real account from public information. This is a proposed review procedure, with no claim that current DXAP contributions already follow it. Its measurable value would be the proportion of permitted reports converted into independently reproducible fixtures without exposing unnecessary private detail. Predictive improvement would require a separate held-out study.

Sources

Related field notes