A contributed example belongs to the release that produced it
By DX Research Group · · Data and learning flywheels
A proposed lineage record prevents old failures and later fixes from sharing an invented context.
A contributed example should retain the release that produced the observed behavior. Replaying it under a later harness answers whether that later harness handles the saved case; it cannot rewrite what the owner originally saw. We propose a contribution record with separate identities for the observed event, curated fixture and each replay.
The controls paper companion records harness revisions and their measured trace effects. It illustrates why lineage matters: behavior can change when the surrounding instructions change even when a base model remains the same. A contribution system needs enough version information to attribute a correction to the relevant component instead of placing every failure in a single model bucket.
Three records for one correction
In an illustrative sequence, an owner observes a rejection on release A on Monday. A curator receives the report on Wednesday after release B has changed the error parser. On Thursday, release C passes a reconstructed fixture. The observation retains release A, the curator record carries Wednesday's review time, and the replay points to release C. Replacing A with the current release during ingestion would create a false regression against C.
We would store the model identifier, harness configuration and tool-response schema available for the original turn. Relevant account and mandate versions belong in the restricted source record. The export should retain synthetic equivalents where authorized transformation permits them. Missing version fields remain unknown; the curator should avoid guessing from the report date when deployments can overlap.
The fixture itself needs a revision identity. If the curator changes a stale tool response into a fresh one, the expected behavior may change. That new artifact should link to its parent and describe the edit. A passing result on the altered fixture supplies evidence about the altered input, while the original failure remains in the observation history.
Permission lineage travels alongside technical lineage. A replay may use a corrected fixture internally while a public export permits only a shorter synthetic example. Matching technical content does not imply matching permitted use. Each consumer checks the artifact version it actually receives.
Use release-aware denominators
A proposed release review would count eligible fixtures by source release and failure family. Suppose an illustrative corpus contains eight cases from A and two from B. If C passes all ten, the report should show those populations and the reconstruction completeness. If only five could be reproduced because old schemas are unavailable, five passes cannot be described as ten repaired failures.
For new behavior, a prospective cohort should collect cases after C becomes active, keeping owner feedback and automatic trace checks distinguishable. Historical replay measures known-case handling; prospective evidence tests whether related failures still arise under current inputs.
This proposed architecture would let DXRG use precise contributions to improve regression coverage while preserving the original evidence. It establishes no current contribution-training infrastructure or measured forecasting gain. The immediate research advance would be a defensible statement of which release produced a failure, which artifact was tested and which later release changed its handling.