When should a contributed case become a release test?

A useful user report can identify a failure before it can establish a fix. The release problem is deciding how that report becomes a reproducible case, which parts developers may inspect, and what independent evidence justifies shipping the resulting change. We propose an offline admission process that makes those transitions explicit.

The DXAP chat guide lets owners discuss decisions and review proposed changes. The controls paper companion describes historical prompt revisions tested against replayed scenarios. Together they motivate a possible feedback-to-release path: owners supply precise cases, researchers reproduce the mechanism, and independent cases assess the candidate. This proposed process would require contribution permissions and privacy handling before any user material entered a shared evaluation set.

A report becomes three objects

Imagine an illustrative report: “I asked for the conditions missing from the last entry, but the answer described an older configuration.” Preserve the submitted report as evidence of the complaint. Build a minimized reproduction containing the relevant configuration versions and decision references. Then define an expected relationship: the explanation must attribute the decision to the configuration active for that turn and distinguish later edits.

Those are three different objects. The original report preserves what the owner observed. The reproduction makes the error testable without unrelated account history. The expected relationship specifies correctness without requiring a particular sentence. If the record cannot establish which configuration was active, classify the case as unresolved and request the missing evidence through the appropriate contribution process.

The admission decision should name the case's owner, permitted uses, and expiry or withdrawal treatment. A contribution allowed for support diagnosis may need separate permission for shared research or model training. Developers should receive the smallest authorized reproduction, while evaluators retain enough provenance to reconstruct its expected answer.

Keep discovery separate from assessment

In an illustrative batch of 30 reports, suppose 18 become reproducible cases, seven remain ambiguous, and five are duplicates of an existing failure. These figures are a planning example. They produce 18 admitted reproductions, not 30 independent tests. Group the duplicate reports under their mechanism and retain the admission reasons.

Developers can use the admitted cases to design a fix. A separate evaluator then constructs unseen instances of the same relationship, using different configuration changes and decision times. A release decision should require improvement on those independent instances, acceptable behavior on unaffected explanation tasks, and preserved owner approval requirements. The original discovery cases remain visible as reproduction evidence, with their exposure to developers clearly marked.

For a candidate that answers every old report correctly but misattributes newer turns, the useful conclusion is that reproduction succeeded and generalization failed. That finding still improves the diagnosis. It should keep the candidate in development rather than turning the support queue into a success denominator.

This process gives user support a concrete role in research. Precise contributions increase the range of mechanisms we can test. Their value comes from attributable, permitted cases and independently verified repairs. A growing volume of feedback becomes an engineering asset only when the release record preserves that distinction.