Does support improve understanding or merely bring owners back?

By DX Research Group · · Incentives and participation

A support experiment separates immediate comprehension from voluntary review retention over four weeks.

Support can change what an owner understands and whether they return. Those effects can move in different directions. We would test both directly before treating a higher return rate as evidence that people learned how to supervise their agents.

DXAP's activity guide distinguishes a completed turn, a submitted order and a recorded fill. Its chat guide explains how proposed changes become active through owner confirmation. These boundaries make comprehension measurable through scenarios with answers grounded in the product workflow.

Give the same question two forms of help

A proposed study recruits 120 consenting owners after their first Activity review. Randomize 60 to a ten-minute guided explanation and 60 to an equally timed walkthrough of navigation. Both groups retain ordinary help and access. The guided explanation covers a turn that waits, an order awaiting a fill and a settings proposal awaiting confirmation. The navigation walkthrough shows where those records live without teaching the interpretation rule.

Measure comprehension immediately and again seven days later using different examples. Score whether the owner correctly distinguishes the three states and chooses the appropriate next observation. The seven-day assessment tests retained understanding rather than recall of the support script. The reviewer sees answers without knowing assignment.

Define retention separately: at least one voluntary, qualifying Activity review during days 15 through 28. A qualifying review requires an owner to identify the turn they inspected and answer one neutral question about its state. Reminder-driven assessment visits belong to a separate count. The denominator remains all 60 assigned owners per arm.

Consider illustrative counts: 48 of 60 guided owners pass the immediate test versus 30 of 60 navigation owners, an increase of 30 percentage points. At day seven, 39 versus 27 pass, a 20-point difference. Later voluntary review counts are 24 versus 23, only a 1.7-point difference. This pattern would support a comprehension effect while leaving a durable participation effect uncertain. These figures describe the calculation, rather than an observed DXAP study.

A reversed pattern also matters. If review returns rise while scenario accuracy remains unchanged, support may be providing social contact or a reason to reopen the product. We would then vary contact frequency separately from explanatory content in a subsequent study.

Preserve agency in the outcome

A knowledgeable owner may pause after discovering a problem. Record voluntary pause and concern reporting alongside review returns, so informed restraint remains visible. Neither live trading persistence nor account balance should substitute for the comprehension score.

Keep support staff time and unscheduled follow-up in the study ledger. Differences in extra help can explain apparent content effects. Record assessment nonresponse explicitly and show both assigned-population results and observed-answer scores. The next product decision follows the mechanism: retain explanations that improve understanding, revise prompts that merely pull people back, and evaluate any resulting harness repair on independent cases.

Sources

Related field notes