Memory Retention and Operational Observability

By DX Research Group · · State and memory

Choose retention by the incidents you must reconstruct and the information each record exposes.

Retention should be designed around specific diagnostic questions and authorized information handling. We would preserve enough evidence to reconstruct important failures while measuring the storage and exposure cost of each record class. Keeping everything forever and deleting everything quickly are both poor substitutes for that decision.

The linked records in our state and memory framework establish what a decision used. The trace feedback framework explains which failures those records can distinguish. This proposed worksheet makes the retention tradeoff concrete without prescribing a universal duration.

Three record classes, different costs

Assume an illustrative runtime creates 1,000 turns each day. Each raw tool bundle uses 20 kilobytes, each rendered context 10 kilobytes, and each compact trace receipt 2 kilobytes. Ignoring compression and storage overhead, retaining all three for 30 days uses 960,000 kilobytes: 32 per turn times 1,000 times 30.

A hypothetical tiered policy retains raw bundles and contexts for seven days, while retaining receipts for 30. Its volume is 270,000 kilobytes: 30 times 1,000 times 7, plus 2 times 1,000 times 30. The reduction is 690,000 kilobytes, or 71.875% of the full-retention volume. These are decimal units and hypothetical record sizes, rather than measured production costs.

The smaller footprint has a diagnostic price. After day seven, a receipt may show that a tool result was used without preserving the full payload needed to investigate an omitted qualifier. A payload hash can verify a supplied candidate artifact, but it cannot reconstruct missing content. The worksheet must state that loss plainly.

Reconstruct incidents at the expiry boundary

We would define incident classes before comparing policies. Duplicate economic-event processing may require identifiers and reducer boundaries. A misleading summary may require original source text and the compressed output. An incorrect policy decision may require exact input values and policy version. Different classes can therefore have different minimal records.

The fixture evaluates reconstruction at day six and day eight under the tiered policy. Each incident gets a result: fully reconstructible, partially reconstructible with named missing evidence, or unavailable. The aggregate coverage should retain the incident-class denominators. An overall percentage weighted toward easy identifier checks could conceal the loss of harder source-content investigations.

Storage volume and diagnostic coverage remain separate axes. We would also document access restrictions and deletion requirements for each artifact class. Better debugging capability does not create permission to retain sensitive content longer. A retained receipt should contain only the fields justified by its diagnostic function.

These calculations describe an engineering proposal and provide no current DXAP retention guarantee. The practical decision is whether the reduced volume is worth the specific investigations lost after expiry. A retention review should end with one incident that remains reconstructible and one that becomes weaker, alongside their dates. That comparison makes the cost visible to both operators and owners before a duration becomes an unnoticed default.

Sources

Related field notes