Handle Duplicate Records at Historical API Page Boundaries
By DX Research Group · · Market data
An inclusive-boundary fixture preserves equal-time events while preventing repeated rows from inflating a market feature.
Deduplicate historical pages using the source's record identity, and preserve all distinct records that share the boundary timestamp. Advancing a time cursor by one unit can erase valid equal-time events; blindly appending an inclusive next page can count the boundary twice. We would treat pagination as a documented acquisition contract with completeness checks.
Hyperliquid's info documentation directs time-range pagination to use the last returned timestamp. Coinbase uses response-header cursors. These mechanisms need different collectors, rather than one assumed page-number abstraction.
Five returned rows contain four events
Assume a synthetic time-range API with inclusive start boundaries, stable event IDs, and a page limit of three. This fixture states its assumptions explicitly; endpoint-specific guarantees must be verified before applying the procedure.
| Page | Event IDs and times |
|---|---|
| First | A at 1000, B at 1001, C at 1002 |
| Second, start 1002 | C at 1002, D at 1002 |
Appending pages yields five rows but only four events. Dropping every row at timestamp 1002 from the second page preserves C and loses D. Advancing the start to 1003 also loses D. Deduplicating by stable ID retains A, B, C, and D exactly once.
If every event has quantity 2, the correct total is 8. Raw append reports 10. Timestamp-wide removal reports 6. This arithmetic makes both failure directions visible: duplication inflates the feature, while boundary skipping removes real activity.
Equal-time saturation needs a recovery path
A harder case occurs when more events share one timestamp than the page limit. A timestamp-only cursor may repeatedly return the same prefix. Detect stalled progress by comparing the event-ID set and cursor across pages. Completion then requires another supported cursor, narrower endpoint, documented source guarantee, or an explicit incomplete interval.
An empty response and an exhausted page boundary have different meanings from a failed request. Preserve the request parameters, response headers, page count, and terminal reason. Retries should be idempotent at the event level, allowing repeated acquisition without repeated economic records.
For records that can change after publication, identical IDs with different payloads require a version policy. Keep the first observation and the revision evidence rather than treating a conflicting payload as an ordinary duplicate. That distinction matters for news, funding, and settlement histories.
Audit the boundary, then compute features
A proposed collector test would replay the two pages above, duplicate the second response, and add enough equal-time records to stall the cursor. Expected outputs are a total quantity of 8 in the simple case and a declared incomplete interval in the saturation case.
Our event-sampling note explains how repeated observations distort features. Our funding-accounting note shows why economic records need independent identity. This contribution is a boundary-specific acquisition check. It can establish fixture completeness and idempotency, while full historical coverage requires the source's pagination guarantees and saved terminal receipts.