How Does a Tick-Level Data API Handle Out-of-Order Data? From Timestamp Sorting to Duplicate Record Cleanup

When building a real-time market data system with a Tick-level data API, successfully receiving data does not necessarily mean that the processing results are correct. If an older market update arrives after a newer one, it may overwrite the latest price. Duplicate records received after a WebSocket reconnection can also affect volume calculations and subsequent processing.

For financial technology developers, the key is not simply to sort the data. It is to distinguish event time from receive time and combine event identifiers, a reordering buffer, and late-data handling rules to process anomalies appropriately.

Why Does Tick Data Arrive Out of Order?

Tick data is usually streamed continuously over WebSocket. Network latency, message queue backlogs, and multithreaded processing can cause the arrival order to differ from the order of event timestamps.

For example, suppose the system receives a market update with a timestamp of 10:00:00.120, followed by another update timestamped 10:00:00.110. If the application updates the latest price every time a message arrives, the later-arriving record may cause the displayed price to revert to an older value.

Two concepts need to be distinguished:

• Event time: The timestamp associated with the market event.
• Receive time: The time when the client receives the market data record.

The following is simulated data used to illustrate out-of-order delivery. It does not represent actual market prices.

The third record has an earlier event time than the second, so it illustrates out-of-order arrival. The fourth record shares the same example sequence number and price as the second. It can be used to demonstrate how a duplicate event might be identified.

In a real system, records should not be deleted solely because their prices and timestamps match. Different market events can have the same price and may even share the same timestamp. When determining whether a record is a duplicate, prioritize event identifiers or sequence numbers provided by the data source.

How to Sort by Timestamp and Deduplicate Using Sequence Numbers

One basic data correction method is to collect a batch of received Tick records and sort them by a normalized event timestamp.

AllTick's real-time transaction quote subscription documentation lists streaming fields such as code, seq, tick_time, price, and volume. In this context, seq represents the quote sequence number, while tick_time represents the market data timestamp. This subscription provides the latest real-time transaction price; it is not the same as an interface for querying historical tick-by-tick transactions.

The following simplified Python example uses these fields. It assumes that timestamps have already been normalized to comparable integers. Deduplication by sequence number is enabled only when it has been confirmed that seq can reliably identify the same record within the relevant code scope.

def clean_ticks(ticks, seq_is_unique_per_code=False):
    seen = set()
    accepted = []
    rejected = 0

    for index, tick in enumerate(ticks):
        code = tick.get("code")
        tick_time = tick.get("tick_time")
        seq = tick.get("seq")

        if not code or tick_time is None:
            rejected += 1
            continue

        try:
            event_time = int(tick_time)
        except (TypeError, ValueError):
            rejected += 1
            continue

        # Deduplicate only after sequence-number uniqueness is confirmed
        if seq_is_unique_per_code and seq is not None:
            key = (code, str(seq))
            if key in seen:
                continue
            seen.add(key)

        # Use the sequence number to help order records with the same timestamp
        try:
            seq_key = (0, int(seq))
        except (TypeError, ValueError):
            seq_key = (1, str(seq or ""))

        accepted.append(
            (event_time, seq_key, index, tick)
        )

    accepted.sort(key=lambda item: (
        item[0], item[1], item[2]
    ))

    return [item[3] for item in accepted], rejected

This code performs three tasks: validates required fields, conditionally removes duplicates, and sorts records by event time and sequence number. The original arrival order is retained when all sorting keys are identical.

Two additional considerations matter when integrating the code. First, if seq can be reused or cannot reliably identify the same event after reconnection, (code, seq) should not be used directly as the deduplication key. Second, if timestamp units are inconsistent, they must be normalized first. Otherwise, sorting the values numerically may not produce the correct chronological order.

Deduplication is disabled by default in this example. Set seq_is_unique_per_code to True only after confirming the meaning and uniqueness rules of the sequence number in the actual data. The example handles a single batch and does not implement state management across batches. It is not a complete production-ready processing system.

How Should Late-Arriving Data Be Handled in Real Time?

Batch sorting can correct the order of data that has already been collected. In a real-time system receiving a continuous WebSocket stream, however, the application must also decide when to release data for downstream processing and how to handle records that arrive after an allowed delay.

A common approach is to maintain a bounded reordering buffer and use event time to establish a watermark. For example, the system can subtract an allowed lateness interval from the maximum event timestamp observed so far. Records earlier than the resulting watermark can enter the normal processing flow. Older records arriving after the watermark has advanced are handled according to a separate late-data policy.

The allowed lateness interval should be determined by actual data arrival patterns and business requirements. A single fixed value is not necessarily appropriate for every instrument.

For real-time price displays, a late record whose event time precedes the currently published state should not directly overwrite the latest price. For historical statistics or candlestick reconstruction, late records can be stored and used to correct affected aggregates when necessary. Because these scenarios have different objectives, they should not share a simple overwrite rule.

Deduplication must also account for batch boundaries. Keeping processed sequence numbers only within a single function call will not identify records that reappear in a later batch or after a service restart. Systems requiring ongoing deduplication should persist the necessary state according to the data source's identification rules and check data continuity after reconnecting.

What Should You Verify Before Integrating AllTick?

When building a Tick data processing module with AllTick, first inspect the real-time subscription data structure for the target instrument. Then design the sorting, deduplication, and late-data handling logic accordingly. Do not infer field semantics from names alone or assume that every instrument provides the same event identification capabilities.

In particular, verify the actual timestamp unit and the scope within which the sequence number is unique. AllTick's official real-time transaction quote subscription documentation labels tick_time as milliseconds, but the page's example displays a 10-digit timestamp. This discrepancy requires further confirmation. During integration, rely on actual responses and confirmed interface specifications rather than converting the value to seconds or milliseconds without verification.

If you are developing a real-time market data service, start at the AllTick official website and navigate to the relevant product and documentation pages to confirm the streaming fields for your target instrument. Then validate the processing logic with test data containing out-of-order records, duplicates, and late-arriving updates.

For a Tick-level data API, timestamp sorting addresses event order, sequence-based deduplication helps identify events, and late-data rules determine how the system handles records that do not arrive when expected. Designing these three components separately helps reduce the risks of older prices overwriting newer states, duplicate aggregation, and inconsistencies in backtesting data.