How Can You Check for Missing Trading Days and Abnormal Candlesticks After Downloading Historical Financial Data?

After downloading historical financial data, being able to open the file does not mean the data is ready for analysis or backtesting. Some trading days may be missing, the same candlestick may have been downloaded more than once, or a record’s high may be lower than its open or close. If these issues are not identified in advance, they can affect technical indicator calculations, strategy signals, and backtest results.

For developers using a historical market data API, checking data quality involves more than counting records. The downloaded data should be compared against the expected trading calendar, timestamps, and basic OHLC price rules. Any suspicious records should be saved for further review.

First, Determine Which Trading Days Should Have Data

When checking for missing trading days, do not simply assume that data should be available every weekday. Stock markets may have holidays, trading suspensions, or unscheduled closures. Trading calendars also differ across markets. For forex, cryptocurrencies, and other financial instruments, the checks should follow the relevant trading schedules.

The first step is to prepare a list of expected trading dates for the relevant market and instrument. For daily candlesticks, comparing the dates in the downloaded data with the expected trading dates can reveal missing days. For intraday data, such as 1-minute or 5-minute candlesticks, you also need to generate the expected timestamps based on trading hours. Otherwise, missing candlesticks within an otherwise present trading day may go undetected.

When downloading historical financial data, pay attention to the API query range and the maximum number of records returned by a single request. If you need a longer historical period, retrieve the data in batches according to the relevant API rules and check for duplicates after combining the results. A successful request does not necessarily mean that the entire requested historical period has been retrieved.

Check Timestamps and OHLC Price Relationships

Time-related checks should focus on three main issues: timestamps that cannot be parsed, duplicate timestamps for the same instrument and interval, and data whose actual time range does not match the expected range. Timestamp units and time zones should also be standardized. Otherwise, the same candlestick might be interpreted as occurring at different times or assigned to the wrong trading date.

Price checks can use the basic relationships between OHLC fields. A valid candlestick should satisfy the following conditions:

• The high is greater than or equal to both the open and the close.
• The low is less than or equal to both the open and the close.
• The high is greater than or equal to the low.
• The open, high, low, and close are all valid numeric values.

A sudden, large price increase or decrease is also worth investigating, but it should not automatically be treated as a data error. Actual market movements, corporate actions, differences in price-adjustment methods, or missing data can all lead to price jumps. Mark suspicious records first, then compare them with the original data and the relevant market conditions.

Use Python to Detect Missing Trading Days and Abnormal Candlesticks

The following example uses pandas to check downloaded data for a single instrument and a single candlestick interval. It assumes the historical data has been organized in history_kline.csv with six fields: bar_time, trade_date, open, high, low, and close. The market_calendar.csv file is assumed to contain the expected trading dates for the relevant market.

The bar_time field should be standardized in advance as a timezone-aware timestamp or normalized UTC time, according to the format returned by the API. The trade_date field should be generated according to the trading-date rules of the relevant market.

import numpy as np
import pandas as pd

# Load the downloaded historical candlestick data
df = pd.read_csv("history_kline.csv")

required = {
    "bar_time", "trade_date",
    "open", "high", "low", "close"
}

missing_columns = required - set(df.columns)
if missing_columns:
    raise ValueError(
        f"Missing required fields: {sorted(missing_columns)}"
    )

# Assume bar_time has been standardized to a parseable time format
df["bar_time"] = pd.to_datetime(
    df["bar_time"], errors="coerce", utc=True
)

price_cols = ["open", "high", "low", "close"]

for col in price_cols:
    df[col] = pd.to_numeric(df[col], errors="coerce")

# 1. Check for invalid timestamps
invalid_time = df[df["bar_time"].isna()].copy()

# 2. Check for duplicate timestamps
valid_time = df[df["bar_time"].notna()]
duplicates = valid_time[
    valid_time.duplicated("bar_time", keep=False)
].copy()

# 3. Check for missing prices, non-finite values,
#    and abnormal OHLC relationships
price_array = df[price_cols].to_numpy(dtype=float)
bad_price = pd.Series(
    ~np.isfinite(price_array).all(axis=1),
    index=df.index
)

bad_ohlc = (
    (df["high"] < df[["open", "close"]].max(axis=1))
    | (df["low"] > df[["open", "close"]].min(axis=1))
    | (df["high"] < df["low"])
) & ~bad_price

invalid_prices = df[bad_price].copy()
abnormal_ohlc = df[bad_ohlc].copy()

# 4. Compare actual trading dates with the market calendar
calendar = pd.read_csv("market_calendar.csv")

if "trade_date" not in calendar.columns:
    raise ValueError(
        "The market calendar is missing the trade_date field"
    )

expected = pd.to_datetime(
    calendar["trade_date"], errors="coerce"
)

if expected.isna().any():
    raise ValueError(
        "The market calendar contains unparseable dates"
    )

actual = pd.to_datetime(
    df["trade_date"], errors="coerce"
)

expected_dates = set(expected.dt.strftime("%Y-%m-%d"))
actual_dates = set(actual.dt.strftime("%Y-%m-%d").dropna())

missing_dates = sorted(expected_dates - actual_dates)

# 5. Save the results for later review
invalid_time.to_csv("invalid_timestamps.csv", index=False)
duplicates.to_csv("duplicate_bars.csv", index=False)
invalid_prices.to_csv("invalid_prices.csv", index=False)
abnormal_ohlc.to_csv("abnormal_ohlc.csv", index=False)

pd.Series(
    missing_dates, name="missing_trade_date"
).to_csv("missing_trade_dates.csv", index=False)

print("Invalid timestamps:", len(invalid_time))
print("Duplicate candlestick records:", len(duplicates))
print("Invalid price records:", len(invalid_prices))
print("Abnormal OHLC records:", len(abnormal_ohlc))
print("Missing trading days:", len(missing_dates))

After running the script, you can review separate lists of invalid timestamps, duplicate records, invalid prices, abnormal OHLC records, and missing trading days. Instead of deleting all suspicious data indiscriminately, save the records by issue type so that each problem can be reviewed separately.

This example assumes the input fields have already been mapped correctly and is intended for a single instrument and a single interval. If the file contains multiple instruments or intervals, duplicate detection must use the instrument symbol, interval, and timestamp together.

The comparison with the trading calendar mainly identifies days for which no data is present. If a trading day contains some data but is missing several intraday candlesticks, you need to generate the expected timestamps based on the relevant trading hours and interval, then compare them with the actual records.

Do Not Automatically Fill Missing Data After Finding Anomalies

A missing trading day may correspond to a normal market closure, or it may indicate that the downloaded date range is incomplete. Duplicate records may result from overlapping time ranges in batch requests. Abnormal prices can also be caused by genuine market movements or data-processing issues. Review the anomaly lists alongside the original responses and market calendar to determine the cause.

In particular, avoid automatically filling every missing candlestick with the previous candle’s closing price. This can create artificial records with no price movement and change moving averages, volatility calculations, and strategy signals. If data filling is required for a specific business purpose, establish a separate rule, preserve the original data, and clearly mark the records that have been filled.

When using AllTick to retrieve historical market data, visit the AllTick developer portal to find the relevant API documentation and verify the instrument, candlestick interval, query range, and retrieval limits. Keeping historical data retrieval, trading-calendar validation, and anomaly review as separate processes makes it easier to prepare downloaded data for analysis and backtesting.