logo alphaweighted

Streaming, latency & mutability

Streaming market data is essential to any real time financial system. AWTS supports live streaming of candle (aggregate) data on multiple timeframes, and seamless streaming of historic data through to live - to make implementation of your apps and strategies as simple as possible.

However there are many complexities involved in real-time streaming, which vary by vendor/broker, and not all of which can be abstracted away by the Data services modules.

Specifically:

  1. Historic data can change, e.g. as a result of stock splits
  2. Intraday data can change, e.g. because of late reporting from alternate exchanges/dark pools
  3. There is a lag between the end of the time period for a candle and when that candle becomes available (i.e. when it "closes"), and the mechanics for this differ between markets and data vendors

Realistically, at hourly timeframes and higher these complexities are unlikely to be material to your operations. On lower timeframes though, the effect can be significant.

The AWTS approach

AWTS attempts to smooth the data streaming process as much as possible so that you, developing strategies, need not write additional boilerplate to cover edge cases.

This means AWTS data modules will (where possible):

  • simulate latency when streaming historic data
  • include data updates/corrections when streaming
  • transparency synchronise data streams across vendors
  • automatically handle disconnects/reconnects

However, the characteristics of each data vendor cannot always be 100% handled transparently; please read carefully the documentation for each data module you use; any exceptions or special considerations are noted there.

What's the impact?

Speed vs accuracy

Delivering candles quickly means they are subject to correction. Delivering candles slowly means they can be more correct (sic) - but the delay may have other impacts (e.g. in fills when trading from the data).

Historic data diverges

Backtesting on historic data is not subject to the same delays, lags and corrections. Essentially the historic data is perfect and already-corrected.

What to do?

Realistically the practical impact should be small or negligible, especially at higher timeframes; as a proportion of overall volume the cost of a few seconds, vs the difference of a few missing trades should be minimal.

However at lower timeframes (e.g. M5 or lower),

We recommend you review correction logs with the Logger module to make sure that data updates/corrections are indeed immaterial vs your strategies.

Latency - or "when does a candle close"?

Consider a 1 minute candle starting at 12:00:00 and ending at 12:01:00. It is one minute long. So the obvious expectation would be for it to 'close' immediately after the end of that minute.

The first latency is the one from you to your broker; i.e. it may take a few hundred ms for that closed candle to reach you. This is fine.

The more complex latency is inside your broker and upstream from your broker (to exchanges / liquidity providers). i.e. they may not provide trade data until sometime after the trade happened. E.g. if a trades happens at 12:00:59 then it should be in the M1 candle - but if your broker is not aware of this until 12:01:01 then that is clearly impossible.

This effect, depending on the vendor, results in data corrections and/or 'late' delivery of candles.

For example:

  • Oanda is a CFD provider, the underlying "exchange" are its liquidity providing partners. Oanda considers a candle "closed" sometime after the end point in time; which appears to be when a tick/trade happens internally that falls into the next candle period'. So at high-volume times, candles can close quickly (in a few hundred ms), at low volume times it can take a few seconds or more.

  • Polygon.io is a data vendor for US equities (and other markets). Streaming aggregates/candles are artificially delayed by ~2 seconds to allow for latency/delays in trades becoming visible to them. However 'closed' candles may be updated up to 15 minutes later as any other late reports arrive to them. Additionally, intraday data may be revised EOD as well.