logo alphaweighted

How to stream market data

This tutorial describes how to use data modules to stream real-time and historic market data using the command line tool and APIs.

The AWTS market data modules provide a number of key features over accessing vendor APIs directly; including automatically handling rate limits, size limits, disconnections and transitioning between historic and real-time data.

The CLI tool wraps the same APIs you can use programmatically in your own scripts, automations and algorithms, so serves as a good starting point for familiarisation and debugging. This tutorial will begin with CLI access to data, and proceed on to code using the AWTS APIs to perform the same functions.

You will need

For this exercise, you will need a running AWTS environment with at least one data module. You will also need the CLI tool installed.

If you don't have a running environment already, sign in to https://account.alphaweighted.com and go to the Configurator to get set up. Other tutorials such as "[Get started with Oanda data](/docs/quickstart/install-oanda/" include instructions on environment setup.

Here we will use Oanda data, via the dataoanda module, but the process is identical for any data vendor.

Part 1: CLI

Finding the instrument

First lets search for SPX:

❯ awts data search SPX
+------------------+------------+
| SYMBOL           | NAME       |
+------------------+------------+
| OANDA:SPX500_USD | US SPX 500 |
+------------------+------------+

We see one matching instrument, with symbol OANDA:SPX500_USD. Symbols in AWTS, by default, follow the form PREFIX:TICKER[:DERIVATIVE] where the PREFIX part denotes the data module providing the data. see Symbology for more information about how AWTS handles symbols.

Real-time streaming

Streaming in real time is easy; just use the streamcandles command and specify the instrument. Streaming will start immediately and new price data will be printed to stdout.

❯ awts data streamcandles OANDA:SPX500_USD --resolution=S5
                Timestamp         Vol         O         H         L         C      Vwap    Spread
     2024-05-10T12:51:10Z           1   5227.20   5227.20   5227.20   5227.20   5227.20      0.50
     2024-05-10T12:51:15Z           0   5227.20   5227.20   5227.20   5227.20   5227.20      0.50
     2024-05-10T12:51:20Z           1   5226.80   5226.80   5226.80   5226.80   5226.80      0.50
     2024-05-10T12:51:25Z           1   5227.20   5227.20   5227.20   5227.20   5227.20      0.50

This is broadly similar behaviour to hitting the data vendor's streaming endpoints directly, but with a few improvements:

  • the AWTS module will automatically handle disconnect/reconnects
  • if the upstream vendor does not support streaming, AWTS will poll/handle as best as is possible and present the same consistent streaming API to you
  • historic data can precede the real-time data automatically; i.e. there's no need to manually start streaming and buffer, fetch recent data, absorb streamed records, remove duplicates; AWTS handles all that

Historic data streaming

The optional --from and --to arguments specify when streamed data should start from and finish at. They accept ISO-standard timestamps of the form yyyy-mm-ddThh:mm:ss with an optional timezone. If timezone is omitted, the CLI tool will assume your local time.

--from can be omitted or in the past, and all data from this point onwards will be immediately streamed when requested.

If --to is omitted, then the stream will continue indefinitely (Ctrl+C to cancel!), but if specified then the streaming will end at that point.

If --to is in the future, then streaming will run in real time until that time is reached. If --to is in the past then historic data up to that point will be streamed immediately.

Using historic dates with --from and --to can be a useful way of exporting millions of records without consuming much RAM or system resources on your workstation.

An example large export of M1 data for 2020-2023 inclusive to CSV format:

❯ awts data streamcandles OANDA:SPX500_USD --format=csv \
 --from=2020-01-01T00:00:00 --to=2024-01-01T00:00:00 > spx.csv

Automation

The CLI tool can be a part of your scripts and automations - e.g. use the --format argument to have the CLI tool emit CSV or JSON data instead, and pipe this into your scripts.

❯ awts data streamcandles OANDA:SPX500_USD --resolution=S5 --format=csv | ./myscript

JSON formatted data in the streams is emitted as one record/candle per line, so your scripts can read a line at a time and unmarshal/unserialize accordingly.

Part 2: API

Coming soon!