This tutorial describes how to use data modules to stream real-time and historic market data using the command line tool and APIs.
The AWTS market data modules provide a number of key features over accessing vendor APIs directly; including automatically handling rate limits, size limits, disconnections and transitioning between historic and real-time data.
The CLI tool wraps the same APIs you can use programmatically in your own scripts, automations and algorithms, so serves as a good starting point for familiarisation and debugging. This tutorial will begin with CLI access to data, and proceed on to code using the AWTS APIs to perform the same functions.
For this exercise, you will need a running AWTS environment with at least one data module. You will also need the CLI tool installed.
If you don't have a running environment already, sign in to https://account.alphaweighted.com and go to the Configurator to get set up. Other tutorials such as "[Get started with Oanda data](/docs/quickstart/install-oanda/" include instructions on environment setup.
Here we will use Oanda data, via the dataoanda module, but the process is identical for any data vendor.
First lets search for SPX:
❯ awts data search SPX
+------------------+------------+
| SYMBOL | NAME |
+------------------+------------+
| OANDA:SPX500_USD | US SPX 500 |
+------------------+------------+
We see one matching instrument, with symbol OANDA:SPX500_USD
. Symbols in AWTS, by default, follow the form PREFIX:TICKER[:DERIVATIVE]
where the PREFIX
part denotes the data module providing the data. see Symbology for more information about how AWTS handles symbols.
Streaming in real time is easy; just use the streamcandles
command and specify the instrument. Streaming will start immediately and new price data will be printed to stdout
.
❯ awts data streamcandles OANDA:SPX500_USD --resolution=S5
Timestamp Vol O H L C Vwap Spread
2024-05-10T12:51:10Z 1 5227.20 5227.20 5227.20 5227.20 5227.20 0.50
2024-05-10T12:51:15Z 0 5227.20 5227.20 5227.20 5227.20 5227.20 0.50
2024-05-10T12:51:20Z 1 5226.80 5226.80 5226.80 5226.80 5226.80 0.50
2024-05-10T12:51:25Z 1 5227.20 5227.20 5227.20 5227.20 5227.20 0.50
This is broadly similar behaviour to hitting the data vendor's streaming endpoints directly, but with a few improvements:
The optional --from
and --to
arguments specify when streamed data should start from and finish at. They accept ISO-standard timestamps of the form yyyy-mm-ddThh:mm:ss
with an optional timezone. If timezone is omitted, the CLI tool will assume your local time.
--from
can be omitted or in the past, and all data from this point onwards will be immediately streamed when requested.
If --to
is omitted, then the stream will continue indefinitely (Ctrl+C to cancel!), but if specified then the streaming will end at that point.
If --to
is in the future, then streaming will run in real time until that time is reached. If --to
is in the past then historic data up to that point will be streamed immediately.
Using historic dates with --from
and --to
can be a useful way of exporting millions of records without consuming much RAM or system resources on your workstation.
An example large export of M1 data for 2020-2023 inclusive to CSV format:
❯ awts data streamcandles OANDA:SPX500_USD --format=csv \
--from=2020-01-01T00:00:00 --to=2024-01-01T00:00:00 > spx.csv
The CLI tool can be a part of your scripts and automations - e.g. use the --format
argument to have the CLI tool emit CSV or JSON data instead, and pipe this into your scripts.
❯ awts data streamcandles OANDA:SPX500_USD --resolution=S5 --format=csv | ./myscript
JSON formatted data in the streams is emitted as one record/candle per line, so your scripts can read a line at a time and unmarshal/unserialize accordingly.
Coming soon!