TechQA.

Bitstamp API returns wrong number of klines when asking latest OHLC

691 views Asked by At

Problem

Working on Bitstamp HTTP API, I found something very odd.
Whenever I ask for the latest OHLC data using the following request:

https://www.bitstamp.net/api/v2/ohlc/{currency_pair}/

required parameters are written in the documentation page for Bitstamp.

I get only 1 data point. Which is wrong because the limit parameter is set to 2. Things get worth as I go further and send the following request again and again. I found this only happens at the first 40~45 seconds when the candle closes for any step (interval) I use.

Example

When the step parameter is set to 60, the server must return a 1-minute-candles and passing the number 2 for limit, for the first 40~45 seconds the response contains 1 data point:

{"data": {"pair": "BTC/USD", "ohlc": [{"high": "19131.67", "timestamp": "1607194800", "volume": "0.00000000", "low": "19131.67", "close": "19131.67", "open": "19131.67"}]}}

but as time goes on and we pass the first 40~45s the response contains 2 data points:

{"data": {"pair": "BTC/USD", "ohlc": [{"high": "19127.87", "timestamp": "1607194860", "volume": "0.49121478", "low": "19104.91", "close": "19104.91", "open": "19127.87"}, {"high": "19111.41", "timestamp": "1607194920", "volume": "0.09581116", "low": "19104.45", "close": "19111.41", "open": "19104.67"}]}}

and also the same story happens for other valid values for step. Setting the step to 300 which must contain 5-minute-candles, the responses are not valid when requesting in the first 40~45s of any 5-minute intervals (for example 12:00-UTC, 12:05-UTC, and so on).

No matter what the parameter step is set to any valid number, the server returns the wrong data at the first 40~45 seconds of starting any time frame.

I have also tried passing the start and end optional parameters.

I have worked with other exchange and broker's API and I have faced no error or wrong responses so far, except using Bitstamp HTTP API.

Note: the value 40~45 seconds is not accurate because I've not been able to measure it accurately.

Code sample in python and the log

Code in python

import logging
import requests

from datetime import datetime
from time import sleep


base_url_v2 = "https://www.bitstamp.net/api/v2/"
logging.basicConfig(filename='app.log',
                    filemode='w',
                    format='[%(asctime)s UTC][%(name)s][%(levelname)s]'
                           ':%(message)s',
                    level=logging.DEBUG,
                    datefmt='%M/%d/%y %H:%M:%S')


def get_ohlc_data(pair: str, step, limit):
    url = f"ohlc/{pair}/"
    request_url = base_url_v2 + url

    params = {
        "step": step,
        "limit": limit
    }
    try:
        logging.info("Sending request...")
        response = requests.get(request_url, params=params)
        logging.debug(f"Request response: {response.text}")

    except Exception as exp:
        logging.error(exp)


def main():
    logging.info("Initialized.")

    # Wait until xx:xx:00 UTC
    while int(datetime.now().strftime("%S")) != 0:
        sleep(0.5)

    # request OHLC data every 5 seconds
    for r in range(12):

        # Wait until xx:xx:x5[or 0] UTC
        while int(datetime.now().strftime("%S")) % 5 != 0:
            sleep(1)

        get_ohlc_data(pair="btcusd",
                      step=60,
                      limit=2)

        sleep(1)

    logging.info("Ended.")


main()

log file

[04/16/20 18:04:16 UTC][root][INFO]:Initialized.
[05/16/20 18:05:00 UTC][root][INFO]:Sending request...
[05/16/20 18:05:00 UTC][urllib3.connectionpool][DEBUG]:Starting new HTTPS connection (1): www.bitstamp.net:443
[05/16/20 18:05:00 UTC][urllib3.connectionpool][DEBUG]:https://www.bitstamp.net:443 "GET /api/v2/ohlc/btcusd/?step=60&limit=2 HTTP/1.1" 200 None
[05/16/20 18:05:00 UTC][root][DEBUG]:Request response: {"data": {"pair": "BTC/USD", "ohlc": [{"high": "20690.59", "timestamp": "1608141840", "volume": "13.32120988", "low": "20677.18", "close": "20677.18", "open": "20690.59"}]}}
[05/16/20 18:05:05 UTC][root][INFO]:Sending request...
[05/16/20 18:05:05 UTC][urllib3.connectionpool][DEBUG]:Starting new HTTPS connection (1): www.bitstamp.net:443
[05/16/20 18:05:06 UTC][urllib3.connectionpool][DEBUG]:https://www.bitstamp.net:443 "GET /api/v2/ohlc/btcusd/?step=60&limit=2 HTTP/1.1" 200 None
[05/16/20 18:05:06 UTC][root][DEBUG]:Request response: {"data": {"pair": "BTC/USD", "ohlc": [{"high": "20681.91", "timestamp": "1608141900", "volume": "0.02455454", "low": "20681.91", "close": "20681.91", "open": "20681.91"}]}}
[05/16/20 18:05:10 UTC][root][INFO]:Sending request...
[05/16/20 18:05:10 UTC][urllib3.connectionpool][DEBUG]:Starting new HTTPS connection (1): www.bitstamp.net:443
[05/16/20 18:05:10 UTC][urllib3.connectionpool][DEBUG]:https://www.bitstamp.net:443 "GET /api/v2/ohlc/btcusd/?step=60&limit=2 HTTP/1.1" 200 None
[05/16/20 18:05:10 UTC][root][DEBUG]:Request response: {"data": {"pair": "BTC/USD", "ohlc": [{"high": "20681.91", "timestamp": "1608141900", "volume": "0.30755454", "low": "20671.52", "close": "20671.52", "open": "20681.91"}]}}
[05/16/20 18:05:15 UTC][root][INFO]:Sending request...
[05/16/20 18:05:15 UTC][urllib3.connectionpool][DEBUG]:Starting new HTTPS connection (1): www.bitstamp.net:443
[05/16/20 18:05:16 UTC][urllib3.connectionpool][DEBUG]:https://www.bitstamp.net:443 "GET /api/v2/ohlc/btcusd/?step=60&limit=2 HTTP/1.1" 200 None
[05/16/20 18:05:16 UTC][root][DEBUG]:Request response: {"data": {"pair": "BTC/USD", "ohlc": [{"high": "20681.91", "timestamp": "1608141900", "volume": "1.43352856", "low": "20671.52", "close": "20674.94", "open": "20681.91"}]}}
[05/16/20 18:05:20 UTC][root][INFO]:Sending request...
[05/16/20 18:05:20 UTC][urllib3.connectionpool][DEBUG]:Starting new HTTPS connection (1): www.bitstamp.net:443
[05/16/20 18:05:20 UTC][urllib3.connectionpool][DEBUG]:https://www.bitstamp.net:443 "GET /api/v2/ohlc/btcusd/?step=60&limit=2 HTTP/1.1" 200 None
[05/16/20 18:05:20 UTC][root][DEBUG]:Request response: {"data": {"pair": "BTC/USD", "ohlc": [{"high": "20681.91", "timestamp": "1608141900", "volume": "1.43352856", "low": "20671.52", "close": "20674.94", "open": "20681.91"}]}}
[05/16/20 18:05:25 UTC][root][INFO]:Sending request...
[05/16/20 18:05:25 UTC][urllib3.connectionpool][DEBUG]:Starting new HTTPS connection (1): www.bitstamp.net:443
[05/16/20 18:05:25 UTC][urllib3.connectionpool][DEBUG]:https://www.bitstamp.net:443 "GET /api/v2/ohlc/btcusd/?step=60&limit=2 HTTP/1.1" 200 None
[05/16/20 18:05:25 UTC][root][DEBUG]:Request response: {"data": {"pair": "BTC/USD", "ohlc": [{"high": "20681.91", "timestamp": "1608141900", "volume": "1.43352856", "low": "20671.52", "close": "20674.94", "open": "20681.91"}]}}
[05/16/20 18:05:30 UTC][root][INFO]:Sending request...
[05/16/20 18:05:30 UTC][urllib3.connectionpool][DEBUG]:Starting new HTTPS connection (1): www.bitstamp.net:443
[05/16/20 18:05:31 UTC][urllib3.connectionpool][DEBUG]:https://www.bitstamp.net:443 "GET /api/v2/ohlc/btcusd/?step=60&limit=2 HTTP/1.1" 200 None
[05/16/20 18:05:31 UTC][root][DEBUG]:Request response: {"data": {"pair": "BTC/USD", "ohlc": [{"high": "20687.00", "timestamp": "1608141900", "volume": "1.65890659", "low": "20671.52", "close": "20676.56", "open": "20681.91"}]}}
[05/16/20 18:05:35 UTC][root][INFO]:Sending request...
[05/16/20 18:05:35 UTC][urllib3.connectionpool][DEBUG]:Starting new HTTPS connection (1): www.bitstamp.net:443
[05/16/20 18:05:36 UTC][urllib3.connectionpool][DEBUG]:https://www.bitstamp.net:443 "GET /api/v2/ohlc/btcusd/?step=60&limit=2 HTTP/1.1" 200 None
[05/16/20 18:05:36 UTC][root][DEBUG]:Request response: {"data": {"pair": "BTC/USD", "ohlc": [{"high": "20687.00", "timestamp": "1608141900", "volume": "1.65890659", "low": "20671.52", "close": "20676.56", "open": "20681.91"}]}}
[05/16/20 18:05:40 UTC][root][INFO]:Sending request...
[05/16/20 18:05:40 UTC][urllib3.connectionpool][DEBUG]:Starting new HTTPS connection (1): www.bitstamp.net:443
[05/16/20 18:05:40 UTC][urllib3.connectionpool][DEBUG]:https://www.bitstamp.net:443 "GET /api/v2/ohlc/btcusd/?step=60&limit=2 HTTP/1.1" 200 None
[05/16/20 18:05:40 UTC][root][DEBUG]:Request response: {"data": {"pair": "BTC/USD", "ohlc": [{"high": "20687.00", "timestamp": "1608141900", "volume": "2.55967640", "low": "20671.52", "close": "20675.47", "open": "20681.91"}]}}
[05/16/20 18:05:45 UTC][root][INFO]:Sending request...
[05/16/20 18:05:45 UTC][urllib3.connectionpool][DEBUG]:Starting new HTTPS connection (1): www.bitstamp.net:443
[05/16/20 18:05:45 UTC][urllib3.connectionpool][DEBUG]:https://www.bitstamp.net:443 "GET /api/v2/ohlc/btcusd/?step=60&limit=2 HTTP/1.1" 200 None
[05/16/20 18:05:45 UTC][root][DEBUG]:Request response: {"data": {"pair": "BTC/USD", "ohlc": [{"high": "20690.59", "timestamp": "1608141840", "volume": "13.32120988", "low": "20677.18", "close": "20677.18", "open": "20690.59"}, {"high": "20687.00", "timestamp": "1608141900", "volume": "3.10476012", "low": "20671.52", "close": "20686.60", "open": "20681.91"}]}}
[05/16/20 18:05:50 UTC][root][INFO]:Sending request...
[05/16/20 18:05:50 UTC][urllib3.connectionpool][DEBUG]:Starting new HTTPS connection (1): www.bitstamp.net:443
[05/16/20 18:05:51 UTC][urllib3.connectionpool][DEBUG]:https://www.bitstamp.net:443 "GET /api/v2/ohlc/btcusd/?step=60&limit=2 HTTP/1.1" 200 None
[05/16/20 18:05:51 UTC][root][DEBUG]:Request response: {"data": {"pair": "BTC/USD", "ohlc": [{"high": "20690.59", "timestamp": "1608141840", "volume": "13.32120988", "low": "20677.18", "close": "20677.18", "open": "20690.59"}, {"high": "20687.00", "timestamp": "1608141900", "volume": "3.55776012", "low": "20671.52", "close": "20686.54", "open": "20681.91"}]}}
[05/16/20 18:05:55 UTC][root][INFO]:Sending request...
[05/16/20 18:05:55 UTC][urllib3.connectionpool][DEBUG]:Starting new HTTPS connection (1): www.bitstamp.net:443
[05/16/20 18:05:55 UTC][urllib3.connectionpool][DEBUG]:https://www.bitstamp.net:443 "GET /api/v2/ohlc/btcusd/?step=60&limit=2 HTTP/1.1" 200 None
[05/16/20 18:05:56 UTC][root][DEBUG]:Request response: {"data": {"pair": "BTC/USD", "ohlc": [{"high": "20690.59", "timestamp": "1608141840", "volume": "13.32120988", "low": "20677.18", "close": "20677.18", "open": "20690.59"}, {"high": "20694.99", "timestamp": "1608141900", "volume": "3.77376012", "low": "20671.52", "close": "20694.99", "open": "20681.91"}]}}
[05/16/20 18:05:57 UTC][root][INFO]:Ended.

Request

  1. How can I fix this issue?
  2. Is there any community, discussion group, I can join for Bitstamp API ussies?
api cryptography finance ohlc
Original Q&A
1

There are 1 answers

1
alpha_waves_6969 On

first off, very precise problem description, heads up.

It helps converting the unix time into readable time and you will see the answer to your question.
Like other REST APIs from exchanges/vendors you have a distribution limit from the platform. So that is not your request limit but the amount of data coming back.
So by increasing the limit variable, you increase the lookback in time.
So for a step=60 and a limit=2, you will receive the last 2 minutes of OHLC data. For limit=3 it would be the last 3 minutes. For a step=300 and limit=2, it would be the 2 last 5 minutes.

Below some code to translate the API answer to understand it a bit better:

# amending your get ohlc data function a bit:
import pandas as pd
import json
def get_ohlc_data(pair: str, step, limit):
    url = f"ohlc/{pair}/"
    request_url = base_url_v2 + url

    params = {
        "step": step,
        "limit": limit
    }
    try:
        logging.info("Sending request...")
        response = requests.get(request_url, params=params)
        # including the json.dumps() function
        response = json.dumps(response.text)
        return response["data"]["ohlc"]

        #logging.debug(f"Request response: {response.text}")

    except Exception as exp:
        logging.error(exp)


# now your main function just without the waitings:
def main():
    req = get_ohlc_data(pair="btcusd", step=60, limit=8)

    for i in range(len(req)):
        df = pd.DataFrame.from_dict(req[i], orient="index").T
        df["timestamp"] = pd.to_datetime(df["timestamp"], unit="s")
        print(df)

main()

This eventually will return you the following:

        high           timestamp      volume       low     close      open
0  50558.85 2021-03-07 18:51:00  1.44848029  50515.63  50546.20  50558.85
        high           timestamp      volume       low     close      open
0  50567.90 2021-03-07 18:52:00  0.80101510  50540.16  50554.20  50558.10
        high           timestamp      volume       low     close      open
0  50545.71 2021-03-07 18:53:00  3.49063548  50490.38  50512.13  50545.71
        high           timestamp      volume       low     close      open
0  50573.18 2021-03-07 18:54:00  1.29261148  50508.50  50519.68  50508.50
        high           timestamp      volume       low     close      open
0  50553.27 2021-03-07 18:55:00  1.45939999  50465.08  50553.27  50526.51
        high           timestamp       volume       low     close      open
0  50539.70 2021-03-07 18:56:00  13.61910047  50440.12  50457.87  50539.70
        high           timestamp       volume       low     close      open
0  50467.67 2021-03-07 18:57:00  10.23238701  50403.76  50428.09  50467.36
        high           timestamp      volume       low     close      open
0  50438.21 2021-03-07 18:58:00  1.50492458  50376.29  50401.85  50418.65

So in the end, there is nothing wrong with the API nor its stream. Just looks a bit misleading with the unix time it streams. Also, if you want to pass a start and end variable, you would need to send them as unix time. As the API only accepts unix time.

This can be done through:

from datetime import datetime
datetime.utcnow().timestamp()

Related Questions in API

Related Questions in CRYPTOGRAPHY

Related Questions in FINANCE

Related Questions in OHLC

Popular Questions

Popular Tags

javascript python java c# php android html jquery c++ css ios sql mysql r reactjs node.js arrays c asp.net json python-3.x ruby-on-rails .net sql-server swift django angular objective-c pandas excel

Trending Questions