Best way to send huge data in streaming mode through Python flask application?

48 views Asked by At

I want to create flask application which returns the huge data but the problems is, my api is crashing when it is ask for huge dataset like 6 million records. I want to make my flask application which gives streaming data, where I will just send 100000 records at a time.

What my application looks like:

import clickhouse_connect
import orjson
from flask import Flask

app = Flask(__name__)


@app.route("/")
def send_data():
    client = clickhouse_connect.get_client(host='XXXX', port=XXXX, username='XXXX', password='XXXXX', database = 'XXXXX')
    df = client.query_df("SELECT * FROM Table_Name")
    #table_df.to_dict(orient='records')
    print(type(df.to_dict(orient='dict')))
    return df.to_dict(orient='dict')

Suggestions are welcome make my application even faster.

0

There are 0 answers