I want to create flask application which returns the huge data but the problems is, my api is crashing when it is ask for huge dataset like 6 million records. I want to make my flask application which gives streaming data, where I will just send 100000 records at a time.
What my application looks like:
import clickhouse_connect
import orjson
from flask import Flask
app = Flask(__name__)
@app.route("/")
def send_data():
client = clickhouse_connect.get_client(host='XXXX', port=XXXX, username='XXXX', password='XXXXX', database = 'XXXXX')
df = client.query_df("SELECT * FROM Table_Name")
#table_df.to_dict(orient='records')
print(type(df.to_dict(orient='dict')))
return df.to_dict(orient='dict')
Suggestions are welcome make my application even faster.