Kafka Consumer with Flask Application

Question

I have Flask application running to serve the API endpoint, in the same application I have to implement also kafka consumer in order to consume events from kafka stream, but because of the kafka consumer should run forever, I am using python threads to run the kafka consumer in the background, I have some questions please

• is using threading ok? and What is the best practice to run Kafka consumer with flask app in general ?
• If using threading is ok, how to force the flask app to stop if the kafka consumer thread stops for any reason ?

Answer 1

While possible, I would recommend two separate processes, rather than embed the Kafka consumer in the Flask app. You could use supervisor to run Kafka and Flask/gunicorn processes both at the same time.

Otherwise, for example, use Kafka Connect to write to a database. Have Flask query the database. Or externally run ksqlDB or Kafka Streams Interactive Query feature (requires JVM development, not Python) and query that.