In a test, I am consuming messages with prefetch and manual ack using callback where I am also performing a database write with a cursor. However, I have to commit after each call. Is there a way for pika to indicate when it has finished the prefetched data?
consumer_channel.basic_consume(
queue=queue_name, on_message_callback=callback, auto_ack=False)
consumer_channel.start_consuming()
Right now, my only option seems to be to do the commit in the on_message_callback call to function callback which results in a lots of unnecessary commits.
edit: to clarify, if I have autocommit in the DB or I perform a commit to persist the data using the callback, it does one commit per message. I'd like to wait until the end of the prefetched data being process to execute the commit.
So
- Get X amount of records
- Write to db with cursor
- Acknowledge Message
- End of prefetched data, (perform DB commit)