How to keep topic offset position while recreating AWS MSK sink connector for Snowflake

Question

Currently we have an AWS MSK cluster and AWS MSK sink connector for Snowflake, and we would like to add additional topics to the existing AWS MSK Connector. We are concern about topic offset in case of recreating MSK connect. Seems like AWS MSK doesn't have feature to dynamically update configuration(in the roadmap), and there is only delete/recreate option available. But this option will reset offset position in connector.

Does somebody know how to the preserve offset with AWS MSK Connect?

As the second option, if we will deploy Kafka connector in docker, can we store offset internally? So after updating configuration and restarting connector, it will start from the offset position where it left?

Thanks

Answer 1

Sink connector offsets are managed using the Kafka consumer group protocol. Each sink connector creates a group named connect-{CONNECTOR_NAME}. As long as the consumer group exists, any successive sink connectors that you create with the same CONNECTOR_NAME value will continue from the last committed offset.

Simply put: as long as you preserve the name of the connector that sinks data to Snowflake, it will catch the offset automatically, you don't need to worry about it. More information about offsets here: https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-workers.html