Flink checkpoints retained

Question

I have deployed task manager and jobmanager in docker container and configured following in flink-conf.yaml.

state.checkpoints.dir: file:///tmp/flink/checkpoints
state.storage.fs.memory-threshold: 0
state.backend.type: rocksdb
state.backend.incremental: true
state.checkpoints.num-retained: 100

when i checked the tmp/flink/checkpoints directory of task manager it has lot of chk-X directories though i have configured only 100 to be there. But in jobmanager tmp/flink/checkpoints directory has only one chk-X(latest check point directory).

Then what is the use of

state.checkpoints.num-retained

Is it not working in my case? if so what am i missing?.

Or is my understanding wrong ?

Answer 1

1) Modify flink-conf.yaml file

Locate the flink-conf.yaml file in your Flink installation directory.

Open the flink-conf.yaml file in a text editor.

Search for the following configuration property:

state.checkpoints.num-retained: 1

Change the value to the desired number of retained checkpoints. For example:

state.checkpoints.num-retained: 3

Save the changes to the file.

2) Specify Configuration in Job Submission

Alternatively, you can set the configuration option when submitting your Flink job using the -D command-line option:

>./bin/flink run -Dstate.checkpoints.num-retained=3 your-flink-job.jar

Replace your-flink-job.jar with the actual JAR file of your Flink job.

After making the changes, Flink will retain the specified number of completed checkpoints for your job. Adjust the value based on your requirements and the resources available in your environment. Keep in mind that retaining more checkpoints can consume additional storage space.