Can we create a Flink DataStream from a Postgres table

Question

I have to read configs from a Postgres table and broadcast it to use it to filter a main DataStream. I am using Flink broadcast state for this . It worked fine when I got the config from local socket.

The use case is to read latest config from Postgres in Flink job without restarting the job.

Can we create a Flink data stream from Postgres table ?If possible then is it efficient as it will keep alive the JDBC connection forever?

Answer 1

You can create a custom JDBC source to periodically read data from postgresql. The interval depends on how often your configuration information is updated. If your data update period is long, you can try to release the connection each time you obtain configuration data from the source.