Use of KsqlDB as a HIstorian DB

Question

I want to make a historian database that records and retrieves data from Kafka topics to a permanent database.

I am trying to use the KsqlDB to do it, I have trying to use tables and streams from ksql but it disappears with the death of Kafka topics (extension of the retention period). My question is "is it possible to use ksqldb as a historian for kafka? " Thanks.

Answer 1

Kafka data certainly can be permanent (log.retention.ms=-1 or cleanup.policy=compact), although you don't need ksqlDB to make it so. By default, a table should be indefinitely available as it would be backed by a compacted Kafka topic

Anecdotally, the New York Times stores all their articles in a singly partitioned Kafka topic to sort their articles by time.

Note that ksqlDB is a processing engine built on Kafka Streams. It's not a database.