On a very high level, I will be providing a multi-tenant platform interface to allow users create "Projects" - translated to Kafka topics for a variety of producer types - RSS feeds, SQL Server queries and file locations.
To retreive RSS data, for example, I believe we will need to create Kafka works to fetch RSS data on a schedule - thus needing to add new multi-threaded workers (or jobs to a worker) dynamically - not through configuration files.
The application abstraction layer will separate the user from Kafka and ElasticSearch. I can't be allowing all users access through Control Center.
In another use case where I may offer some type of managed services, I could use CC.
Given this use I would like to get some details about the appropriate way to accomplish this - as there still seem to be some gaps - i.e. we can't use configuration files as the topics need to be created dynamically.
How can I programmatically/dynamically create data pipelines and workers using Kafka and Java?