Meltano Kafka Monk.io integration

Question

I'm playing a bit with Monk.io and Kafka - Meltano integration. So, I would like to create a Monk.io Kafka cluster and provision a new connection on Meltano. I'm using https://github.com/lensesio/fast-data-dev for Kafka env. What would be the best approach and have the most sense? I've planed to do it this way: Create runnables of Kafka and Meltano and create Monk actions for Meltano template. Those actions would have custom Meltano loader that would provide pipe to Kafka.

Answer 1

I haven't tested the integration with Kafka but should be similar to Postgres in terms of execution in actions i.e to add a loader or extractor if those Kafka plugins exist for Meltano.

  actions:
    defines: actions
    add:
      arguments:
        resource-type:
          type: string
        resource-name:
          type: string
      code: exec("meltano", "/bin/sh", "-c", `cd elt && meltano add ${args["resource-type"]} ${args["resource-name"]}`)

The workflow and template are abstracted here https://github.com/monk-io/monk-dataops so it's more a question if Meltano has a good Kafka plugin