Serverless framework for trigger

Question

I am looking for a serverless framework(free) , where i can create a kafka trigger and when triggered a kube function is to be invoked (python)

I have tried nuclio but the problem is that i have kafka version higher and they do not support higher than 2.4.

I want something like:

apiVersion: "nuclio.io/v1beta1"
kind: "NuclioFunction"
spec:
  runtime: "python:3.6"
  handler: NuclioKafkaHandler:consumer
  minReplicas: 1
  maxReplicas: 1
  triggers:
    myKafkaTrigger:
      kind: kafka-cluster
      attributes:
        initialOffset: earliest
        topics:
          - nuclio
        brokers:
          - kafka-bootstrap:9092
        consumerGroup: Consumer

And a kube function like:

def consumer(context, event):
    context.logger.debug(event.body)
    print(event.trigger.kind)

As simple as these two files and i have already existing kafka cluster so i just want to have trigger on that.

what are the possible alternatives apart from nuclio? I looked into kubeless seemed complicated. Fission does not support python.

Answer 1

The exact same use case is possible with Fission which is an open source serverless framework for Kubernetes.

You can create a Message Queue trigger for Kafka and associate it with a serverless function like this:

fission mqt create --name kafkatest --function consumer --mqtype kafka --mqtkind keda --topic request-topic --resptopic response-topic --errortopic error-topic

This would trigger a function called consumer whenever there's a message in the request-topic queue of Kafka.

You can also associate meta data like authentication information as secrets or flags like pollingintervals, max retries etc.

Reference: https://fission.io/docs/usage/triggers/message-queue-trigger-kind-keda/kafka/

Answer 2

I don't know much about Nuclio but the scenario you described looks possible with Knative.

Simplest way, you can create a Knative Service for your consumer. For the Kafka part, you can use a KafkaSource to get the events into Knative Eventing system. In your KafkaSource, you can tell it to call the Knative Service when there's an event coming from Kafka.

Above is the simplest way. If you need more advanced features, there is also support for filtering based on event types or having multiple consumers subscribed to events and more features.

Red Hat's Knative Tutorial has a section for serverless eventing with Kafka.