Apache Flink in Kubernetes

Question

Could anyone please let me know how I can setup Flink in my Serverless platform (FaaS) to perform event driven operations?

I looked at Flink functions and it seems to be promising. Could anyone clarify on the below?

1. What I need to install in my FaaS env. to trigger the flink function when an event (file changes in my s3 bucket) occurs?

2. I don't have big data platform and so planning to use flink in my serverless/kubernetes env.

Thanks in advance!!

Answer 1

To use StateFun You would generally need:

1. An Ingress that would trigger the functions.
2. The actual code that would react to your events (the stateful function) Dockerized
3. A way to lunch your application

Specifically:

1. Every stateful function application starts with an Ingress, basically that is a funnel of events that your functions can react to. In your case, you can use Amazon Kinesis as your Ingress, and make sure that your S3 events will end up there.

2. The next thing that you would need, is to get yourself familiar with a stateful function SDK, either in Java or in Python and write the logic that deals with the incoming events. The result of that stage would be a Docker image.

3. Then, you need to lunch the image obtained at (2) and for that you can use Kubernetes (you don't have to). There are Helm charts provided for your convenience and a simple utility to generate the necessary k8s resources.