Super-Operator In Kubernetes

Question

I need to automate the provisioning of a complex application in Kubernetes. It's a complex, multi-step process that involves provisioning of some cluster-wide resources and some app-specific resources. The cluster-wide resources are:

• Istio
• A few Operators (Cert Manager, Prometheus Operator, Postgres Operator, among others)

Then I want to create an application (let's call it Foo) which leverages Istio and the aforementioned operators. It will create statefulsets, services, Certificates, a Postgres database, Istio gateways, Prometheus PodMonitors, etc.

There will be multiple Foo's created, each configured differently (since the Kubernetes cluster will be used to provide Foo applications as a multi-tenant service).

What's the idiomatic way to do this? I think I should write a Foo controller which assumes that Istio and the other operators (prometheus, cert-manager, postgres, etc) already exist.

Is it possible to write a meta ClusterOfFoos operator that installs Istio, installs the required operators, and then installs the Foo controller?

If so, how does one go about provisioning operators (normally installed through Helm) from within a controller?

So far I have looked into using helm to do this, but there are too many dependencies and Helm just tends to create all resources at once, which makes some things fail (eg. when a deployment refers to a Secret that hasn't yet been created by cert-manager).

Answer 1

The Operator Lifecycle Manager is really well suited for the task.

When you create operator Foo, you can package it in the OLM way by creating a bundle which contains the ClusterServiceVersion needed to inform OLM of dependencies that need to be resolved before install and during upgrades. These can just be a list of APIs you need - and OLM will find and install the set of latest versions of the operators that own each API.

All your dependencies are operators available in the Operatorhub.io Catalog so they are available for install and dependency resolution as soon as you install OLM.

You can also configure certain dependencies by including these objects in the bundle itself. According to the docs, the following objects are supported as of the time of this post:

Secret
ClusterRole
ClusterRoleBinding
ConfigMap
ServiceAccount
Service
Role
RoleBinding
PrometheusRule
ServiceMonitor
PodDisruptionBudget
PriorityClasse
VerticalPodAutoscaler
ConsoleYAMLSample
ConsoleQuickStart
ConsoleCLIDownload
ConsoleLink

The Operator SDK can help you with bootstrapping the bundle.

Answer 2

By using GitOps workflow you can automate complex applications in Kubernetes.

• You need to define cluster-wide resources and application specific resources in a YAML file.
• By using GitOps tools you can continuously deploy kubernetes resources and they will automatically deploy the changes in the cluster.
• Use Helm chart to install Istio and make sure dependencies in the Helm chart are created in order.
• You can create a custom controller by FOO where it can read configuration of YAML files.
• Use kubernetes CRDs to define configuration of each FOO; they will allow you to create custom resources which are specific for each application.
• By using Helm; it will read the configuration from the CRD and generate correct YAML values.

The above described approach will allow you to create multiple FOO applications with different configurations and ensure that the required resources are installed in correct order.

You can check this article from codefresh regarding GitOps Workflow and official kubernetes page. You can also check Working with Multiple Applications and Environments and how Argo CD is useful for this scenario.