Problem
I am trying to use terraform to deploy a cluster with a predefined set of applications/resources. I want to deploy operators to manage my resources, but I am running into an issue when attempting to set resource annotations. The simple issue being that I am unsure how to pass the annotations to the resources.
I am still new to OLM so I am very likely just not aware of the existing solution, but any searches I run don't seem to explain how any annotations can be passed from OLM to any operators managed resources.
My question
Is it possible to pass annotations to OLM natively? or am I supposed to annotate the resource after its deployed? if so what happens when the resources is redeployed by OLM? If an OLM provisioned operator will just "operate" any follow-on resources, then can I just set the annotations in that resource?
- A1: this seems to be "NO OLM does not natively support annotations"?
- A2: OLM does not currently pass on any values to managed resources other than some Tolerations and resource limit/needs. (I have an issue submitted on this)
- A3: Unsure as to what OLM does when a resource is recreated, but on upgrade most changes will be lost. Looks like patching is required here.
:sad-face:
Should I try and move away from OLM to a Helm focused deployment instead? I know some applications have both OLM subscriptions AND Helm charts to deploy operators, but I'm not clear on the pros vs cons of the two. Any clarity there would be appreciated.
- Status: WIP
What is required/ how big of a lift is it to extend the Subscription
resource to enable passing values?
- Status: WIP
Are there any other approaches that you know of to ensure my resources have the right annotations?
Thanks in advance for any advice!
Updates
Update 1
I found a resource in the external-secrets-operator called OperatorConfig
that seems to be a solution? But I can't find any external documentation on it, so this may just be an external-secrets-operator
thing?
Update 2
I found some talk by Cert-Manager on OLM configuration limitations. The way they describe it is less than heartening. Source
Update 3
Following up, I see that there are not a lot of options for enabling configurations via OLM. The issue seems to be that each time OLM updates an operator it overwrites the CSV of the operator, the Subscriptions seems to be the part that persists between different operator versions, but it has limited configuration options. Source
I have taken a shallow dive into what would be required to expand the subscription object, and created an issue (kind/feature) in the OLM repo on GitHub. Issue
As of right now, I think the answer is that "OLM is not currently capable of passing persistent values to operator managed resources"? Though, I have to question how OLM intends to remain relevant w/o it? Never the less, I am leaving this question open while I continue trying out my other potential options. I am also adding to my question with a 3rd option, "making a PR for the Subscription resource"
My current attempts
Example Templated Subscription
CRD
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
annotations: <maybe-here?> # I found one example with an annotations block here, but it was the CSV not subscriptions, and it was null valued (empty).
name: ${svc-name}-operator
namespace: operators
spec:
channel: ${channel}
name: ${op-name}
source: ${catalog}
sourceNamespace: olm
installPlanApproval: Automatic
configs:
annotations: <maybe-here?> #except the docs say that only a select set of values are supported
Example local.merged.operators
"splunk" : {
isDefaultApp = true
name = "splunk"
source = "operatorhubio-catalog"
channel = "stable"
}
Example Terraform for Subscription
CRD kubectl apply
# Cycles through a pre-filtered list of map objects
# pulls the related templated subscription file
# then applies that
resource "kubernetes_manifest" "subscriptions" {
for_each = local.merged.operators
manifest = yamldecode(file(pathexpand("${path.module}/olm/subscriptions/${each.key}.yaml")))
depends_on = [local_file.subscriptions]
}