We are currently using EMR to run our Spark service and we want to migrate to Kubernetes. One of the things we thought of doing is using karpenter as the autoscaler of the Spark Kubernetes cluster we want to create.
I want to know, is there a way to scale the cluster up or down according to spark needs? According to the number of executors the drives requests or something like that?