Any easy way to keep track/ maintain kubectl config (./kube/config) if you are connecting to multiple clusters

Question

Any suggestions on how track of kubectl configs(~/.kube/config) which allows you to access the kubernetes clusters? I have clusters running on different environments(local/prod) where i connect to the same namespace where project is deployed on and whenever i need to connect to a particular cluster, i run the below to configure ( different commands on aws/gcp/ microk8s etc) and the configuration gets attached to ~/.kube/config. Is there any easy way to know where you are connected or track which config is being used? Its a disaster waiting to happen unless you do a explicit check.

aws eks --region region update-kubeconfig --name cluster_name

Current method used:

• Either (cat ~/.kube/config) i check the to see what cluster im connecting to.
• move the config to some other directory and move the config back once im done.
• kubectl get nodes to see where I'm connected.

Answer 1

Using kubectl

Kubectl has built in support for managing contexts. After you add a context in ~/.kube/config file, manually or, via aws eks update-kubeconfig, you can use the config sub-command to switch between contexts.

To view all saved contexts and highlight the current one:

kubectl config get-contexts

To just view the current context:

kubectl config current-context

To switch to another context

kubectl config use-context <context-name>

To delete a context:

kubectl config delete-context <context-name>

Specific configuration file

Sometimes it might be the case that all the cluster connections cannot be in the same kube config file, but instead, user has separate kube config files per cluster.

To run kubectl with a specific configuration, one can use --kubeconfig argument:

kubectl --kubeconfig ./someConfig -n someNs get pods

Shell Aliases

And when running from Linux shell or windows powershell, one can also use "aliases".

Linux Bash example:

Use bash alias to define commands as aliases:

# Define a kubectl alias for specific cluster
alias myCluster="kubectl --kubeconfig ./myClusterConfig"

# Define a kubectl alias for specific cluster and specific namespace
alias myClusterNs="kubectl --kubeconfig ./myClusterConfig -n myNamespace"

Usage:

# Using cluster kubectl alias
myCluster -n myNamespace get pods

# Using cluster kubectl alias with namespace
myClusterNs get pods

The alias definitions can be saved to ~/.profile for permanent usage.

Windows Powershell example:

In Windows Powershell, a function can be defined as follows:

function myCluster { kubectl --kubeconfig .\myClusterConfig $args }

And used as:

myCluster -n myNamespace get pods

More arguments like -n <namespace> can also be specified in function definition before $args. Make sure to properly quote (") the arguments with special characters on windows.

Answer 2

If you don't mind using a UI tool, lens (https://k8slens.dev/) is really awesome. You can register multiple clusters, give them names and also different pictures.

For command line, there are shell extenstions that add the current cluster + namespace to the shell's prompt, eg. https://github.com/jonmosco/kube-ps1

For organization I store a seperate kubeconfig file for every cluster in my file system in a nested folder structure and access them with functions definedin my .zshrc file (zshell config file), eg:

env-dev-foo() {
    KUBECONFIG="/home/user/.kube/otherkubeconfig/dev/foo/config"
}

env-prod-bar() {
    KUBECONFIG="/home/user/.kube/otherkubeconfig/prod/bar/config"
}

env-prod-legacy() {
    KUBECONFIG="/home/user/.kube/otherkubeconfig/prod/legacy/config"
    PATH=$PATH:<path-to-legacy-kubectl
    PATH=$PATH:<path-to-legacy-helm
    connect-via-vpn
    create-ssh-tunnel-to-customer-system
}

You can do all sorts of stuff in that functions beside just switching your kubeconfig. Eg. if you need to deal with legacy clusters, you might want to use a kubectl/helm binary in a different version. Or maybe you need to create an ssh tunnel in order to connect to that cluster or connect via VPN.