How to integrate grafana agent running in AKS with grafana cloud

Question

I have installed grafana agent in my Azure Kubernetes cluster by using helm chart

Link: https://artifacthub.io/packages/helm/grafana/grafana-agent

Step-1:

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm install my-release grafana/grafana-agent

Step-2:

I have created grafana cloud account by refering to following documentation: https://grafana.com/auth/sign-up/create-user

Step-3:

I have updated configmap of grafana agent by adding agent.yaml: | information

apiVersion: v1
data:
  config.river: "logging {\n\tlevel  = \"info\"\n\tformat = \"logfmt\"\n}\n\ndiscovery.kubernetes
    \"pods\" {\n\trole = \"pod\"\n}\n\ndiscovery.kubernetes \"nodes\" {\n\trole =
    \"node\"\n}\n\ndiscovery.kubernetes \"services\" {\n\trole = \"service\"\n}\n\ndiscovery.kubernetes
    \"endpoints\" {\n\trole = \"endpoints\"\n}\n\ndiscovery.kubernetes \"endpointslices\"
    {\n\trole = \"endpointslice\"\n}\n\ndiscovery.kubernetes \"ingresses\" {\n\trole
    = \"ingress\"\n}"
  agent.yaml: |
    global:
        scrape_interval: 60s
        external_labels:
          cluster: example.cluster.dev
    configs:
      - name: integrations
        remote_write:
        - url: https://prometheus-prod-13-prod-us-east-0.grafana.net/api/prom/push
          basic_auth:
            username: xxxx
            password: xxxx
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: my-release
    meta.helm.sh/release-namespace: default
  creationTimestamp: "2023-06-22T13:45:11Z"
  labels:
    app.kubernetes.io/instance: my-release
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: grafana-agent
    app.kubernetes.io/version: v0.34.2
    helm.sh/chart: grafana-agent-0.16.0
  name: my-release-grafana-agent
  namespace: default
  resourceVersion: "35520419"
  uid: 3980b7b3-e09a-48ce-b8d6-3d9e681d5b10

Got username and password details from grafana cloud

But I am not able see any metrics being sent to my grafana cloud.

I am completely new to grafana cloud. Please help me to resolve this error. I am new to grafana tech stack.

Thanks in Advance

Answer 1

To integrate AKS with Grafana Cloud using Grafana Agent, please follow the steps below:

Steps:

1. Go to the Grafana portal, i.e., .grafana.net. Click on Home > Observability > Kubernetes > Configuration.

2. Select the correct data sources.

3. Install the dashboard and alert rules. You can install them by clicking the button in the portal.

4. Click on the configuration instructions, which will guide you through the installation process.

Note:

If, after completing step 3, your pod continues to crash, it may be due to missing configuration in the scrape_configs list in the agent.yaml file created during the deployment of the ConfigMap. Review the logs to identify the line number and remove the corresponding index from the scrape_configs list.

In my case, the pod is crashing due to the entry job_name: integrations/grafana-mimir/kubelet. After removing this entry, wait for 60 seconds, and you should start seeing metrics in the dashboard you installed during step 2.

After updating the configmap. Please delete the pod so that it will fetch latest files.