Grafana with kubeflow

Question

I am trying to integrate Grafana with my kubeflow in order to monitor my model.

I have no clue from where to start as I am not able to find anything in the documentation.

Can someone help?

Answer 1

To run Grafana with kubeflow, follow the steps:

1. create namespace

kubectl create namespace knative-monitoring

2. setup monitoring components

kubectl apply --filename https://github.com/knative/serving/releases/download/v0.13.0/monitoring-metrics-prometheus.yaml

3. Launch grafana board via port forwarding

kubectl port-forward --namespace knative-monitoring $(kubectl get pod --namespace knative-monitoring --selector="app=grafana" --output jsonpath='{.items[0].metadata.name}') 8080:3000

Access the grafana dashboard on http://localhost:8080.

Answer 2

It depends on your configuration. I had a MiniKF instance running on an EC2 VM and needed to specify the address was 0.0.0.0 for the port-forwarding method to work.

kubectl port-forward --namespace knative-monitoring \
  $(kubectl get pod --namespace knative-monitoring \
    --selector="app=grafana" --output jsonpath='{.items[0].metadata.name}') \
  --address 0.0.0.0 8080:3000

Then you should be able to access the grafana dashboard at http://{your-kf-ip}:8080

Answer 3

You can also expose it via istio, using this virtualservice:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: grafana-vs
  namespace: kubeflow
spec:
  gateways:
  - kubeflow-gateway
  hosts:
  - '*'
  http:
  - match:
    - method:
        regex: GET|POST
      uri:
        prefix: /istio/grafana/
    rewrite:
      uri: /
    route:
    - destination:
        host: grafana.istio-system.svc.cluster.local
        port:
          number: 3000

So if you're visiting your kubeflow dashboard usually via https://kubeflow.example.com, having this exposed through kubeflow-gateway will allow you to access it via https://kubeflow.example.com/istio/grafana/

If you're not using Istio's grafana but Knative's, you can change the destination accordingly.

You might also need to change the root url of grafana via an env variable in grafana's deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
  namespace: istio-system
spec:
  template:
      containers:
      - env:
        - name: GF_SERVER_ROOT_URL
          value: https://kubeflow.example.com/istio/grafana