TechQA.

Not able to see log in datadog

51 views Asked by At
  1. deployed using helm
datadog:
  apiKeyExistingSecret: datadog-secret
  appKeyExistingSecret: datadog-secret
  remote_configuration:
    enabled: true
  clusterName: "localhost"
  forwarder_storage_max_size_in_bytes: 500000000
  dogstatsd_metrics_stats_enable: true
  site: datadoghq.com
  kubelet:
    tlsVerify: false
  apm:
    # enabled: true
    portEnabled: true
    port: 8126 # default
    instrumentation:
      enabled: true
  logLevel: TRACE
  logs:
    enabled: true
    containerCollectAll: true
  kubeStateMetricsEnabled: true
  networkMonitoring:
    enabled: true
  orchestratorExplorer:
    enabled: true
  clusterAgent:
    enabled: true
    replicas: 2
    createPodDisruptionBudget: true
  1. value.yml file contains the Pod is used for .net application.
# Default values for weatherforecast.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

replicaCount: 1

image:
  repository: weatherforecast
  pullPolicy: IfNotPresent
  # Overrides the image tag whose default is the chart appVersion.
  tag: "main"

imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""

serviceAccount:
  # Specifies whether a service account should be created
  create: true
  # Automatically mount a ServiceAccount's API credentials?
  automount: true
  # Annotations to add to the service account
  annotations: {}
  # The name of the service account to use.
  # If not set and create is true, a name is generated using the fullname template
  name: ""

podAnnotations: {
  "ad.datadoghq.com/weatherforecast.logs": '[{"source":"weatherforecast","service":"weatherforecast2"}]'
}
podLabels: {}

podSecurityContext: {}
# fsGroup: 2000

securityContext: {}
# capabilities:
#   drop:
#   - ALL
# readOnlyRootFilesystem: true
# runAsNonRoot: true
# runAsUser: 1000

service:
  type: NodePort
  port: 80
# service:
#   type: LoadBalancer
#   port: 80

ingress:
  enabled: false
  className: ""
  annotations: {}
  # kubernetes.io/ingress.class: nginx
  # kubernetes.io/tls-acme: "true"
  hosts:
  - host: localhost
    paths:
    - path: /
      pathType: ImplementationSpecific
  tls: []
  #  - secretName: chart-example-tls
  #    hosts:
  #      - chart-example.local

resources: {}
# We usually recommend not to specify default resources and to leave this as a conscious
# choice for the user. This also increases chances charts run on environments with little
# resources, such as Minikube. If you do want to specify resources, uncomment the following
# lines, adjust them as necessary, and remove the curly braces after 'resources:'.
# limits:
#   cpu: 100m
#   memory: 128Mi
# requests:
#   cpu: 100m
#   memory: 128Mi

livenessProbe:
  httpGet:
    path: /
    port: http
readinessProbe:
  httpGet:
    path: /
    port: http

autoscaling:
  enabled: false
  minReplicas: 1
  maxReplicas: 100
  targetCPUUtilizationPercentage: 80
  # targetMemoryUtilizationPercentage: 80

# Additional volumes on the output Deployment definition.
volumes: []
# - name: foo
#   secret:
#     secretName: mysecret
#     optional: false

# Additional volumeMounts on the output Deployment definition.
volumeMounts: []
# - name: foo
#   mountPath: "/etc/foo"
#   readOnly: true

nodeSelector: {}

tolerations: []

affinity: {}

deployment file

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "weatherforecast.fullname" . }}
  labels:
    {{- include "weatherforecast.labels" . | nindent 4 }}
spec:
  {{- if not .Values.autoscaling.enabled }}
  replicas: {{ .Values.replicaCount }}
  {{- end }}
  selector:
    matchLabels:
      {{- include "weatherforecast.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      {{- with .Values.podAnnotations }}
      annotations:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      labels:
        {{- include "weatherforecast.labels" . | nindent 8 }}
        {{- with .Values.podLabels }}
        {{- toYaml . | nindent 8 }}
        {{- end }}
    spec:
      {{- with .Values.imagePullSecrets }}
      imagePullSecrets:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      serviceAccountName: {{ include "weatherforecast.serviceAccountName" . }}
      securityContext:
        {{- toYaml .Values.podSecurityContext | nindent 8 }}
      containers:
        - name: {{ .Chart.Name }}
          securityContext:
            {{- toYaml .Values.securityContext | nindent 12 }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          ports:
          - name: http
            containerPort: {{ .Values.service.port }}
            protocol: TCP
          # livenessProbe:
          #   {{- toYaml .Values.livenessProbe | nindent 12 }}
          # readinessProbe:
          #   {{- toYaml .Values.readinessProbe | nindent 12 }}
          resources:
            {{- toYaml .Values.resources | nindent 12 }}
          {{- with .Values.volumeMounts }}
          volumeMounts:
            {{- toYaml . | nindent 12 }}
          {{- end }}
      {{- with .Values.volumes }}
      volumes:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      {{- with .Values.nodeSelector }}
      nodeSelector:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      {{- with .Values.affinity }}
      affinity:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      {{- with .Values.tolerations }}
      tolerations:
        {{- toYaml . | nindent 8 }}
      {{- end }}

Issues

  1. Not able to see the log of pod on datadog
  2. Network is not configured
  3. System.Probe is failed on datadog agent
  4. Orchestrator is partial.
  5. Datadog agent is failing for System.Probe only

I have been given permission to access the API as a remote configuration. Additionally, I want to push logs to DataDog from my pod. I am using the 7.51.0 version of agent and cluster. Dataagent push the details of clustes,pod,node,namespace etc details to datadog but after some time they also disappears.

enter image description here

enter image description here

enter image description here enter image description here


    orchestrator_pod
    ----------------
      Instance ID: orchestrator_pod:888ebc42a3817b00 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/orchestrator_pod.d/conf.yaml.default
      Total Runs: 51
      Metric Samples: Last Run: 0, Total: 0
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 6ms
      Last Execution Date : 2024-03-15 01:24:13 UTC (1710465853000)
      Last Successful Execution Date : 2024-03-15 01:24:13 UTC (1710465853000)


    telemetry
    ---------
      Instance ID: telemetry [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/telemetry.d/conf.yaml.default
      Total Runs: 51
      Metric Samples: Last Run: 4, Total: 204
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2024-03-15 01:24:20 UTC (1710465860000)
      Last Successful Execution Date : 2024-03-15 01:24:20 UTC (1710465860000)


    uptime
    ------
      Instance ID: uptime [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/uptime.d/conf.yaml.default
      Total Runs: 51
      Metric Samples: Last Run: 1, Total: 51
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2024-03-15 01:24:12 UTC (1710465852000)
      Last Successful Execution Date : 2024-03-15 01:24:12 UTC (1710465852000)

  Check Initialization Errors
  ===========================


      etcd (6.1.0)
      ------------

      instance 0:

        could not invoke 'etcd' python check constructor. New constructor API returned:
Traceback (most recent call last):
  File "/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/etcd/etcd.py", line 74, in __init__
    super(Etcd, self).__init__(
  File "/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/base/checks/openmetrics/base_check.py", line 126, in __init__
    raise CheckException(
datadog_checks.base.errors.CheckException: The agent could not connect to any of the following URLs: ['https://192.168.65.3:2379/metrics', 'http://192.168.65.3:2379/metrics'].
Deprecated constructor API returned:
Etcd.__init__() got an unexpected keyword argument 'agentConfig'
      kube_controller_manager (5.0.0)
      -------------------------------

      instance 0:

        could not invoke 'kube_controller_manager' python check constructor. New constructor API returned:
Traceback (most recent call last):
  File "/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/kube_controller_manager/kube_controller_manager.py", line 115, in __init__
    super(KubeControllerManagerCheck, self).__init__(
  File "/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/kube_controller_manager/sli_metrics.py", line 25, in __init__
    super(SliMetricsScraperMixin, self).__init__(*args, **kwargs)
  File "/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/base/checks/openmetrics/base_check.py", line 126, in __init__
    raise CheckException(
datadog_checks.base.errors.CheckException: The agent could not connect to any of the following URLs: ['https://192.168.65.3:10257/metrics', 'https://localhost:10257/metrics', 'http://192.168.65.3:10252/metrics', 'http://localhost:10252/metrics'].
Deprecated constructor API returned:
KubeControllerManagerCheck.__init__() got an unexpected keyword argument 'agentConfig'
      kube_scheduler (4.8.0)
      ----------------------

      instance 0:

        could not invoke 'kube_scheduler' python check constructor. New constructor API returned:
Traceback (most recent call last):
  File "/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/kube_scheduler/kube_scheduler.py", line 150, in __init__
    super(KubeSchedulerCheck, self).__init__(
  File "/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/kube_scheduler/sli_metrics.py", line 25, in __init__
    super(SliMetricsScraperMixin, self).__init__(*args, **kwargs)
  File "/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/base/checks/openmetrics/base_check.py", line 126, in __init__
    raise CheckException(
datadog_checks.base.errors.CheckException: The agent could not connect to any of the following URLs: ['https://192.168.65.3:10259/metrics', 'https://localhost:10259/metrics', 'http://192.168.65.3:10251/metrics', 'http://localhost:10251/metrics'].
Deprecated constructor API returned:
KubeSchedulerCheck.__init__() got an unexpected keyword argument 'agentConfig'
  Loading Errors
  ==============
    etcd
    ----
      Core Check Loader:
        Check etcd not found in Catalog

      JMX Check Loader:
        check is not a jmx check, or unable to determine if it's so

      Python Check Loader:
        could not configure check instance for python check etcd: could not invoke 'etcd' python check constructor. New constructor API returned:
Traceback (most recent call last):
  File "/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/etcd/etcd.py", line 74, in __init__
    super(Etcd, self).__init__(
  File "/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/base/checks/openmetrics/base_check.py", line 126, in __init__
    raise CheckException(
datadog_checks.base.errors.CheckException: The agent could not connect to any of the following URLs: ['https://192.168.65.3:2379/metrics', 'http://192.168.65.3:2379/metrics'].
Deprecated constructor API returned:
Etcd.__init__() got an unexpected keyword argument 'agentConfig'

    kube_controller_manager
    -----------------------
      Core Check Loader:
        Check kube_controller_manager not found in Catalog

      JMX Check Loader:
        check is not a jmx check, or unable to determine if it's so

      Python Check Loader:
        could not configure check instance for python check kube_controller_manager: could not invoke 'kube_controller_manager' python check constructor. New constructor API returned:
Traceback (most recent call last):
  File "/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/kube_controller_manager/kube_controller_manager.py", line 115, in __init__
    super(KubeControllerManagerCheck, self).__init__(
  File "/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/kube_controller_manager/sli_metrics.py", line 25, in __init__
    super(SliMetricsScraperMixin, self).__init__(*args, **kwargs)
  File "/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/base/checks/openmetrics/base_check.py", line 126, in __init__
    raise CheckException(
datadog_checks.base.errors.CheckException: The agent could not connect to any of the following URLs: ['https://192.168.65.3:10257/metrics', 'https://localhost:10257/metrics', 'http://192.168.65.3:10252/metrics', 'http://localhost:10252/metrics'].
Deprecated constructor API returned:
KubeControllerManagerCheck.__init__() got an unexpected keyword argument 'agentConfig'

    kube_scheduler
    --------------
      Core Check Loader:
        Check kube_scheduler not found in Catalog

      JMX Check Loader:
        check is not a jmx check, or unable to determine if it's so

      Python Check Loader:
        could not configure check instance for python check kube_scheduler: could not invoke 'kube_scheduler' python check constructor. New constructor API returned:
Traceback (most recent call last):
  File "/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/kube_scheduler/kube_scheduler.py", line 150, in __init__
    super(KubeSchedulerCheck, self).__init__(
  File "/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/kube_scheduler/sli_metrics.py", line 25, in __init__
    super(SliMetricsScraperMixin, self).__init__(*args, **kwargs)
  File "/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/base/checks/openmetrics/base_check.py", line 126, in __init__
    raise CheckException(
datadog_checks.base.errors.CheckException: The agent could not connect to any of the following URLs: ['https://192.168.65.3:10259/metrics', 'https://localhost:10259/metrics', 'http://192.168.65.3:10251/metrics', 'http://localhost:10251/metrics'].
Deprecated constructor API returned:
KubeSchedulerCheck.__init__() got an unexpected keyword argument 'agentConfig'

enter image description here

kubernetes-helm datadog
Original Q&A
0

There are 0 answers

Related Questions in KUBERNETES-HELM

Related Questions in DATADOG

Popular Questions

Popular Tags

javascript python java c# php android html jquery c++ css ios sql mysql r reactjs node.js arrays c asp.net json

Trending Questions