openmetadata/ingestion Environment variables not working properly

Question

Ingestion framework: Ingestions not applying the ENV variables properly

I tried to deploy the docker image openmetadata/ingestion:1.2.2 in the Kubernetes environment, I set all the required environment variables in the deployment.yaml, My DB_SCHEME was postgresql+psycopg2, When I set DB_PASSWORD in the env it uses the SQLite instead of Postgres if no DB_PASSWORD it chooses the Postgres and throws a connection error

Below are the env variables used.

env:
    - name: AIRFLOW_DB_SCHEME
      value: postgresql+psycopg2
    - name: DB_SCHEME
      value: postgresql+psycopg2
    - name: AIRFLOW_DB
      value: airflow
    - name: AIRFLOW_DB_USER
      value: open_metadata_airflow
    - name: DB_USER
      value: open_metadata_airflow
    - name: AIRFLOW_DB_HOST
      value: example.postgres.com
    - name: DB_HOST
      value: example.postgres.com
    - name: AIRFLOW_DB_PORT
      value: '5432'
    - name: DB_PORT
      value: '5432'
    - name: AIRFLOW__API__AUTH_BACKENDS
      value: 'airflow.api.auth.backend.basic_auth,airflow.api.auth.backend.session'
    - name: AIRFLOW__CORE__EXECUTOR
      value: LocalExecutor
    - name: AIRFLOW__OPENMETADATA_AIRFLOW_APIS__DAG_GENERATED_CONFIGS
      value: /opt/airflow/dag_generated_configs
    - name: AIRFLOW_DB_PASSWORD
      value: "airflow_pwd"
    - name: DB_PASSWORD
      value: "airflow_pwd"

Note: Duplicated ENV was used because of the documentation issue please see here https://github.com/open-metadata/OpenMetadata/issues/12282

To Reproduce

Set DB_SCHEME=postgresql+psycopg2 and DB_PASSWORD=*** in the env to reproduce this issue.

Expected behavior It should work based on the environment variables

Version:

• OpenMetadata version: 1.2.2
• OpenMetadata Ingestion docker image version: openmetadata/ingestion:1.2.2

Answer 1

we map DB_USER, DB_SCHEME, DB_USER for both OpenMetadata and Airflow. Can you set these values and remove the AIRFLOW_DB_SCHEME from your env. Here are the docs https://docs.open-metadata.org/v1.2.x/deployment/bare-metal#configure-postgres-connection