Airflow 1.10.13, 2020-11-24 issues after update with pip install

Question

Below is the configuration which worked till December 1st

1. composer-1.11.2-airflow-1.10.6
2. Python – 3.6
3. 'dbt==0.17.0',
4. 'google-cloud-storage',
5. 'google-cloud-secret-manager==1.0.0',
6. 'protobuf==3.12.2'

With the above configuration we are observing below error. ERROR: Cannot install dbt and protobuf==3.12.2 because these package versions have conflicting dependencies.\n\nThe conflict is caused by:\n The user requested protobuf==3.12.2\n dbt-bigquery 0.17.0 depends on protobuf<3.12 and >=3.6.0\n\nTo fix this you could try to:\n1. loosen the range of package versions you've specified\n2. remove package versions to allow pip attempt to solve the dependency conflict\n\nERROR: ResolutionImpossible

Post receiving this error message we have tried many version changes to resolve the above issue. Below listed are few of the fixes applied and errors we have observed for those fixes.

1. Fix applied: downgraded protobuf version from 3.12.2 to 3.11.3 as per the above error Error: Requested docutils<0.16,>=0.10 from https://files.pythonhosted.org/packages/d4/12/6c3fd74a590c7327c98cae008c11d536029fa9cd7924de477e8cb8804186/docutils-0.15.1-post1.tar.gz#sha256=f33ddb723332c6d6b6d99731ee1fc0c35eb4044a2df5cca1c64c8aa78eaf22cb (from botocore<1.15,>=1.5.0->dbt-redshift==0.17.0->dbt==0.17.0) has different version in metadata: '0.15.1'

2. Fix applied: Specified docutils version to 0.15.2 as to fix the above error Error: SyntaxError: Missing parentheses in call to 'print'. Did you mean print("Generating %s..." % output)?\n ----------------------------------------\nERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

3. Fix applied: Tried upgrading python version from 3.6 to 3.8. Error: The path python3.8 (from --python=python3.8) does not exist.

4. Fix applied: Removed package versioning for above libraries to allow pip attempt to solve the dependency conflict. Error: Command errored out with exit status 1:\n command: /tmp/venv__vuo821/bin/python3.6 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-ghz6thkq/protobuf_a5303ec5c2df4986bb9b05948ae30052/setup.py'"'"'; file='"'"'/tmp/pip-install-ghz6thkq/protobuf_a5303ec5c2df4986bb9b05948ae30052/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-gi38587q\n cwd: /tmp/pip-install-ghz6thkq/protobuf_a5303ec5c2df4986bb9b05948ae30052/\n Complete output (6 lines):\n Traceback (most recent call last):\n File "", line 1, in \n File "/tmp/pip-install-ghz6thkq/protobuf_a5303ec5c2df4986bb9b05948ae30052/setup.py", line 52\n print "Generating %s..." % output\n ^\n SyntaxError: Missing parentheses in call to 'print'. Did you mean print("Generating %s..." % output)?\n ----------------------------------------\nERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output

5. Fix applied: Upgraded dbt to 0.17.2 Error: Received SIGTERM. Terminating subprocesses.

6. Fix applied: Tried downgrading setuptools version to 49.6.0 Error: SyntaxError: Missing parentheses in call to 'print'. Did you mean print("Generating %s..." % output)?\n ------------------------------- ---------\nERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

7. Fix applied: upgraded Composer and Airflow versions to composer-1.13.2-airflow-1.10.9 Error: Got similar errors with above configuration combinations

Answer 1

Which pip version are you using? If you upgrade pip to a newer version, please note that the dependency resolution algo has fundamentally changed since [email protected]: https://pip.pypa.io/en/stable/user_guide/#changes-to-the-pip-dependency-resolver-in-20-3-2020. Backtracking will happen to ensure there are no conflict dependencies: https://pip.pypa.io/en/stable/user_guide/#dependency-resolution-backtracking.

Stick to an older pip version (<20) may work in your case