Error when installing `snowflake-connector-python` to GCP Cloud Composer

Question

I'm trying to use install snowflake-connector-python to a Cloud Composer environment but keep receiving an error that pops up in the UI, "Failed to install PyPI Packages. Check the Cloud Build log for details". The build log doesn't seem overly helpful and is very long, so I'll just show some of the notable things here.

This is a warning that shows after installing snowflake-connector-python and its dependencies.

Installing collected packages: pycryptodomex, asn1crypto, oscrypto, jmespath, botocore, s3transfer, boto3, isodate, msrest, azure-core, azure-storage-blob, azure-common, snowflake-connector-python
Successfully installed asn1crypto-1.4.0 azure-common-1.1.25 azure-core-1.8.1 azure-storage-blob-12.5.0 boto3-1.14.63 botocore-1.17.63 isodate-0.6.0 jmespath-0.10.0 msrest-0.6.19 oscrypto-1.2.1 pycryptodomex-3.9.8 s3transfer-0.3.3 snowflake-connector-python-2.3.2
+ [[ -z fail ]]
+ python3 -m pipdeptree --warn fail
Warning!!! Possibly conflicting dependencies found:
* botocore==1.17.63
 - docutils [required: >=0.10,<0.16, installed: 0.16]

This is the final error notification.

The command '/bin/sh -c bash installer.sh $COMPOSER_PYTHON_VERSION  fail' returned a non-zero code: 1
ERROR
ERROR: build step 0 "gcr.io/cloud-builders/docker" failed: step exited with non-zero status: 1

I'm running the image composer-1.12.1-airflow-1.10.10 in zone us-central1-c and is using python 3. I've tried installing different versions of the package with no luck, and have tried doing it via the UI and via gcloud. Any help as to the root of this problem would be much appreciated!

Answer 1

I tried trying to install older versions of snowflake-connector-python but that didn't work.

What did end up working was adding docutils==0.15 to the list of packages to install. This fixed the dependency conflict warning and led to a successful build.

Previously, I've received dependency conflict warnings that led to successful builds. But for this case, I actually needed to resolve that conflict.

Answer 2

The answer from nate did not work for me, the PyPi package install succeeds in CloudBuild but then the Web Server fails to update and rolls back installing the packages.

In order to get this working we had to downgrade our Composer to the September release which comes with docutils==0.15.2.

The image we used was composer-1.12.1-airflow-1.10.9, it seems that any image in airflow 1.10.10 or above has this issue as they come with docutils==0.16 as standard. You can see which version of docutils is in an image by using this link and expanding the PyPi packages column.