I am setting up a AWS Managed Workflows for Apache Airflow (MWAA) environment using airflow version 2.6.3, and I am using a requirements.txt file to install my desired python packages as suggested by Amazon's Development Guide here. However, pip is not able to install the requested dependencies because, apparently, there are conflicts.
My requirements.txt file is the following:
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.6.3/constraints-3.10.txt"
awswrangler~=3.4.0
The error logs in CloudWatch show the following errors:
ERROR: Cannot install -r /usr/local/airflow/requirements/requirements.txt because these package versions have conflicting dependencies.
The conflict is caused by:
awswrangler 3.4.2 depends on numpy<2.0 and >=1.18
awswrangler 3.4.1 depends on numpy<2.0 and >=1.18
awswrangler 3.4.0 depends on numpy<2.0 and >=1.18
The user requested (constraint) numpy==1.24.4
As we see, the package awswrangler requires numpy with version between 1.18 and 2.0. The constraints specify numpy==1.24.4, which is clearly in the range, but pip fails anyway because of a dependency problem.
I've tried pinning numpy version in the requirements.txt file before awswrangler, but the error persists.
Any ideas how I could fix this problem? Does not look like a problem in my environment configuration.
Many thanks in advance.