Resolving dependencies fails on boto3 and s3fs using poetry

Question

I can install boto3, s3fs and pandas using :

pip install boto3 pandas s3fs

But it fails with poetry :

poetry add boto3 pandas s3fs

Here is the error :

Because no versions of s3fs match >2023.3.0,<2024.0.0
 and s3fs (2023.3.0) depends on aiobotocore (>=2.4.2,<2.5.0), s3fs (>=2023.3.0,<2024.0.0) requires aiobotocore (>=2.4.2,<2.5.0).
And because no versions of aiobotocore match >2.4.2,<2.5.0
 and aiobotocore (2.4.2) depends on botocore (>=1.27.59,<1.27.60), s3fs (>=2023.3.0,<2024.0.0) requires botocore (>=1.27.59,<1.27.60).
And because boto3 (1.26.91) depends on botocore (>=1.29.91,<1.30.0)
 and no versions of boto3 match >1.26.91,<2.0.0, s3fs (>=2023.3.0,<2024.0.0) is incompatible with boto3 (>=1.26.91,<2.0.0).
So, because engexploit-k8s-pod-operator-images depends on both boto3 (^1.26.91) and s3fs (^2023.3.0), version solving failed.

Answer 1

I was using s3fs through the pandas.read_csv method. I solved the question using only boto3, like in this answer.

Answer 2

I don't know how much time I've spent over the years handling version conflicts between s3fs, aiobotocore and boto3... But it's alot! I've found adding these three dependencies to pyproject.toml in this order usually works for me:

s3fs = {extras = ["boto3"], version = ">=2023.12.0"}
boto3 = "*"
botocore = "*"

Note that I use whatever the latest version of s3fs is - which, at this time, is 2023.12.0.

Answer 3

After some more research, I think this is the bottom line. botocore and aiobotocore are very tightly linked, so there is exactly one version of botocore that matches a given aibotocore version. So we are stuck with an older version of botocore if we need s3fs because it depends on aiobotocore.

(Ref: https://github.com/fsspec/s3fs/issues/357#issuecomment-687376253)

SO, a good bet is to use venv (usage is beyond the scope of this answer) and pin all of the s3fs/aiobotocore/botocore versions to match each other according to the errors pip -r feeds back using requirements.txt. Now if you also ad in a third fixed requirement on botocore (tsk tsk on the package maintainers) (i.e. pyathena), this method isn't foolproof for arbitrary old versions you want to match up.