mypy cannot infer type of generator comprehension correctly

Question

I am using the stub files provided by data-science-types to have mypy be able to check my pandas related code. Sadly I get the following behaviour:

For

import pandas as pd

def test() -> pd.DataFrame:
    pass

pd.concat((test() for _ in range(10)))

mypy reports test.py:6: error: Argument 1 to "concat" has incompatible type "Generator[DataFrame, None, None]"; expected "Union[Sequence[DataFrame], Mapping[str, DataFrame]]". If I use pd.concat([test() for _ in range(10)]) instead mypy is happy again. Can somebody explain to me what's going on there?

Just in case it's relevant. I am using python3.8.5, pandas 1.1.2, mypy 0.782 and data-science-types 0.2.18.

Answer 1

First of all since the generator expression in my code has SendType and ReturnType of None so Generator[DataFrame, None, None] is the correct type see docs. So as chepner has pointed out the problem lies in the expected type. Even though pandas.concat excepts generators data-science-types does not have it as possible input type for it. I would have considered that a bug but on their github page data-science-types they write

Philosophy: [...] Often the actual APIs in the libraries is more permissive than the type signatures in our stubs; but this is (usually) a feature and not a bug.

And I consider the problem solved.

Answer 2

You are confusing a generator with a sequence. A sequence is, by definition,

An iterable which supports efficient element access using integer indices via the __getitem__() special method and defines a __len__() method that returns the length of the sequence.

A generator supports neither, and it's not a kind of mapping, either, so you can't pass one to pd.concat.