How do you check if a typeshed stub (.pyi) file matches the implementation?

Question

You can use mypy's stubgen.py or some other tools to generate .pyi files automatically, but stubgen contains the additional suggestion to check the stub files for accuracy.

If I autogenerated stubs then modified them to fix errors or make things more stringent that weren't picked up on by the generator, how can I then check that they agree with the implementation? Or as the implementation changes and I had hand-generated files, how can I make sure they stay in sync?

Answer 1

I found this question as a result of needing to do this too, so here is what I've subsequently found...

Apparently, a tool called stubtest was recently added to mypy (undocumented currently) which will validate the stubs against the implementation insofar as is possible. Here is a contrived example:

$ cat <<EOF > simple.py
> def thing(obj):
>     if isinstance(obj, float):
>         return 0.0
>     elif isinstance(obj, int):
>         return 0
>     else:
>         raise TypeError("Unsupported type passed to thing.")
> 
> EOF

I create a stubfile which uses knowledge of the implementation to write a specific type annotation (definitely not auto-generated - that would be hard to do!):

$ cat <<EOF > simple.pyi
> from typing import Any, Union
> 
> def thing(obj: Union[float, int]): ...
> 
> EOF

Running stubtest yeilds no output, and a zero exit code:

$ python -m mypy.stubtest simple

$ echo $?
0

Now I update the implementation to have another argument:

cat <<EOF > simple.py
> def thing(obj, something_else):
>     if isinstance(obj, float):
>         return 0.0
>     elif isinstance(obj, int):
>         return 0
>     else:
>         raise TypeError("Unsupported type passed to thing.")
> 
> EOF

And re-run the stubtest:

$ python -m mypy.stubtest simple
error: simple.thing is inconsistent, stub does not have argument "something_else"
Stub: at line 3
def (obj: Union[builtins.float, builtins.int]) -> Any
Runtime: at line 1 in file stub-testing/simple.py
def (obj, something_else)

$ echo $?
1

I'm still fairly weak in my understanding of the mypy search path, and couldn't get the direct stubtest entrypoint to work (as opposed to the python -m mypy.stubtest form).

I didn't yet test this against anything more complex than this toy example - I suspect there is a lot of devil in the detail.