Different Mypy behavior with bare vars and TypedDicts

Question

I'm still very much learning Python and started using Mypy from quite early on to avoid forgetting to use correct types and check for things missing in my code. Up to now most things played well and I was astonished what the checker reads from the code and not to complain about.

Today I found some situation that puzzles me. With the following very simplified example code

from typing import Optional

myvar: Optional[int] = None

myvar = 42

if myvar < 1:
    print("Error")

Mypy has nothing to complain because it obviously knows myvar cannot be None in the if statement.

Strange enough if I use a TypedDict this does not hold true as within the following piece:

from typing import Optional, TypedDict

class MyDictType(TypedDict):
    mykey: Optional[int]

mydict1: MyDictType = {'mykey': None}

mydict1['mykey'] = 42

if mydict1['mykey'] < 1:
    print("Error")

Here Mypy complains Unsupported operand types for > ("int" and "None") and Left operand is of type "Optional[int]". It "forces" me to directly check like

if mydict1['mykey'] is not None and mydict1['mykey'] < 1:
    print("Error")

Therefore my question is: Why doesn't Mypy recognize the very same thing (that mydict1['mykey'] can't be None) with TypedDict?

Answer 1

Your two examples are not identical and thus results in the different behavior, caused by assignment rules (and the lack of true Optional type for assignments in Python, more on this later) governing Python and thus Mypy. In the first example you have replaced myvar with a completely new assignment 42 which has the type int, not Optional[int]. In the second example there lacks a completely new assignment to mydict1 (as type checking is ultimately done against assignments at the underlying name - as the assignment of a new value to key 'mykey' only affects the opaque inner value inside mydict1, without a material change to the assignment to mydict1. This however only answers part of the question, as Mypy will correctly flag type mismatch issues where you to assign 'forty-two' to myvar, so what did Mypy really saw?

To find out, let's use reveal_type helper from Mypy. Add it to the line immediately after the affected assignments:

from typing import Optional

myvar: Optional[int] = None
reveal_type(myvar)

myvar = 42
reveal_type(myvar)

if myvar < 1:
    print("Error")

Running the above through Mypy (mypy 1.8.0)

so78038503_1.py:4: note: Revealed type is "Union[builtins.int, None]"
so78038503_1.py:7: note: Revealed type is "builtins.int"
Success: no issues found in 1 source file

Note Mypy simply assume myvar became an int in this instance and thus will not flag the simple comparison as an error (despite the fact that the underlying type annotation has not changed one bit, this may be verified by checking locals()['__annotations__'] at the relevant lines in the example scripts).

As for the second example, do the same reveal_type check:

from typing import Optional, TypedDict

class MyDictType(TypedDict):
    mykey: Optional[int]

mydict1: MyDictType = {'mykey': None}
reveal_type(mydict1['mykey'])

mydict1['mykey'] = 42
reveal_type(mydict1['mykey'])

if mydict1['mykey'] < 1:
    print("Error")

The check will note that mydict1['mykey'] maintains the type Union[builtins.int, None] without dropping down to builtins.int.

so78038503_2.py:7: note: Revealed type is "Union[builtins.int, None]"
so78038503_2.py:10: note: Revealed type is "Union[builtins.int, None]"
so78038503_2.py:12: error: Unsupported operand types for > ("int" and "None")  [operator]
so78038503_2.py:12: note: Left operand is of type "int | None"
Found 1 error in 1 file (checked 1 source file)

Do note that pyright will have not flag any issues (pyright 1.1.351) with either examples as it assumes the final values in both cases to be Literal[42] which is a builtins.int. I personally think that this behavior in both type checkers are incorrect as it may result in brittleness as a subtle change in the assignment with an actual Optional[int] will reveal the lack of the guard against None, which completely defeats the whole point of type checking in the first place, as you undoubtedly noted. To demonstrate this, instead of assigning a static value, let's change the assignment to the result of a function that's annotated to return an Optional[int] (to emulate a subsequent code change later in the development cycle):

from typing import Optional

def answer() -> Optional[int]:
    return 42

myvar: Optional[int] = None
reveal_type(myvar)

myvar = answer()
reveal_type(myvar)

if myvar < 1:
    print("Error")

This will then result in the expected error:

so78038503_3.py:7: note: Revealed type is "Union[builtins.int, None]"
so78038503_3.py:10: note: Revealed type is "Union[builtins.int, None]"
so78038503_3.py:12: error: Unsupported operand types for > ("int" and "None")  [operator]
so78038503_3.py:12: note: Left operand is of type "int | None"
Found 1 error in 1 file (checked 1 source file)

Should the return type of that function be changed to simply int (e.g. def answer() -> int:), Mypy will once again consider the type assigned to myvar be simply just an int, as Python does not have a real Optional type like other languages where the assignment of a non-None value must be encapsulated to be marked, so doing myvar = 42 will be a syntax error. For example, your intended example with the conditional check has the equivalent code in Rust:

let mut myvar: Option<i64> = None;
myvar = Some(42);

if myvar < Some(1) {
    println!("Error");
}

Otherwise doing myvar = 42 will simply result in "expected `Option<i64>`, found integer" - this is how an actual strongly typed languages deal with distinction between bare int vs Optional[int] (while also allowing comparison between different optional values explicitly without extra guard against None whenever not necessary). This example unfortunately demonstrates how Python's type hinting currently is still quite brittle and limiting, and how this can potentially result in unexpected outcomes when code might be refactored later.