Pylance doesn't recognize a variable's type from certain kinds of assert statements

[ElijahSink]

Environment data

• Language Server version: v2020.10.3
• OS and version: Win10 running in WSL 2
• Python version: 3.8.6

Expected behaviour

Pylint should constrain the variable's type based on the assertion statement, like this:

def foo(bar: Dict):
    key = bar.get("a")
    FOO_MAP = {
        'b': 'somestr',
        'c': 'otherstr'
    }
    assert isinstance(key, str)  # previously key's type was Unknown | None, now it's Unknown (why not str?)
    return FOO_MAP[key]          # pylint correctly shows no error

Actual behaviour

def foo(bar: Dict):
    key = bar.get("a")
    FOO_MAP = {             # type is Dict[str, str]              
        'b': 'somestr',
        'c': 'otherstr'
    }
    assert key in FOO_MAP   # for this assertion to pass, key must be a str
    return FOO_MAP[key]     # error: 
                            #  Argument of type "Unknown | None" cannot be assigned to parameter "k" of type "str" in function "__getitem__"
                            #    Type "Unknown | None" cannot be assigned to type "str"
                            #     Type "None" cannot be assigned to type "str" Pylance (reportGeneralTypeIssues)

If we have a lot of data coming in from external sources, we need a way of communicating to Pylance that a variable is of a certain type, and preferably in the most concise way possible. Naturally, I could say

assert isinstance(key, str)
assert key in FOO_MAP

but it's irksome to write that extra line (or just an and expression) when the information is implied by the type of FOO_MAP.

Furthermore, it seems that even straightforward assertions like

assert isinstance(key, str) # key is Unknown (not str?)

are communicating no more information than a very loose

assert key is not None   # key is Unknown (expected)

Why is that? Is this expected behavior?

Logs

Will provide if necessary.

[jp-diegidio]

Pylint is too forgiving (how is it that you do not get pyright instead under pylance? I have disabled all other linters...) but your bar: Dict is only partially specified (it boils down to Dict[Unknown, Unknown]): try bar: Dict[str, str] instead, then bar.get (without a default) will give you type str | None.

[ElijahSink]

That would work, except in reality bar is not a Dict[str, str], it has values of mixed type, e.g., Dict[str, Unknown]. I'm hoping that I can use assertions to narrow the type of a specific value from that dictionary that I know will be a string.

[erictraut]

Thanks for the bug report. There is an inconsistency in the way pylance is applying type narrowing to an Any/Unknown type and a type that is a union that happens to contain an Any/Unknown type. In your case, the type inferred for key happens to be Union[Unknown, None]. If you annotate the type of key to Any, you will get the behavior you're looking for. I'll work on making this behavior consistent for unions.

[ElijahSink]

What's the meaning of Unknown compared with Any?

Indeed, when I annotate the type of key to Any, there is no error, even when I remove all assertions statements (so that doesn't seem to explain the difference in type narrowing behavior between the two types of asserts that I mentioned). Why did the assert isinstance(key, str) work, where the other assertion didn't?

I see what you mean about the inconsistent behavior with regard to unions...

    ...
    key = bar.get("a") 
    # pylance says key is Unknown | None
    assert isinstance(key, str) 
    # after this pylance says key is Unknown
    assert isinstance(key, str)
    # now pylance says key is str
    return FOO_MAP[key]

Perhaps you could treat the type narrowing procedure as a fixed-point algorithm?

[erictraut]

Any is defined in PEP 484, and it is used in a type annotation to indicate that any value can be assigned to that symbol. Unknown is a special form of Any that pylance uses. It is an "implicit Any" — one that was not explicitly provided by the program. For example, Unknown is used in cases where you call a function whose return type is not annotated. In most cases, Unknown and Any are treated the same. This is the case with isinstance type narrowing.

The reason a double assert works here is because the first assert narrows the type from Union[Unknown, None] to just Unknown. The second assert then further narrows it to str. The first assert should have been sufficient.

[ElijahSink]

Thanks for the clarification!

[erictraut]

I've updated the isinstance and issubclass type narrowing logic to be more consistent when handling union and non-union source types. With this change, no error is reported for your code example above.

This fix will be included in the next release of pylance.

Thanks again for reporting the problem.

[ElijahSink]

Glad to hear it!
Does Pylance support type narrowing based on assertions like

def foo(bar: Dict):
    key = bar.get("a")
    FOO_MAP = {             # type is Dict[str, str]              
        'b': 'somestr',
        'c': 'otherstr'
    }
    assert key in FOO_MAP   # for this assertion to pass, key must be a str
    return FOO_MAP[key]

In this case, the type of key can be certainly known given that it's confirmed to be present in a dictionary with a known key type.

[erictraut]

No, this isn't support. In general, "type narrowing" refers to a process by which a known type is made "narrower" either by eliminating subtypes of a union or by choosing a subclass of a provided class. For example, str is narrower than Union[str, int], and int is narrower than float.

The Any type is not typically narrowed. The one exception is with the isinstance call, which can narrow Any to a specific type.

[ElijahSink]

The Any type is not typically narrowed. The one exception is with the isinstance call, which can narrow Any to a specific type.

This is surprising to me, since the example I gave has the same narrowing power as a call to isinstance. It's not a huge issue by any means, but my tendency would be to want to narrow types whenever possible, and Any appears to symbolize the union of all existing or hypothetical types.

Pylance has been quite helpful in pointing out ambiguities and outright wrong code in our projects, so I'm thankful for all the work you all do!

[erictraut]

No, this case is quite different. The implementation of the in operator is very complex. It involves the evaluation of several "magic methods" (__contains__, __iter__, __next__) on the right-hand operand. If the type of that left-hand operand is unknown, it is not possible for the type checker to determine whether these methods are supported. So no narrowing can be safely done in these cases.

[jakebailey]

This issue has been fixed in version 2020.11.0, which we've just released. You can find the changelog here: https://github.com/microsoft/pylance-release/blob/master/CHANGELOG.md#2020110-4-november-2020