Replace overloads in os.path stubs with Unions of TypeVars

[FichteFoll]

Support for this has been added in a recently merged mypy PR (python/mypy#14396) and this commit makes full use of it, getting rid of all overloads in the os.path stubs.

Continues the efforts of #2053 (and the previous attempts referenced there).

Introduces a new TypeAlias GenericOrLiteralPath and replaces functions of the following styles:

@overload
def a(path: PathLike[AnyStr]) -> AnyStr: ...
@overload
def a(path: AnyStr) -> AnyStr: ...

@overload
def b(path: PathLike[AnyStr]) -> AnyStr: ...
@overload
def b(path: AnyOrLiteralStr) -> AnyOrLiteralStr: ...

with:

def a(path: GenericPath[AnyStr]) -> AnyStr: ...
def b(path: GenericPath[AnyOrLiteralStr]) -> AnyOrLiteralStr: ...

Note that quite unexpectedly running mypy_test.py does not result in a failure on the latest stable locally and I don't know why (at least not at this time of day), so running against CI to find out more.

Related: python/mypy#12554

[hauntsaninja]

Note that a) typeshed CI currently runs with mypy 0.991, b) mypy doesn't support LiteralString (it currently mostly just treats it like str), so PRs like this will need careful testing with pyright

[FichteFoll]

Fails in pyright. I'll take a look tomorrow on how to setup a pyright environment for me to test locally.
https://github.com/python/typeshed/actions/runs/3864921546/jobs/6588076084#step:7:30

[FichteFoll]

Problems have been addressed.

What remains to be tested is whether a LiteralString is properly transferred into the return type because the AnyStr (or AnyStr_co) type var(s) used in GenericPath (and PathLike) are restricted to str and bytes respectively, but from shallow thinking a LiteralString should satisfy this.
Unfortunately, I don't really know how to test that at this moment.

mypy_primer results will probably be useless due to mypy not supporting LiteralString as of now.

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

vision (https://github.com/pytorch/vision)
+ torchvision/prototype/datasets/_home.py:10: error: Returning Any from function declared to return "str"  [no-any-return]
+ torchvision/prototype/datasets/_home.py:16: error: Returning Any from function declared to return "str"  [no-any-return]

werkzeug (https://github.com/pallets/werkzeug)
+ src/werkzeug/utils.py:436: error: Value of type variable "AnyOrLiteralStr" of "join" cannot be "Union[str, PathLike[Any], Any]"  [type-var]
+ src/werkzeug/utils.py:436: error: Incompatible types in assignment (expression has type "Union[str, PathLike[Any], Any]", variable has type "Optional[str]")  [assignment]
+ src/werkzeug/utils.py:438: error: Value of type variable "AnyOrLiteralStr" of "abspath" cannot be "Union[str, PathLike[Any], Any]"  [type-var]
+ src/werkzeug/utils.py:438: error: Incompatible types in assignment (expression has type "Union[str, PathLike[Any], Any]", variable has type "Optional[str]")  [assignment]
+ src/werkzeug/utils.py:440: error: Argument 1 to "stat" has incompatible type "Optional[str]"; expected "Union[int, Union[str, bytes, PathLike[str], PathLike[bytes]]]"  [arg-type]
+ src/werkzeug/utils.py:585: error: Argument 1 to "isfile" has incompatible type "Optional[str]"; expected "Union[Union[str, bytes, PathLike[str], PathLike[bytes]], int]"  [arg-type]
+ src/werkzeug/utils.py:591: error: Argument 1 to "send_file" has incompatible type "Optional[str]"; expected "Union[PathLike[Any], str, IO[bytes]]"  [arg-type]

streamlit (https://github.com/streamlit/streamlit)
+ lib/streamlit/file_util.py: note: In function "get_streamlit_file_path":
+ lib/streamlit/file_util.py:137:5: error: Returning Any from function declared to return "str"  [no-any-return]

jinja (https://github.com/pallets/jinja)
+ src/jinja2/environment.py:857: error: Value of type variable "AnyOrLiteralStr" of "join" cannot be "Union[str, PathLike[Any], Any]"  [type-var]

[FichteFoll]

Primer results:

• streamlit is unrelated. https://github.com/streamlit/streamlit/blob/984c80d79ed185df0120d3ff302b10abe5cc3ef5/lib/streamlit/file_util.py#L127-L137
• pytorch is unrelated. https://github.com/pytorch/vision/blob/35f68a09f94b2d7afb3f6adc2ba850216413f28e/torchvision/prototype/datasets/_home.py#L7-L10
• werkzeug and jinja notes all seem to be related to either Any or None being used as a parameter to some path function.
  • perhaps because PathLike is used without a generic parameter? That or it's the old version of mypy.
    https://github.com/pallets/werkzeug/blob/21c0a01a745ec817f05f37ece3e70a321e62260f/src/werkzeug/utils.py#L428-L436
  • jinja not checked, probably the same issue though

[JelleZijlstra]

The streamlit and pytorch changes are definitely not unrelated: they're the direct result of changes made by this PR. What seems to be going on is that mypy now infers Any for cases where it previously inferred a non-Any type, which means that this PR would cause a regression.

[FichteFoll]

Right, it appears to be the same issue as with werkzeug where the typevar is infered to Any.

Unfortunately I have no idea or time to figure out how to proceed with this, for the moment at least.

Edit: Feel free to close the PR if you deem it not useful or want to clean up stale PRs in the future.

[AlexWaygood]

This has a lot of merge conflicts now, and it doesn't look like maintainers have been given permission to push to the PR branch, so I can't resolve them. As such, I'm closing this as stale for now -- but feel free to open a new PR if you'd like to fix the conflicts and see what mypy_primer looks like now that mypy 1.1.1 is out.

[FichteFoll]

it doesn't look like maintainers have been given permission to push to the PR branch

I suspect that Github doesn't support that when creating pull requests from organizations as opposed to your personal account (because I use an organization for management of my forks).