How to provide stubs to be used by global_annotations_decorator?

[jolaf]

For example, I have typeshed, a project containing stubs for Python standard library.
And I'd like pytypes to check that my code calls standard library functions according to their annotations.
How do I configure pytypes to use the annotations in typeshed?

Whatever I tried didn't work.

[Stewori]

I never found time to deal with @overload, and pytypes might crash on loading stub files that contain it. So typeshed may be problematic. Further, stubfiles are searched in the same directory as the corresponding py-file or in the list of folders pytypes.stub_path. I know, nowadays there is the specification PEP-561, but the pytypes.stub_path-solution predates that specification. Once there was #41 but the proposed solution was withdrawn and I never found time to resume that issue. For now, pytypes.stub_path must do it.

[jolaf]

Problem with @overload is a separate issue, I agree.

But the problem I hit was with file locations, not content.

For example, I have a typeshed-like project of my own where I put the stubs I make for 3-rd party libraries. For exampe, I have stubs there for a simple one-module library netifaces that is installed system-wide by root. So, the stubs for it turn on to be in ~/my-typeshed/netifaces/__init__.pyi. However when I do import netifaces, pytypes for some reason looks for (I inserted prints to find out) netifaces.cpython-36m-x86_64-linux-gnu.pyi.

pytypes.stub_path wouldn't help with it, moreover, if stubs were in multiple files, and/or with submodules, I'd have to manually add each submodule to stub_path.

So it seems there's a need for some reasonable logic for looking for stubs, something that could handle:

• A single directory with subdirectories for packages/modules;
• Subdirectories for submodules;
• module/__init__.pyi as well as module.pyi;
• typeshed hierarchy, like stdlib/3.6, probably like a special option.

Probably the logic mypy uses would do, as it's already adopted by and familiar to those using mypy which is probably the most important consumer of .pyi stubs for now:
https://mypy.readthedocs.io/en/latest/running_mypy.html#how-mypy-handles-imports

[Stewori]

cpython-36m-x86_64-linux-gnu

Interesting. Do you have an idea where this string comes from? Is a C-extension involved? Pytypes locates the stubfile based on the __file__ attribute of the module object. The full logic is a nightmare - I always hated the complexity required to get this seemingly simple matching right.

See these functions:
https://github.com/Stewori/pytypes/blob/master/pytypes/util.py#L69
https://github.com/Stewori/pytypes/blob/master/pytypes/stubfile_manager.py#L107
https://github.com/Stewori/pytypes/blob/master/pytypes/stubfile_manager.py#L92
https://github.com/Stewori/pytypes/blob/master/pytypes/stubfile_manager.py#L152

Can you spot anything that causes cpython-36m-x86_64-linux-gnu?

[jolaf]

It probably comes from __file__.
Yes, it's a C-extension module.
It's one more reason why it needs stubs. :)

$ python3
Python 3.6.8 (default, Jan 14 2019, 11:02:34) 
[GCC 8.0.1 20180414 (experimental) [trunk revision 259383]] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import netifaces
>>> netifaces.__file__
'/usr/lib/python3/dist-packages/netifaces.cpython-36m-x86_64-linux-gnu.so'

[Stewori]

Okay, perfect. Then we need better code to clean the filename, right?
Can you suggest a genuine line of Python to accomplish this?

[jolaf]

Probably
https://github.com/Stewori/pytypes/blob/master/pytypes/util.py#L71
should be

bn = os.path.basename(module.__file__).partition('.')[0]

instead of rpartition.

[Stewori]

I wonder why I used rpartition in first place. Can there be a case where the proper module name would contain a dot? Anyway, thanks for spotting this! If you like, file a PR to be accordingly credited.

Otherwise I can add this fix. However, I am currently working on another step towards 3.7 support. I prefer not to mix up work between distinct issues in my local branch, so I would fix this only after committing my current bunch of work. This might take until end of next week.

[jolaf]

I wonder why use .__file__ in the first place, while there's .__name__.

[Stewori]

I remember there were issues with __name__. It was certainly also my first choice. At very least, the main module has __main__ as its __name__ which is useless for finding the stubfile. There may have been further issues why __name__ was not reliable.

[jolaf]

I wonder how mypy resolves this. For example:
https://github.com/python/typeshed/tree/master/stdlib/3/os
Contains path.pyi stubs for os.path module.
However, on Windows:

C:\>python3
Python 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:54:40) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os.path
>>> os.path.__name__
'ntpath'
>>> os.path.__file__
'C:\\Python36\\lib\\ntpath.py'

and on Ubuntu:

$ python3
Python 3.6.8 (default, Jan 14 2019, 11:02:34) 
[GCC 8.0.1 20180414 (experimental) [trunk revision 259383]] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os.path
>>> os.path.__name__
'posixpath'
>>> os.path.__file__
'/usr/lib/python3.6/posixpath.py'

In other words, it seems that the name used to import a module can be different from the name visible through __name__ and __file__.

[Stewori]

Maybe in os there is a platform-specific alias? However, querying such aliases would be tedious if possible at all. I guess the best solution would be a user-configurable lookup table for such "special cases". At least mypy does not have an issue with __main__ as itself takes that role.

[Stewori]

I committed the change you suggest in #78 (comment). Does it fix that particular aspect of this issue for you?
(Certainly the overall solution of this issue is WIP...)

[jolaf]

Yep, thanks.