Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cope with unexpected signature #277

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Cope with unexpected signature #277

wants to merge 2 commits into from

Conversation

peterjc
Copy link

@peterjc peterjc commented Jul 3, 2020

Fixes a regression in numpydoc v1.1.0 compared to v1.0.0 which did not have the _clean_text_signature function, manifested as:

Handler <function mangle_signature at 0x7f8ae571b620> for event 'autodoc-process-signature' threw an exception

I don't have a minimal test case, something strange in Biopython's Bio.Restriction triggered this - a module which creates a vast number of classes which we currently try to exclude from the Sphinx API docs via mocking the import. See biopython/biopython#3024

Fixes a regression in numpydoc v1.1.0 compared to v1.0.0 which did not have the _clean_text_signature function.
@codecov-commenter
Copy link

codecov-commenter commented Jul 3, 2020

Codecov Report

Merging #277 into master will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #277      +/-   ##
==========================================
+ Coverage   93.15%   93.17%   +0.01%     
==========================================
  Files           7        7              
  Lines        1257     1260       +3     
==========================================
+ Hits         1171     1174       +3     
  Misses         86       86              

User defined classes can do strange things.
@rossbar
Copy link
Contributor

rossbar commented Jul 3, 2020

@peterjc I know you said you don't have a minimal test case, but is there a link to the full CI failure somewhere? It'd be nice to be able to see the full traceback (if it exists). I looked for it in the CI logs on biopython/biopython#3022 but didn't see anything docs/sphinx related

@peterjc
Copy link
Author

peterjc commented Jul 3, 2020

Example from TravisCI with minimal clues: https://travis-ci.org/github/biopython/biopython/jobs/704617159

From running locally on macOS with more verbose settings and the following for debuging:

def _clean_text_signature(sig):
    if sig is None:
        return None
    start_pattern = re.compile(r"^[^(]*\(")
    try:
        start, end = start_pattern.search(sig).span()
    except TypeError:
        raise TypeError("Got %r of type %s in _clean_text_signature" % (sig, type(sig))) from None
    start_sig = sig[start:end]
    sig = sig[end:-1]
    sig = re.sub(r'^\$(self|module|type)(,\s|$)','' , sig, count=1)
    sig = re.sub(r'(^|(?<=,\s))/,\s\*', '*', sig, count=1)
    return start_sig + sig + ')'

Got:

...
[app] emitting event: 'build-finished'(ExtensionError("Handler <function mangle_signature at 0x7f9d7c2f2290> for event 'autodoc-process-si

Traceback (most recent call last):
  File "/Users/xxx/miniconda3/lib/python3.7/site-packages/sphinx/events.py", line 110, in emit
    results.append(listener.handler(self.app, *args))
  File "/Users/xxx/miniconda3/lib/python3.7/site-packages/numpydoc/numpydoc.py", line 208, in mangle_signature
    or _clean_text_signature(getattr(obj, '__text_signature__', None)))
  File "/Users/xxx/miniconda3/lib/python3.7/site-packages/numpydoc/numpydoc.py", line 221, in _clean_text_signature
    raise TypeError("Got %r of type %s in _clean_text_signature" % (sig, type(sig))) from None
TypeError: Got Bio.Restriction.Restriction.__text_signature__ of type <class 'Bio.Restriction.Restriction.__text_signature__'> in _clean_text_signature

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/xxx/miniconda3/lib/python3.7/site-packages/sphinx/cmd/build.py", line 280, in build_main
    app.build(args.force_all, filenames)
  File "/Users/xxx/miniconda3/lib/python3.7/site-packages/sphinx/application.py", line 348, in build
    self.builder.build_update()
  File "/Users/xxx/miniconda3/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 299, in build_update
    len(to_build))
  File "/Users/xxx/miniconda3/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 311, in build
    updated_docnames = set(self.read())
  File "/Users/xxx/miniconda3/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 418, in read
    self._read_serial(docnames)
  File "/Users/xxx/miniconda3/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 439, in _read_serial
    self.read_doc(docname)
  File "/Users/xxx/miniconda3/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 479, in read_doc
    doctree = read_doc(self.app, self.env, self.env.doc2path(docname))
  File "/Users/xxx/miniconda3/lib/python3.7/site-packages/sphinx/io.py", line 221, in read_doc
    pub.publish()
  File "/Users/xxx/miniconda3/lib/python3.7/site-packages/docutils/core.py", line 218, in publish
    self.settings)
  File "/Users/xxx/miniconda3/lib/python3.7/site-packages/sphinx/io.py", line 126, in read
    self.parse()
  File "/Users/xxx/miniconda3/lib/python3.7/site-packages/docutils/readers/__init__.py", line 77, in parse
    self.parser.parse(self.input, document)
  File "/Users/xxx/miniconda3/lib/python3.7/site-packages/sphinx/parsers.py", line 102, in parse
    self.statemachine.run(inputlines, document, inliner=self.inliner)
  File "/Users/xxx/miniconda3/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 171, in run
    input_source=document['source'])
  File "/Users/xxx/miniconda3/lib/python3.7/site-packages/docutils/statemachine.py", line 242, in run
    context, state, transitions)
  File "/Users/xxx/miniconda3/lib/python3.7/site-packages/docutils/statemachine.py", line 459, in check_line
    return method(match, context, next_state)
  File "/Users/xxx/miniconda3/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 2769, in underline
    self.section(title, source, style, lineno - 1, messages)
  File "/Users/xxx/miniconda3/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 327, in section
    self.new_subsection(title, lineno, messages)
  File "/Users/xxx/miniconda3/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 395, in new_subsection
    node=section_node, match_titles=True)
  File "/Users/xxx/miniconda3/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 282, in nested_parse
    node=node, match_titles=match_titles)
  File "/Users/xxx/miniconda3/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 196, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/Users/xxx/miniconda3/lib/python3.7/site-packages/docutils/statemachine.py", line 242, in run
    context, state, transitions)
  File "/Users/xxx/miniconda3/lib/python3.7/site-packages/docutils/statemachine.py", line 459, in check_line
    return method(match, context, next_state)
  File "/Users/xxx/miniconda3/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 2342, in explicit_markup
    nodelist, blank_finish = self.explicit_construct(match)
  File "/Users/xxx/miniconda3/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 2354, in explicit_construct
    return method(self, expmatch)
  File "/Users/xxx/miniconda3/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 2097, in directive
    directive_class, match, type_name, option_presets)
  File "/Users/xxx/miniconda3/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 2146, in run_directive
    result = directive_instance.run()
  File "/Users/xxx/miniconda3/lib/python3.7/site-packages/sphinx/ext/autodoc/directive.py", line 146, in run
    documenter.generate(more_content=self.content)
  File "/Users/xxx/miniconda3/lib/python3.7/site-packages/sphinx/ext/autodoc/__init__.py", line 829, in generate
    sig = self.format_signature()
  File "/Users/xxx/miniconda3/lib/python3.7/site-packages/sphinx/ext/autodoc/__init__.py", line 432, in format_signature
    self.object, self.options, args, retann)
  File "/Users/xxx/miniconda3/lib/python3.7/site-packages/sphinx/events.py", line 127, in emit_firstresult
    for result in self.emit(name, *args, allowed_exceptions=allowed_exceptions):
  File "/Users/xxx/miniconda3/lib/python3.7/site-packages/sphinx/events.py", line 118, in emit
    (listener.handler, name)) from exc
sphinx.errors.ExtensionError: Handler <function mangle_signature at 0x7f9d7c2f2290> for event 'autodoc-process-signature' threw an exception

Extension error:
Handler <function mangle_signature at 0x7f9d7c2f2290> for event 'autodoc-process-signature' threw an exception
make: *** [html] Error 2
ERROR: InvocationError for command /usr/bin/make -C Doc/api/ html (exited with code 2)

@m-melis
Copy link

m-melis commented Jul 4, 2020

This will possibly also fix #276 👍

@peterjc
Copy link
Author

peterjc commented Jul 4, 2020

Yes, I think this would fix #276 too - hopefully that might give a more useful test case?

@thequackdaddy
Copy link
Contributor

@peterjc What is the value of sig when this fails? It could be a bytes object.

It might be better to convert the bytes object to a string...

if isinstance(sig, bytes):
    sig = sig.decode()

@peterjc
Copy link
Author

peterjc commented Jul 4, 2020

I failed to work out what exactly the object was in my case, something created dynamically was my hunch.

Handling bytes explicitly certainly wouldn't hurt though.

@jnothman
Copy link
Member

jnothman commented Jul 5, 2020 via email

@peterjc
Copy link
Author

peterjc commented Jul 5, 2020

That sounds good - do you want me to try that, or can I leave this with you?

@thequackdaddy
Copy link
Contributor

thequackdaddy commented Jul 5, 2020

I experimented with this for a bit. It looks like the troublesome problem is that Bio.Restriction.Restriction is actually a module. Thus, I don't think it should even call the whole mangle_signature bit. Do modules have signatures?

I found editing numpydoc.numpydoc.mange_signature as follows fixed it...

(I only added the check if obj is a ModuleType. Remember to from types import ModuleType somewhere.)

def mangle_signature(app, what, name, obj, options, sig, retann):
    # Do not try to inspect classes that don't define `__init__`
    if (inspect.isclass(obj) and
        (not hasattr(obj, '__init__') or
            'initializes x; see ' in pydoc.getdoc(obj.__init__))):
        return '', ''

    if isinstance(obj, ModuleType):
        return

    if not (isinstance(obj, Callable) or
            hasattr(obj, '__argspec_is_invalid_')):
        return

    if not hasattr(obj, '__doc__'):
        return
    doc = get_doc_object(obj, config={'show_class_members': False})
    sig = (doc['Signature']
           or _clean_text_signature(getattr(obj, '__text_signature__', None)))
    if sig:
        sig = re.sub("^[^(]*", "", sig)
        return sig, ''

@jnothman
Copy link
Member

jnothman commented Jul 6, 2020

Do classes without __init__ not sometimes have meaningful __new__ definitions?

@peterjc
Copy link
Author

peterjc commented Jul 12, 2020

Following testing on biopython/biopython#3045 it seems we only see the problem with the restriction code when using autodoc_mock_imports = [..., "Bio.Restriction.Restriction"]

Does mock do something predictable to the signatures?

@m-melis
Copy link

m-melis commented Jul 13, 2020

Mock could be the right track to fix the problem. We use autodoc_mock_imports = ["pytest", "torch", "torchvision", "cleverhans", "tensorflow"] in our config (the one is crashing with #276 ).

Base automatically changed from master to main March 5, 2021 12:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants