TypeError, when sphinx.ext.viewcode is enabled

[crazyscientist]

Describe the bug

For our project, we have both the autodoc and viewcode extensions enabled, and the project makes heavy use of lxml.

When building the docs, a TypeError: cannot pickle 'lxml.objectify._ObjectifyElementMakerCaller' object occurs.

Without deeper analysis, a quick workaround is to disable the viewcode extension.

How to Reproduce

Check out the project, install the dependencies and run Sphinx:

git clone https://github.com/SUSE/osc-tiny.git
cd osc-tiny
git checkout v0.10.5
pip install -r requirements.txt -r requirements_devel.txt
cd doc
make html

Environment Information

Platform:              linux; (Linux-5.15.0-124-generic-x86_64-with-glibc2.35)
Python version:        3.12.7 (main, Oct  1 2024, 08:52:12) [GCC 11.4.0])
Python implementation: CPython
Sphinx version:        8.1.3
Docutils version:      0.21.2
Jinja2 version:        3.1.4
Pygments version:      2.18.0

Sphinx extensions

extensions = [
    'sphinx.ext.autodoc',
    'sphinx.ext.viewcode',
]

Additional context

This is the full content of the log file generated

# Platform:         linux; (Linux-5.15.0-124-generic-x86_64-with-glibc2.35)
# Sphinx version:   8.1.3
# Python version:   3.12.7 (CPython)
# Docutils version: 0.21.2
# Jinja2 version:   3.1.4
# Pygments version: 2.18.0

# Last messages:
#   
#   
#   reading sources... [100%]
#   releasenotes
#   
#   
#   looking for now-outdated files...
#   none found
#   pickling environment...
#   failed

# Loaded extensions:
#   sphinx.ext.mathjax (8.1.3)
#   alabaster (1.0.0)
#   sphinxcontrib.applehelp (2.0.0)
#   sphinxcontrib.devhelp (2.0.0)
#   sphinxcontrib.htmlhelp (2.1.0)
#   sphinxcontrib.serializinghtml (2.0.0)
#   sphinxcontrib.qthelp (2.0.0)
#   sphinx.ext.autodoc.preserve_defaults (8.1.3)
#   sphinx.ext.autodoc.type_comment (8.1.3)
#   sphinx.ext.autodoc.typehints (8.1.3)
#   sphinx.ext.autodoc (8.1.3)
#   sphinx.ext.viewcode (8.1.3)
#   sphinxcontrib.jquery (4.1)
#   sphinx_rtd_theme (unknown version)

# Traceback:
Traceback (most recent call last):
  File "/home/andi/virtualenvs/osctiny/lib/python3.12/site-packages/sphinx/cmd/build.py", line 514, in build_main
    app.build(args.force_all, args.filenames)
  File "/home/andi/virtualenvs/osctiny/lib/python3.12/site-packages/sphinx/application.py", line 381, in build
    self.builder.build_update()
  File "/home/andi/virtualenvs/osctiny/lib/python3.12/site-packages/sphinx/builders/__init__.py", line 358, in build_update
    self.build(
  File "/home/andi/virtualenvs/osctiny/lib/python3.12/site-packages/sphinx/builders/__init__.py", line 404, in build
    pickle.dump(self.env, f, pickle.HIGHEST_PROTOCOL)
TypeError: cannot pickle 'lxml.objectify._ObjectifyElementMakerCaller' object

[jayaddison]

Thanks @crazyscientist - could you try to narrow down a more-minimal repro case for this? It seems that lxml and sphinx.ext.viewcode will be required -- and something, presumably extension-related, must be adding a reference to lxml in the BuildEnvironment.. but I can't figure out where yet.

[crazyscientist]

Hi @jayaddison , thank you for the encouragement.

I managed to find the culprit and create a shorter example: https://gist.github.com/crazyscientist/7d53d148efc9cbe8d48756bf25c8fc3d

Steps to reproduce the problem with this gist:

1. Create a new Python project (containing models.py) from the gist
2. Create a venv and install Sphinx and lxml
3. Create a Sphinx project inside the Python project (e.g. with sphinx-quickstart)
4. Replace files in the Sphinx projects with files from the gist
   • source_conf.py -> source/conf.py
   • source_index.rst -> source/index.rst
5. Call make html (which will cause the pickling error)

No errors will be raised, if

• sphinx.ext.viewcode is removed from the extensions list
• line 11 of models.py is commented out

[jayaddison]

Ok, getting closer - thanks @crazyscientist! Does an error continue to occur if ObjectifiedElement is -- separately -- removed? (indepdently from the L11 commenting-out)

(apologies for not actually testing that yet myself.. I may try to soon, but have gotten into too many separate tasks/threads recently)

[crazyscientist]

@jayaddison No worries, happy to help out 🙂

If I remove ObjectifiedElement (both, in the import statement and type annotation), the error message does not change.

If there is an instance of ElementMaker in the file, the exception will be raised. Otherwise, the docs get built as expected.

I don't know whether it is worth noting: Only commenting out line 11, in which ElementMaker() is initialized, allows the docs to be built. The import of ElementMaker in line 8 does not seem to cause a problem.

[jayaddison]

Enabling the autodoc and viewcode extensions, assigning a single module-level ElementMaker variable, and then attempting to autodata-document that variable seems to be enough to replicate this.

The part of the BuildEnvironment that seems to be failing to pickle is the _viewcode_modules attribute.. and in particular a modname that is in fact an lxml.objectify._ObjectifyElementMakerCaller object, not a string as the code seems designed for.

That's created/returned somewhere around here:

sphinx/sphinx/ext/viewcode.py

Lines 57 to 64 in a1510de

	# Allow an attribute to have multiple parts and incidentally allow
	# repeated .s in the attribute.
	value = module
	for attr in attribute.split('.'):
	if attr:
	value = getattr(value, attr)
	return getattr(value, '__module__', None)

📓 For anyone investigating this: the --pdb command-line option for sphinx-build is quite useful!