html parsing fail on empty documents

[kmike]

Exception like this can be raised by functions from extruct.utils:

document = parse_xmldom_html(html_string, encoding=encoding)
File "/usr/local/lib/python3.6/dist-packages/extruct/utils.py", line 16, in parse_xmldom_html
return lxml.html.fromstring(html, parser=parser)
File "/usr/local/lib/python3.6/dist-packages/lxml/html/__init__.py", line 876, in fromstring
doc = document_fromstring(html, parser=parser, base_url=base_url, **kw)
File "/usr/local/lib/python3.6/dist-packages/lxml/html/__init__.py", line 765, in document_fromstring
"Document is empty")
lxml.etree.ParserError: Document is empty

In parsel this is worked around: empty documents are handled explicitly. There is also an issue with null bytes handled. I think we should bring similar fixes to extruct. See https://github.com/scrapy/parsel/blob/e01093cf6342c90445028de28034b3cc3d2ead8b/parsel/selector.py#L38.

[mhoangvslev]

It's 2024 and I don't know why this simple fix is not yet implemented in the code

[kmike]

@mhoangvslev it's not implemented because nobody sent a pull request with a fix.

Why did it happen? I guess the issue has not been affecting many people, and if there were affected people, they haven't put the effort to fix the issue. For example, I recall opening this ticket because I found it as a potential issue, and wanted to document it; it was not actually affecting my code.

Of course, it's better to fix it in the repo, so if you feel motivated, pull requests are welcome.

[mhoangvslev]

For future readers, you only need to "clean" the html string using:

    html = text.strip().replace('\x00', '').encode('utf8')