v3.0.0-alpha.1

HTML2text updates:

1. Moved to Trafilatura 2.0.0
2. Additional extraction of text with markup using xml outputs from Trafilatura
3. Extraction of HTML language tags
4. Streaming input HTMLs directly from LUMIO

Code running stage2 on LUMI for the second data release

See two/README.MD to learn to reproduce stage2 for the second data release

updated langid

langid update: preprocessing, new model
better selected blocksize for trafilatura

v2.0.0-alpha.2

Full Changelog: v2.0.0-alpha.1...v2.0.0-alpha.2

Now pip installable.

v2.0.0-alpha.1

traf.py: timeout