Add TrafilaturaExtractor class
#431
Conversation
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
| expected = [ | ||
| "Let's keep this paragraph: either came does last new took taken making became from.", | ||
| ] |
There was a problem hiding this comment.
Trafilatura has a really bad bug where it is returning the string twice. I can double check all of my logic, but in the case that this is a Trafilatura-specific issue, I am debating whether I should add our own exact deduplication code into this.
(Trafilatura actually has their own deduplicate parameter here, but it did not fix this use case.)
Possible related issue: adbar/trafilatura#634
There was a problem hiding this comment.
I have not scoped to see what conditions cause this.
There was a problem hiding this comment.
Another issue: adbar/trafilatura#768
There was a problem hiding this comment.
Solved by adding support for these extraction and deduplication parameters: https://trafilatura.readthedocs.io/en/latest/settings.html.
Signed-off-by: Sarah Yurick <[email protected]>
Closes #400.