zimscraperlib

Collection of python code to re-use across python-based scrapers

Usage

• This library is meant to be installed via PyPI (zimscraperlib).
• Make sure to reference it using a version code as the API is subject to frequent changes.
• API should remain the same only within the same minor version.

Example usage:

zimscraperlib>=1.1,<1.2

See documentation at Read the Docs for details.

Warning

While this library brings support for downloading videos with yt-dlp, recent changes in Youtube have forced yt-dlp team to require new dependencies for youtube videos (see yt-dlp/yt-dlp#15012). These dependencies are significantly big and not needed for all other backend supported by yt-dlp (only youtube needs it). These dependencies are hence not included in this library dependencies (yet, see #268), you have to install them on your own if you intend to download videos from Youtube.

Dependencies

• libmagic
• wget
• libzim (auto-installed, not available on Windows)
• Pillow
• FFmpeg
• gifsicle (>=1.92)
• libcairo (if you use the image manipulation, this is used for svg conversion)

macOS

brew install libmagic wget libtiff libjpeg webp little-cms2 ffmpeg gifsicle

Linux

sudo apt install libmagic1 wget ffmpeg \
    libtiff5-dev libjpeg8-dev libopenjp2-7-dev zlib1g-dev \
    libfreetype6-dev liblcms2-dev libwebp-dev tcl8.6-dev tk8.6-dev python3-tk \
    libharfbuzz-dev libfribidi-dev libxcb1-dev gifsicle

Alpine

apk add ffmpeg gifsicle libmagic wget libjpeg

Contribution

This project adheres to openZIM's Contribution Guidelines.

This project has implemented openZIM's Python bootstrap, conventions and policies v1.0.2.

pip install hatch
pip install ".[dev]"
pre-commit install
# For tests
invoke coverage

Users

Non-exhaustive list of scrapers using it (check status when updating API):

• openzim/freecodecamp
• openzim/gutenberg
• openzim/ifixit
• openzim/kolibri
• openzim/nautilus
• openzim/nautilus
• openzim/openedx
• openzim/sotoki
• openzim/ted
• openzim/warc2zim
• openzim/wikihow
• openzim/youtube