DiscourseCrawler

Crawls a discourse instance

Usage example

Requires a working Python installed on your machine with the capability to create virtual environments.

1. Clone the repo
2. source bin/init-local.sh
3. bin/discourse_crawler -d db/cientopolis.org.db -u https://forum.cientopolis.org -r

Time estimates

It takes about one hour to crawl the inaturalist.org forum, which has over 160K posts.