GitHub - pseud039/crawler: Composable web crawler engine with interchangeable components for politeness, fetching, revisiting, and parallelization. Inspired by ECS architecture.

Branches Tags

Name		Name	Last commit message	Last commit date
Latest commit History 13 Commits
packages		packages
.env.example		.env.example
.gitignore		.gitignore
desktop.ini		desktop.ini
docker-compose.yml		docker-compose.yml
package.json		package.json
pnpm-lock.yaml		pnpm-lock.yaml
pnpm-workspace.yaml		pnpm-workspace.yaml
turbo.json		turbo.json

About

Composable web crawler engine with interchangeable components for politeness, fetching, revisiting, and parallelization. Inspired by ECS architecture.