New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[GSoC 2021] Integrate Apache Airflow to manage Crawler jobs #102

Open

beshiniii opened this issue Jun 5, 2021 · 0 comments

Assignees

Labels

Projects

GSoC 2021 - Improve CrawlerX Platform

Contributor

beshiniii commented Jun 5, 2021

Is your feature request related to a problem? Please describe.

Airflow is a platform created by the community to programmatically author, schedule and monitor workflows. It provides following advantages.

Easy to manage workflows
Guranteed delivery
Easy to monitor workflows
Easy to schedule Crawler jobs
Supports many many Message broker implementations
Container native support
Dashboard support

sajithaliyanage assigned beshiniii

sajithaliyanage added the gsoc2021 label

sajithaliyanage added this to Backlog in GSoC 2021 - Improve CrawlerX Platform

sajithaliyanage moved this from Backlog to InProgress in GSoC 2021 - Improve CrawlerX Platform

beshiniii mentioned this issue

Adding airflow docker-composer for future reference #117

Merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment