Scrapy-Crawler

System Requirements

• Python 3.10
• pre-commit

Installation Instructions

  pip install -r requirements.txt && pre-commit install

Required Environments

# For scrapy scheduling
SCRAPEOPS_API_KEY =

# For Database connection
DB_HOST = 
DB_PORT = 
DB_USER = 
DB_PASSWORD = 
DB_DATABASE =

# For AWS
AWS_ACCESS_KEY_ID = 
AWS_SECRET_ACCESS_KEY = 
AWS_REGION_NAME = 
AWS_DEFAULT_REGION = 
AWS_LIVE_QUEUE_NAME =

# For Labeling and alerting
SLACK_BOT_LABELING_TOKEN = 
OPENAI_API_KEY =