Welcome to site-scraper-node 👋

A NodeJS program created to scrape URLs from dynamically rendered web apps without the need of spawning headless browsers. It uses much less resources then headless browser alternatives and im currently using this to generate sitemaps for a production web app.

Install

npm install

Configuration

Move config.example.js to config.js
Change baseUrl to the websites base url
Change startUrl to the websites url you want to start scraping on

Usage

npm start

Author

👤 Hayden

• Twitter: @itsheydoon
• Github: @haydencarlson

Show your support

Give a ⭐️ if this project helped you!