🕷️ Crawlio JS SDK

crawlio-js is a Node.js SDK for interacting with the Crawlio web scraping and crawling API. It provides programmatic access to scraping, crawling, and batch processing endpoints with built-in error handling.

Visit Crawlio See Docs

---

📦 Installation

npm install crawlio.js

---

🚀 Getting Started

import { Crawlio } from 'crawlio.js'

const client = new Crawlio({ apiKey: 'your-api-key' })

const result = await client.scrape({ url: 'https://example.com' })
console.log(result.html)

---

🔧 Constructor

new Crawlio(options: CrawlioOptions)

Creates a new Crawlio client.

Options:

Name	Type	Required	Description
apiKey	string	✅	Your Crawlio API key
baseUrl	string	❌	API base URL (default: https://crawlio.xyz)

---

📘 API Methods

scrape(options: ScrapeOptions): Promise<ScrapeResponse>

Scrapes a single page.

await client.scrape({ url: 'https://example.com' })

ScrapeOptions:

Name	Type	Required	Description
url	string	✅	Target URL
exclude	string[]	✅	CSS selectors to exclude
includeOnly	string[]	❌	CSS selectors to include
markdown	boolean	❌	Convert HTML to Markdown
returnUrls	boolean	❌	Return all discovered URLs
workflow	Workflow[]	❌	Custom workflow steps to execute
normalizeBase64	boolean	❌	Normalize base64 content
cookies	CookiesInfo[]	❌	Cookies to include in the request
userAgent	string	❌	Custom User-Agent header for the request

---

crawl(options: CrawlOptions): Promise<CrawlResponse>

Initiates a site-wide crawl.

CrawlOptions:

Name	Type	Required	Description
url	string	✅	Root URL to crawl
count	number	✅	Number of pages to crawl
sameSite	boolean	❌	Limit crawl to same domain
patterns	string[]	❌	URL patterns to match
exclude	string[]	❌	CSS selectors to exclude
includeOnly	string[]	❌	CSS selectors to include

---

crawlStatus(id: string): Promise<CrawlStatusResponse>

Checks the status of a crawl job.

---

crawlResults(id: string): Promise<{ results: ScrapeResponse[] }>

Gets results from a completed crawl.

---

search(query: string, options?: SearchOptions): Promise<SearchResponse>

Performs a search on scraped content.

SearchOptions:

Name	Type	Description
site	string	Limit search to a specific domain

---

batchScrape(options: BatchScrapeOptions): Promise<BatchScrapeResponse>

Initiates scraping for multiple URLs in one request.

BatchScrapeOptions:

Name	Type	Description
url	string[]	List of URLs
options	Omit<ScrapeOptions, 'url'>	Common options for all URLs

---

batchScrapeStatus(batchId: string): Promise<BatchScrapeStatusResponse>

Checks the status of a batch scrape job.

---

batchScrapeResult(batchId: string): Promise<{ results: { id: string; result: ScrapeResponse } }>

Fetches results from a completed batch scrape.

---

🛑 Error Handling

All Crawlio errors extend from CrawlioError. You can catch and inspect these for more context.

Error Types:

• CrawlioError
• CrawlioRateLimit
• CrawlioLimitExceeded
• CrawlioAuthenticationError
• CrawlioInternalServerError
• CrawlioFailureError

---

📄 Types

ScrapeResponse

{
  jobId: string
  html: string
  markdown: string
  meta: Record<string, string>
  urls?: string[]
  url: string
}

CrawlStatusResponse

{
  id: string
  status: 'IN_QUEUE' | 'RUNNING' | 'LIMIT_EXCEEDED' | 'ERROR' | 'SUCCESS'
  error: number
  success: number
  total: number
}

CookiesInfo

{
  name: string
  value: string
  path: string
  expires?: number
  httpOnly: boolean
  secure: boolean
  domain: string
  sameSite: 'Strict' | 'Lax' | 'None'
}

---