Web Scraping Helper

A lightweight TypeScript library for asynchronous web scraping with customizable prompts and callback support.

Installation

# NPM
npm install scrap-ai

# Yarn
yarn add scrap-ai

# Deno
import { ScrapeClient } from "https://deno.land/x/scrap_ai/mod.ts";

Then import and use:

// ESM/TypeScript
import { ScrapeClient } from "scrap-ai";

// CommonJS
const { ScrapeClient } = require("scrap-ai");

Features

• 🤖 AI-powered data extraction
• 🔄 Asynchronous processing with callback support
• 🔒 Secure webhook verification
• 📦 TypeScript support
• 🌐 Cross-platform (Node.js and Deno)

Usage

The library provides a ScrapeClient class for initiating web scraping operations:

import { ScrapeClient } from "scrap-ai";

// Initialize the client with your API key
const scrapeClient = new ScrapeClient(process.env.SCRAP_API_KEY);

// Basic scraping
await scrapeClient.scrape(
  "https://example.com",
  "Extract all product titles and prices",
  "https://your-api.com/webhook"
);

// Scraping with custom ID
await scrapeClient.scrape(
  "https://example.com",
  "Extract product information",
  "https://your-api.com/webhook",
  "optional-custom-id"
);

API Reference

new ScrapeClient(apiKey)

Creates a new scraping client instance.

Parameters

Parameter	Type	Description
apiKey	string	Your API key for authentication

scrapeClient.scrape(url, prompt, callbackUrl, id?)

Initiates a scraping operation and sends results to the specified callback URL upon completion.

Parameters

Parameter	Type	Description
url	string	The URL of the webpage to scrape
prompt	string	Instructions for what data to extract
callbackUrl	string	URL where results will be sent via POST
id?	string	Optional custom identifier for the scraping request

Webhook Verification

The library provides webhook verification to ensure the authenticity of incoming webhook requests:

const isValid = scrapeClient.verifyWebhook({
  body: req.body,
  signature: req.headers["x-webhook-signature"],
  timestamp: req.headers["x-webhook-timestamp"],
});

scrapeClient.verifyWebhook(options)

Verifies that a webhook request is authentic using timing-safe signature comparison.

Parameters

Parameter	Type	Description
options.body	Object	The raw request body as an object
options.signature	string	The signature from x-webhook-signature header
options.timestamp	string	The timestamp from x-webhook-timestamp header
options.maxAge?	number	Maximum age of webhook in milliseconds (default: 5 minutes)

scrapeClient.parseWebhookBody(body)

Parses and validates the webhook body.

Parameters

Parameter	Type	Description
body	string	The raw webhook body as a string

Returns the parsed and validated webhook event.

Example Usage with Express

Here's a complete example of how to use the scraping client with webhook verification in an Express application:

import { ScrapeClient } from "scrap-ai";
import express from "express";

const app = express();
const scrapeClient = new ScrapeClient(process.env.SCRAP_API_KEY);

// Webhook endpoint
app.post("/webhook", express.json(), (req, res) => {
  const isValid = scrapeClient.verifyWebhook({
    body: req.body,
    signature: req.headers["x-webhook-signature"] as string,
    timestamp: req.headers["x-webhook-timestamp"] as string,
  });

  if (!isValid) {
    return res.status(400).send("Invalid webhook signature");
  }

  const event = scrapeClient.parseWebhookBody(JSON.stringify(req.body));
  console.log("Received verified webhook:", event);

  res.status(200).send("OK");
});

// Start scraping
app.post("/start-scrape", async (req, res) => {
  try {
    const result = await scrapeClient.scrape(
      "https://example.com",
      "Extract product information",
      "https://your-api.com/webhook"
    );
    res.json(result);
  } catch (error) {
    res.status(500).json({ error: "Scraping failed" });
  }
});

License

This project is licensed under the MIT License - see the LICENSE file for details.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.