Crawl a Website

Crawl a website and its linked pages to extract comprehensive data. For detailed usage, check out the Crawl API Reference

By default, the crawl skips sublinks that aren’t part of the URL hierarchy you specify. For example, crawling https://example.com/products/ wouldn’t capture pages under https://example.com/promotions/deal-567. To include such links, enable the allowBackwardLinks parameter.

Scrapeless exposes endpoints for starting a crawl request and getting its status and results. By default, crawling is handled asynchronously: start the job first, then check its status until completed. However, with our SDKs, we provide a simple function that handles the entire flow and returns the data once the job is finished.

Installation

npm install @scrapeless-ai/sdk

pnpm add @scrapeless-ai/sdk

Usage

import { ScrapingCrawl } from "@scrapeless-ai/sdk";
 
// Initialize the client
const client = new ScrapingCrawl({
  apiKey: "your-api-key", // Get your API key from https://scrapeless.com
});
 
(async () => {
  const result = await client.crawlUrl(
    "https://example.com",
    {
      limit: 2,
      scrapeOptions: {
        formats: ["markdown", "html", "links"],
        onlyMainContent: false,
        timeout: 15000,
      },
      browserOptions: {
        proxyCountry: "ANY",
        sessionName: "Crawl",
        sessionRecording: true,
        sessionTTL: 900,
      },
    }
  );
 
  console.log(result);
})();

Response

{
  "success": true,
  "status": "completed",
  "completed": 2,
  "total": 2,
  "data": [
    {
      "url": "https://example.com",
      "metadata": {
        "title": "Example Page",
        "description": "A sample webpage"
      },
      "markdown": "# Example Page\nThis is content...",
      ...
    },
    ...
  ]
}

Each crawled page has its own status of completed or failed and can have its own error field, so be cautious of that.

To see the full schema, checkout the API Reference.

Browser Configurations

You can also provide configurations for the session used to execute the scrape job when creating a new session itself; these could include using a proxy.

For a complete list of available parameters, refer to the API Reference or Browser Parameters.

import { ScrapingCrawl } from "@scrapeless-ai/sdk";
 
// Initialize the client
const client = new ScrapingCrawl({
  apiKey: "your-api-key", // Get your API key from https://scrapeless.com
});
 
(async () => {
  const result = await client.crawlUrl(
    "https://example.com",
    {
      limit: 2,
      browserOptions: {
        proxyCountry: "ANY",
        sessionName: "Crawl",
        sessionRecording: true,
        sessionTTL: 900,
      },
    }
  );
 
  console.log(result);
})();

Scrape Configurations

You can also specify optional parameters for the scrape job, such as response formats, enabling main-content-only extraction, setting a maximum page navigation timeout, and more.

import { ScrapingCrawl } from "@scrapeless-ai/sdk";
 
// Initialize the client
const client = new ScrapingCrawl({
  apiKey: "your-api-key", // Get your API key from https://scrapeless.com
});
 
(async () => {
  const result = await client.crawlUrl(
    "https://example.com",
    {
      limit: 2,
      scrapeOptions: {
        formats: ["markdown", "html", "links"],
        onlyMainContent: false,
        timeout: 15000,
      }
    }
  );
 
  console.log(result);
})();

For a full reference on the crawl endpoint, checkout the API Reference.

Batch Scrape Cost Consumption