Scrape a Single Page

The Crawl API allows you to get the data you want from web pages with a single call. You can scrape page content and capture its data in various formats.

Scrapeless exposes endpoints for starting a scrape request and for getting it’s status and results. By default, scraping is handled in an asynchronous manner of first starting the job and then checking it’s status until it is completed. However, with our SDKs, we provide a simple function that handles the whole flow and returns the data once the job is completed.

Installation

npm install @scrapeless-ai/sdk
 

pnpm add @scrapeless-ai/sdk
 

Usage

import { ScrapingCrawl } from "@scrapeless-ai/sdk";
 
// Initialize the client
const client = new ScrapingCrawl({
  apiKey: "your-api-key", // Get your API key from https://scrapeless.com
});
 
(async () => {
  const result = await client.scrapeUrl(
    "https://example.com"
  );
 
  console.log(result);
})();
 

Browser Configurations

You can also provide configurations for the session used to execute the scrape job when creating a new session itself; these could include using a proxy.

Scrapeless automatically handles common CAPTCHA types, including reCAPTCHA v2, Cloudflare Turnstile/Challenge.

No additional setup is required—Scrapeless takes care of it during scraping. 👉 For more details, checkout the Captcha Solving.

To see all the different available browser parameters, checkout the API Reference or Browser Parameters.

import { ScrapingCrawl } from "@scrapeless-ai/sdk";
 
// Initialize the client
const client = new ScrapingCrawl({
  apiKey: "your-api-key", // Get your API key from https://scrapeless.com
});
 
(async () => {
  const result = await client.scrapeUrl(
    "https://example.com",
    {
      browserOptions: {
        proxyCountry: "ANY",
        sessionName: "Crawl",
        sessionRecording: true,
        sessionTTL: 900,
      },
    }
  );
 
  console.log(result);
})();
 

Scrape Configurations

You can also specify optional parameters for the scrape job, such as response formats, enabling main-content-only extraction, setting a maximum page navigation timeout, and more.

import { ScrapingCrawl } from "@scrapeless-ai/sdk";
 
// Initialize the client
const client = new ScrapingCrawl({
  apiKey: "your-api-key", // Get your API key from https://scrapeless.com
});
 
(async () => {
  const result = await client.scrapeUrl(
    "https://example.com",
    {
      formats: ["markdown", "html", "links"],
      onlyMainContent: false,
      timeout: 15000,
    }
  );
 
  console.log(result);
})();
 

For a full reference on the scrape endpoint, checkout the API Reference.