Python SDK

Overview

The official Scrapeless Python SDK provides access to browser automation, scraping, crawling, proxies, search results, and AI chat extraction. This guide follows the SDK repository README, with runnable examples and configuration details.

Requirements

Python 3.8 or later is declared in the package metadata. Browser integrations may require a newer Python version depending on the Playwright or Pyppeteer version installed.

Use a virtual environment to keep SDK dependencies separate from other projects.

Installation

pip install scrapeless

The repository uses python-dotenv to load environment variables, but its current package metadata does not list that dependency. Install it alongside the SDK if needed:

pip install python-dotenv

Install pyppeteer or playwright separately when using the corresponding browser integration.

Authentication / API Key

export SCRAPELESS_API_KEY="YOUR_API_KEY"

Keep your API key in your environment or secret manager instead of committing it to source control.

Scrapeless() reads SCRAPELESS_API_KEY. You can also pass an api_key entry in the client configuration dictionary.

Quick Start

Save this as quickstart.py and run python quickstart.py after setting your API key.

from scrapeless import Scrapeless
from scrapeless.types import UniversalScrapingRequest
 
client = Scrapeless()
result = client.universal.scrape(UniversalScrapingRequest(
    actor='unlocker.webunlocker',
    input={'url': 'https://example.com', 'method': 'GET', 'redirect': False}
))
print(result)

Product Coverage Matrix

Product	SDK service	Coverage
Scraping Browser	`client.browser`	Create and manage remote browser sessions.
Browser Profiles	`client.profiles`	Persist browser data across sessions.
Scraping API	`client.scraping`	Extract structured data using website actors.
Web Unlocker	`client.universal`	Retrieve content from protected websites.
Crawl	`client.scraping_crawl`	Scrape a page or crawl a website.
Google Search API	`client.deepserp`	Extract search engine results.
Proxies	`client.proxies`	Generate proxy connection URLs.
AI Scraper	`client.ai_scraper`	Create AI chat tasks and retrieve their status and results.

Usage Examples

Browser

Install the browser integration with pip install pyppeteer.

Advanced browser session management supporting Playwright and Pyppeteer frameworks, with configurable anti-detection capabilities (e.g., fingerprint spoofing, CAPTCHA solving) and extensible automation workflows:

from scrapeless import Scrapeless
from scrapeless.types import ICreateBrowser
import asyncio
import pyppeteer
 
client = Scrapeless()
 
 
async def example():
    # Create a browser session
    config = ICreateBrowser(
        session_name='sdk_test',
        session_ttl=180,
        proxy_country='US',
        session_recording=True
    )
    session = client.browser.create(config).__dict__
    browser_ws_endpoint = session['browser_ws_endpoint']
    print('Browser WebSocket endpoint created:', browser_ws_endpoint)
 
    # Connect to browser using pyppeteer
    browser = await pyppeteer.connect({'browserWSEndpoint': browser_ws_endpoint})
    try:
        page = await browser.newPage()
        await page.goto('https://example.com')
        print(await page.title())
    finally:
        await browser.close()
 
 
asyncio.run(example())

Browser Profile

Manage browser profiles for persistent sessions.

from scrapeless import Scrapeless
 
client = Scrapeless()
profile = client.profiles.create('My Profile')
print(profile)

Scraping API

Direct data extraction APIs for websites (e.g., e-commerce, travel platforms). Retrieve structured product information, pricing, and reviews with pre-built connectors:

from scrapeless import Scrapeless
from scrapeless.types import ScrapingTaskRequest
 
client = Scrapeless()
request = ScrapingTaskRequest(
    actor='scraper.google.search',
    input={'q': 'nike site:www.nike.com'}
)
result = client.scraping.scrape(request=request)
print(result)

Web Unlocker

Extract data from websites using Web Unlocker (exposed as client.universal).

from scrapeless import Scrapeless
from scrapeless.types import UniversalScrapingRequest
 
client = Scrapeless()
result = client.universal.scrape(UniversalScrapingRequest(
    actor='unlocker.webunlocker',
    input={'url': 'https://example.com', 'method': 'GET', 'redirect': False}
))
print(result)

Crawl

Extract data from single pages or traverse entire domains, exporting in formats including Markdown, JSON, HTML, screenshots, and links.

from scrapeless import Scrapeless
 
client = Scrapeless()
 
result = client.scraping_crawl.scrape_url("https://example.com")
print(result)

Proxy

Generate a proxy URL using your gateway and session settings.

from scrapeless import Scrapeless
from scrapeless.types import ICreateProxy
 
client = Scrapeless()
proxy_url = client.proxies.proxy(ICreateProxy(
    country='US',
    session_duration=30,
    session_id=client.proxies.generate_session_id(),
    gateway='your-proxy-gateway:port'
))
print(proxy_url)

AI Scraper

Extract AI chat content in bulk to monitor brand mentions, compare answers, and analyze competitive intelligence from the latest models. Retrieve URLs, prompts, Markdown answers, citations, and more through one integration.

Supported actors include scraper.chatgpt, scraper.perplexity, scraper.copilot, scraper.gemini, scraper.aimode, scraper.overview, scraper.grok, and scraper.alexa. The input JSON depends on the actor; see the AI Scraper documentation for detailed parameters. The optional webhook JSON contains a callback url.

from scrapeless import Scrapeless
from scrapeless.types import AIScraperTaskRequest
 
 
def main():
    client = Scrapeless()  # Uses SCRAPELESS_API_KEY
    task = client.ai_scraper.create_task(AIScraperTaskRequest(
        actor='scraper.chatgpt',
        input={
            'prompt': 'Most reliable proxy service for data extraction',
            'country': 'US',
            'web_search': True,
        },
        # Optional: webhook={'url': 'https://your-webhook.example.com'},
    ))
    print('Created task:', task)
 
    result = client.ai_scraper.get_task_result(task['task_id'])
    print('Task status and result:', result)
    # If status is 'running', call get_task_result again later.
    # If status is 'failed', message contains the failure reason.
 
 
if __name__ == '__main__':
    main()

Both methods return the API JSON unchanged. Creation returns task_id, status, and, when available, task_result. Result retrieval returns status, task_result when available, and message on failure. Status is success, failed, or running; the SDK does not poll automatically.

create_task accepts an AIScraperTaskRequest or a dictionary; dictionaries also allow additional API parameters. Responses are dictionaries.

Google Search API

from scrapeless import Scrapeless
from scrapeless.types import ScrapingTaskRequest
 
client = Scrapeless()
result = client.deepserp.scrape(ScrapingTaskRequest(
    actor='scraper.google.search',
    input={'q': 'nike site:www.nike.com'}
))
print(result)

For more complete integrations, browse the repository’s examples directory.

Error Handling

Catch ScrapelessError for API request failures. Check an AI Scraper response’s status separately to handle task failures.

from scrapeless import Scrapeless, ScrapelessError
from scrapeless.types import UniversalScrapingRequest
 
client = Scrapeless()
try:
    result = client.universal.scrape(UniversalScrapingRequest(
        actor='unlocker.webunlocker',
        input={'url': 'https://example.com', 'method': 'GET'}
    ))
    print(result)
except ScrapelessError as error:
    print(f'Scrapeless API error: {error}')

The current Python exception exposes the error message; it does not define a status_code attribute.

Configuration / Environment Variables

The API key is required. Endpoint overrides are optional; the table shows their defaults.

Pass a dictionary to Scrapeless to override configuration:

import os
from scrapeless import Scrapeless
 
client = Scrapeless({
    'api_key': os.environ['SCRAPELESS_API_KEY'],
    'timeout': 30000,  # Request timeout in milliseconds
    'base_api_url': 'https://api.scrapeless.com',
    'browser_api_url': 'https://browser.scrapeless.com',
    'scraping_crawl_api_url': 'https://api.scrapeless.com',
})

Explicit configuration takes precedence over environment variables. The default request timeout is 30,000 milliseconds.

Environment variable	Purpose / default
`SCRAPELESS_API_KEY`	Required API key from the dashboard.
`SCRAPELESS_BASE_API_URL`	`https://api.scrapeless.com`
`SCRAPELESS_BROWSER_API_URL`	`https://browser.scrapeless.com`
`SCRAPELESS_CRAWL_API_URL`	`https://api.scrapeless.com`

Support

The SDK is released under the MIT License.

Node.js / TypeScript SDK Go SDK