LlamaIndex
What is LlamaIndex
LlamaIndex helps applications load, index, and retrieve data for LLM workflows. ScrapelessReader uses the Scrapeless Python SDK to load completed results as LlamaIndex documents.
Choose a service
| Product | Reader service | Returns |
|---|---|---|
| Scraping API | google_search, scraping | Organic search results or structured actor results |
| Crawl | scrape, batch_scrape, crawl | Content from one page, a list of URLs, or a linked site |
| Web Unlocker | web_unlocker | Content returned by a Web Unlocker actor |
| AI Scraper | ai_scraper | Completed answers collected from AI platforms |
Pass the service name to ScrapelessReader.load_data(). For actor-based services, use params[“actor”] to select an engine or site scraper, such as scraper.chatgpt. Put the actor’s request fields in params[“input”]. See the Python SDK reference for input schemas.
Quickstart
1. Install the Reader and set your API key
The package requires Python 3.10 or later. The commands below use macOS and Python 3.13. If you need Python, select a Python 3.13 installer on the Python downloads page for macOS. If you already have another supported version, use its Python command in place of python3.13.
mkdir -p llamaindex-quickstart
cd llamaindex-quickstart
python3.13 -m venv .venv
source .venv/bin/activate
python -m pip install llamaindex-scrapeless==0.1.1Get an API key from the Scrapeless dashboard. In llamaindex-quickstart, create a plain-text file named .env and replace the placeholder below. The filename starts with a dot and has no .txt extension. An existing SCRAPELESS_API_KEY environment variable takes precedence over the value in the file.
SCRAPELESS_API_KEY=YOUR_SCRAPELESS_API_KEY2. Load your first page
Save the code as quickstart.py alongside .env. This example uses the Crawl product’s single-page scrape service.
from dotenv import load_dotenv
from llamaindex_scrapeless import ScrapelessReader
load_dotenv(".env")
reader = ScrapelessReader(timeout=120, request_timeout=30)
documents = reader.load_data(
service="scrape",
params={
"url": "https://example.com",
"options": {"formats": ["markdown"], "onlyMainContent": True},
},
)
print(f"Documents: {len(documents)}")
for document in documents:
print("Source:", document.metadata.get("url"))
print("Page status:", document.metadata.get("statusCode", "not provided"))
print(document.text[:300])Run the script from the same terminal and directory.
python quickstart.py3. Check the result
This excerpt is from a run using the PyPI release.
Documents: 1
Source: https://example.com
Page status: 200
This domain is for use in documentation examples without needing permission.Check that the document contains the expected page text and its source points to example.com. With onlyMainContent, the title Example Domain may appear in document.metadata[“title”]. Check the body for challenge pages, sign-in prompts, and unrelated content. Inspect statusCode when present. The script prints not provided when the field is absent.
The Reader waits for a completed result. A successful response with no documents returns []. Failed or malformed results raise an error.
All services, including AI Scraper, return list[Document]. Each document.text is a string, and document.metadata is a dictionary. Structured records may appear as serialized JSON in document.text.
Other services
To try another service, replace the documents = reader.load_data(…) call in quickstart.py. Keep the imports, Reader setup, and output loop.
Scraping API
documents = reader.load_data(
"google_search",
params={"input": {"q": "retrieval augmented generation", "gl": "us", "hl": "en"}},
)Inspect the snippets, titles, and source URLs. To read the full pages, scrape the result URLs separately. The generic scraping service accepts {“actor”: …, “input”: …} and serializes structured records as JSON in document.text by default. The Google Search actor was used to verify this route. Check the response schema for each actor you use.
Crawl
documents = reader.load_data(
"crawl",
params={
"url": "https://example.com",
"options": {
"limit": 3,
"maxDepth": 1,
"scrapeOptions": {"formats": ["markdown"], "onlyMainContent": True},
},
},
)Inspect the content and source of each returned page. The limit sets the maximum number of pages. For a known URL list, use batch_scrape with params={“urls”: […]} and compare the returned sources with the input list when you need complete coverage.
Web Unlocker
documents = reader.load_data(
"web_unlocker",
params={
"actor": "unlocker.webunlocker",
"input": {"url": "https://example.com", "method": "GET", "redirect": False},
},
)Check the extracted text and source URL. The verified response format is HTML wrapped in JSON.
AI Scraper
documents = reader.load_data(
"ai_scraper",
params={
"actor": "scraper.chatgpt",
"input": {
"prompt": "Explain retrieval augmented generation and cite sources.",
"country": "US",
"web_search": True,
"shopping": False,
},
},
)Check the answer text, request context, and any citations. A result can have an empty citation list. The Reader waits for the remote task to succeed and exposes the result through the same document interface.
Resume a task
timeout is the Reader’s polling budget, request_timeout is the timeout for a single SDK request, and poll_interval is the time between status checks. All three are in seconds. The remote task may keep running after a polling timeout. ScrapelessTaskTimeout and a status request’s ScrapelessTaskError carry the service and task_id you can use to resume the task.
To poll the same task once more after a timeout, replace the load_data() call in quickstart.py with the following block. Keep the existing Reader setup and output loop.
from llamaindex_scrapeless import ScrapelessTaskTimeout
params = {
"url": "https://example.com",
"options": {"formats": ["markdown"], "onlyMainContent": True},
}
try:
documents = reader.load_data("scrape", params=params)
except ScrapelessTaskTimeout as error:
documents = reader.load_data(error.service, task_id=error.task_id, params=params)Pass the original parameters along with the saved task ID. Web Unlocker does not support this resume flow. Use load_data to wait synchronously or aload_data to await the result in async Python code. Remote processing may continue if the await is cancelled.
Troubleshooting
| Symptom | What to check |
|---|---|
| Missing API key or authentication failure | Check the .env filename and location, the SCRAPELESS_API_KEY available to Python, and the account’s permissions and credits. |
| Module import fails | Activate the virtual environment used to install the PyPI package, then rerun the script. |
| The call returns only a task ID or times out | Keep the task ID and use the service’s resume flow to wait for the result. |
| Empty list or missing sources | Check the target content. When you require complete coverage, compare the returned sources with the input URLs. |