Crawl

Single-page collection, batch jobs, website crawling, and results.

Should I scrape one page, submit a batch, or crawl a website?

Use a single-page request for one known URL, a batch for a known URL list, or a website crawl to discover linked pages. See the Crawl guide.

Why do I get a job identifier instead of page content?

Crawl jobs are asynchronous. Keep the returned identifier and retrieve status and results for that job. Supported SDK helpers can manage this flow for you.

Why does a completed job contain failed pages?

Inspect individual page results as well as the overall job. A job can finish with unsuccessful pages. Use each page’s status and error to decide what needs investigation or a targeted retry.

Why were some linked pages skipped?

Check the crawl limit and URL scope. By default, links outside the starting URL hierarchy are excluded; allowBackwardLinks changes that behavior. A missing page can also be unreachable or fail during collection.

Which output format should I use?

Choose Markdown for text processing, HTML when your parser needs markup, or links for URL discovery. Set the output formats your next processing step needs.

Should I resubmit the job if my client times out?

First check the status of the existing job if you received its identifier. A client timeout alone does not prove that server-side work stopped. Set a bounded polling policy instead of repeatedly creating new jobs.

What should I send when reporting incomplete results?

Include the job identifier, starting URLs, collection options, expected page count, and a few examples of missing or failed pages. Share the smallest case that demonstrates the issue through Contact Support.

Choose another category from the FAQ navigation, or contact support.