scrape-do-python

A Python SDK for the Scrape.do web-scraping proxy API.

Built on httpx and pydantic v2, with strict request validation, automatic retries on gateway errors, sticky-session validation, and SDK-native lifecycle hooks.

---

Installation

pip install scrape-do-python

---

Quickstart

Sync API Example

from scrape_do import ScrapeDoClient

with ScrapeDoClient() as client: # (1)!
    response = client.get( # (2)!
        "https://example.com",
        super=True,
        render=True,
        return_json=True,
        show_frames=True,
    )

    print(response.is_proxy_error)
    print(response.frames[0].url)
    print(response.remaining_credits)

1. The Scrape.do API Token is pulled from the SCRAPE_DO_API_KEY environment variable. It can also be provided explicitly via the client's api_token argument

2. client.get() routes the keyword arguments through Pydantic-validated RequestParameters, builds the proxied URL, executes through httpx, and wraps the result in a strongly-typed ScrapeDoResponse

---

Features

Typing And ValidationClient Features

Type-Checked Request Parameters

Request parameters are fully type-checked and automatically validated via the RequestParameters Pydantic model

Strongly-Typed Responses

ScrapeDoResponse exposes the parsed JSON envelope, browser action results, screenshots, and network/websocket logs as Pydantic models for easy access

Browser Automation

Pydantic models for Browser Actions provide validation and type-hinting for the playWithBrowser API parameter

Smart Routing

ScrapeDoClient.request() accepts either **api_kwargs, a pre-built RequestParameters object, or a raw api.scrape.do URL

Automatic Retries

The client retries Scrape.do gateway errors (429 / 502 / 510) with a configurable backoff strategy (static float or Callable). Default is jittered exponential

Sticky-Session Validation

Supply a session_validator callback to detect proxy node rotations and raise RotatedSessionError when your target's session state is lost.

SDK-Native Event Hooks

request / response / retry lifecycle hooks fire at the logical execution boundary, distinct from httpx's transport-level hooks to prevent telemetry corruption

---

Additional Information

Status

• Roadmap → What's Coming Next

• Changelog → What's Already Shipped

How To Help

• Contributing → Local Setup, PR Guidelines, Tests

Examples

Check out the Examples Page for different usage examples