Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit 9cf3f3f

Browse filesBrowse files
nihalwasherenihalwashere
committed
Document extract feature
- New concept page: extract primitive overview, schema formats, credits - New guide: structured extraction with code examples (JS, Python, cURL, CLI) - Updated API reference with extract section - Added pages to navigation
1 parent f594b8d commit 9cf3f3f
Copy full SHA for 9cf3f3f

4 files changed

+385-1Lines changed: 385 additions & 1 deletion

File tree

Expand file treeCollapse file tree
Open diff view settings
Filter options
Expand file treeCollapse file tree
Open diff view settings
Collapse file

‎api-reference/read.mdx‎

Copy file name to clipboardExpand all lines: api-reference/read.mdx
+18Lines changed: 18 additions & 0 deletions
  • Display the source diff
  • Display the rich diff
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,24 @@ Set `maxConcurrency` to limit how many browser slots a batch or crawl job uses.
2121

2222
Check your current usage with [Queue status](/api-reference/account/queue-status).
2323

24+
## Extract
25+
26+
Add the `extract` parameter to pull structured data from the page alongside markdown. Provide a JSON Schema, a shorthand schema, a natural language prompt, or both.
27+
28+
```bash
29+
curl -X POST https://api.reader.dev/v1/read \
30+
-H "x-api-key: $READER_KEY" \
31+
-H "Content-Type: application/json" \
32+
-d '{
33+
"url": "https://example.com/product",
34+
"extract": {
35+
"schema": { "title": "string", "price": "number", "in_stock": "boolean" }
36+
}
37+
}'
38+
```
39+
40+
The response includes an `extracted` field with the structured data and extraction metadata. Adds 2 credits to the scrape cost. Only supported for single-URL scrapes (not batch or crawl). See [Extract](/home/concepts/extract) for details.
41+
2442
## Idempotency
2543

2644
Pass an `x-idempotency-key` header to deduplicate retried POSTs. Reader caches the original response for 24 hours and returns it verbatim on any subsequent request with the same key.
Collapse file

‎home/concepts/extract.mdx‎

Copy file name to clipboard
+151Lines changed: 151 additions & 0 deletions
  • Display the source diff
  • Display the rich diff
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,151 @@
1+
---
2+
title: "Extract"
3+
description: "Pull structured data from any web page using an LLM"
4+
---
5+
6+
Extract lets you pull structured data from a web page in a single API call. You provide a URL and a schema (or a natural language prompt), and Reader scrapes the page, converts it to clean markdown, sends it to an LLM, and returns validated structured JSON alongside the usual markdown output.
7+
8+
## How it works
9+
10+
Extract is an optional parameter on `POST /v1/read`. The scrape pipeline runs as usual, then the extracted markdown is sent to an LLM with your schema for structured extraction.
11+
12+
```
13+
URL --> Scrape --> Markdown --> LLM + Schema --> Structured JSON
14+
```
15+
16+
The `extracted` field appears in the response alongside `markdown`, `html`, and other standard fields. If extraction fails, the scrape result is still returned with an error in `metadata.extraction.extractionError`.
17+
18+
## Schema formats
19+
20+
### JSON Schema (full)
21+
22+
Standard JSON Schema with types, descriptions, and nested objects:
23+
24+
```json
25+
{
26+
"extract": {
27+
"schema": {
28+
"type": "object",
29+
"properties": {
30+
"title": { "type": "string", "description": "The product name" },
31+
"price": { "type": "number", "description": "Price without currency symbol" },
32+
"in_stock": { "type": "boolean" }
33+
}
34+
}
35+
}
36+
}
37+
```
38+
39+
### Shorthand
40+
41+
A simpler format where keys map directly to type names:
42+
43+
```json
44+
{
45+
"extract": {
46+
"schema": {
47+
"title": "string",
48+
"price": "number",
49+
"in_stock": "boolean"
50+
}
51+
}
52+
}
53+
```
54+
55+
Shorthand is automatically expanded to full JSON Schema internally. Supported types: `string`, `number`, `integer`, `boolean`, `array`, `object`.
56+
57+
### Prompt-only
58+
59+
No schema, just a natural language instruction. Returns freeform JSON:
60+
61+
```json
62+
{
63+
"extract": {
64+
"prompt": "List the 3 main topics discussed on this page"
65+
}
66+
}
67+
```
68+
69+
### Prompt + Schema
70+
71+
Combine both for maximum control. The prompt guides the LLM while the schema enforces structure:
72+
73+
```json
74+
{
75+
"extract": {
76+
"schema": { "title": "string", "price": "number" },
77+
"prompt": "Focus on the primary product, ignore accessories"
78+
}
79+
}
80+
```
81+
82+
## Response shape
83+
84+
```json
85+
{
86+
"success": true,
87+
"data": {
88+
"url": "https://example.com/product",
89+
"markdown": "# Widget Pro\n\nPrice: $49.99...",
90+
"extracted": {
91+
"title": "Widget Pro",
92+
"price": 49.99,
93+
"in_stock": true
94+
},
95+
"metadata": {
96+
"statusCode": 200,
97+
"duration": 2345,
98+
"scrapedAt": "2026-07-05T12:00:00.000Z"
99+
}
100+
}
101+
}
102+
```
103+
104+
## Null handling
105+
106+
If a field in your schema does not exist on the page, Reader returns `null` for that field. It will never hallucinate values that are not present in the content.
107+
108+
```json
109+
{
110+
"extracted": {
111+
"title": "Widget Pro",
112+
"price": 49.99,
113+
"ceo_name": null,
114+
"ipo_date": null
115+
}
116+
}
117+
```
118+
119+
## Credits
120+
121+
Extract adds 2 credits on top of the scrape cost:
122+
123+
| Operation | Credits |
124+
|-----------|---------|
125+
| Standard scrape | 1 |
126+
| Standard scrape + extract | 3 |
127+
| Premium scrape + extract | 5 |
128+
129+
If the scrape succeeds but extraction fails, the full cost (scrape + extract) is still charged because the LLM call was attempted.
130+
131+
## Limitations
132+
133+
- **Scrape mode only.** Extract is not supported for batch or crawl operations.
134+
- **Token limit.** Page content is capped at 30,000 tokens before being sent to the LLM. Long pages are intelligently truncated (70% from the beginning, 30% from the end).
135+
- **No caching.** Extraction always runs fresh, even if the scrape result is cached. Cached scrape responses do not include `extracted`.
136+
- **Validation retry.** If the LLM output does not match your schema, Reader retries once with additional context about the validation error.
137+
138+
## Error handling
139+
140+
Extract never fails the entire request. If the LLM call fails (timeout, rate limit, invalid response), the scrape result is still returned with `extracted: null` and an error in `metadata.extraction.extractionError`:
141+
142+
```json
143+
{
144+
"extracted": null,
145+
"metadata": {
146+
"extraction": {
147+
"extractionError": "Extraction service rate limited, try again later"
148+
}
149+
}
150+
}
151+
```
Collapse file

‎home/guides/llm/structured-extraction.mdx‎

Copy file name to clipboard
+213Lines changed: 213 additions & 0 deletions
  • Display the source diff
  • Display the rich diff
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,213 @@
1+
---
2+
title: "Structured Extraction"
3+
description: "Extract structured data from any web page with schemas or prompts"
4+
---
5+
6+
This guide shows how to use the `extract` parameter on `POST /v1/read` to pull structured data from web pages.
7+
8+
## Product data
9+
10+
Extract product information from an e-commerce page:
11+
12+
<CodeGroup>
13+
14+
```bash cURL
15+
curl -X POST https://api.reader.dev/v1/read \
16+
-H "x-api-key: rdr_YOUR_KEY" \
17+
-H "Content-Type: application/json" \
18+
-d '{
19+
"url": "https://example.com/product/widget-pro",
20+
"extract": {
21+
"schema": {
22+
"name": "string",
23+
"price": "number",
24+
"currency": "string",
25+
"in_stock": "boolean",
26+
"description": "string"
27+
}
28+
}
29+
}'
30+
```
31+
32+
```javascript JavaScript
33+
import { ReaderClient } from "@vakra-dev/reader-js";
34+
35+
const client = new ReaderClient({ apiKey: "rdr_YOUR_KEY" });
36+
37+
const result = await client.read({
38+
url: "https://example.com/product/widget-pro",
39+
extract: {
40+
schema: {
41+
name: "string",
42+
price: "number",
43+
currency: "string",
44+
in_stock: "boolean",
45+
},
46+
},
47+
});
48+
49+
if (result.kind === "scrape") {
50+
console.log(result.data.extracted);
51+
// { name: "Widget Pro", price: 49.99, currency: "USD", in_stock: true }
52+
}
53+
```
54+
55+
```python Python
56+
from reader_py import ReaderClient
57+
58+
client = ReaderClient(api_key="rdr_YOUR_KEY")
59+
60+
result = client.read(
61+
url="https://example.com/product/widget-pro",
62+
extract={"schema": {"name": "string", "price": "number"}},
63+
)
64+
65+
if result.kind == "scrape":
66+
print(result.data.extracted)
67+
```
68+
69+
</CodeGroup>
70+
71+
## Table data
72+
73+
Extract structured data from HTML tables (e.g., a pricing page):
74+
75+
```json
76+
{
77+
"url": "https://example.com/pricing",
78+
"extract": {
79+
"schema": {
80+
"type": "object",
81+
"properties": {
82+
"plans": {
83+
"type": "array",
84+
"items": {
85+
"type": "object",
86+
"properties": {
87+
"name": { "type": "string" },
88+
"price_monthly": { "type": "number" },
89+
"features": { "type": "array", "items": { "type": "string" } }
90+
}
91+
}
92+
}
93+
}
94+
}
95+
}
96+
}
97+
```
98+
99+
## Using field descriptions
100+
101+
Add `description` to schema fields to guide the LLM on what to extract:
102+
103+
```json
104+
{
105+
"url": "https://example.com/article",
106+
"extract": {
107+
"schema": {
108+
"type": "object",
109+
"properties": {
110+
"title": {
111+
"type": "string",
112+
"description": "The main article headline"
113+
},
114+
"author": {
115+
"type": "string",
116+
"description": "The author's full name"
117+
},
118+
"published_date": {
119+
"type": "string",
120+
"description": "Publication date in YYYY-MM-DD format"
121+
},
122+
"summary": {
123+
"type": "string",
124+
"description": "A one-sentence summary of the article"
125+
}
126+
}
127+
}
128+
}
129+
}
130+
```
131+
132+
## Prompt-only mode
133+
134+
When you do not need a fixed schema, use a natural language prompt:
135+
136+
```json
137+
{
138+
"url": "https://example.com/about",
139+
"extract": {
140+
"prompt": "Extract the company name, founding year, and number of employees"
141+
}
142+
}
143+
```
144+
145+
The response will be freeform JSON based on what the LLM finds on the page.
146+
147+
## Combining prompt and schema
148+
149+
Use both together -- the prompt provides context while the schema enforces structure:
150+
151+
```json
152+
{
153+
"url": "https://example.com/product",
154+
"extract": {
155+
"schema": { "name": "string", "price": "number" },
156+
"prompt": "Focus on the main product listing, ignore recommended products in the sidebar"
157+
}
158+
}
159+
```
160+
161+
## Handling missing fields
162+
163+
If a field in your schema does not exist on the page, it comes back as `null`. This is intentional -- Reader will never hallucinate values:
164+
165+
```json
166+
{
167+
"extracted": {
168+
"title": "Widget Pro",
169+
"price": 49.99,
170+
"warranty_years": null
171+
}
172+
}
173+
```
174+
175+
## CLI usage
176+
177+
```bash
178+
# With schema
179+
reader scrape https://example.com/product \
180+
--extract-schema '{"title": "string", "price": "number"}'
181+
182+
# With prompt
183+
reader scrape https://example.com/about \
184+
--extract-prompt "Extract the company name and founding year"
185+
186+
# Both
187+
reader scrape https://example.com/product \
188+
--extract-schema '{"name": "string", "price": "number"}' \
189+
--extract-prompt "Focus on the primary product"
190+
```
191+
192+
## Error handling
193+
194+
Extract never breaks the scrape response. If the LLM fails, you still get the markdown:
195+
196+
```javascript
197+
const result = await client.read({
198+
url: "https://example.com",
199+
extract: { schema: { title: "string" } },
200+
});
201+
202+
if (result.kind === "scrape") {
203+
// Markdown is always available
204+
console.log(result.data.markdown);
205+
206+
// Check if extraction succeeded
207+
if (result.data.extracted) {
208+
console.log("Extracted:", result.data.extracted);
209+
} else {
210+
console.log("Extraction failed:", result.data.metadata.extraction?.extractionError);
211+
}
212+
}
213+
```

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.