sitemapExport is a Go-based CLI tool that crawls a sitemap or RSS feed, extracts content from web pages using CSS selectors, and compiles the data into various formats such as txt, json, jsonl, and md.

The primary use case is to extract content into a file that can be used as contextual data for AI. For example, extracting your docs site as a structured text or JSON file to power a solid AI support chatbot (tutorial here).

• Crawl a sitemap or RSS feed to extract content from pages.
• Extract page content using a specified CSS selector.
• Optionally filter URLs with a simple pattern (e.g., blog/*).
• Generate a structured list of pages with:
  • Page title
  • URL
  • Meta description (if available)
  • Meta tags (if available)
  • Extracted content
• Output formats supported:
  • Plain text (txt)
  • JSON (json)
  • JSON Lines (jsonl)
  • Markdown (md)
• Supports non-interactive usage via flags or environment variables.

Grab the pre-built sitemapExport binary from the releases page, make it executable, and run it.

Requires Go 1.23 or later.

1. Clone the repository:

   git clone https://github.com/rlnorthcutt/sitemapExport.git
   cd sitemapExport

2. Build the binary:

   go build

   For a smaller binary:

   go build -ldflags="-s -w"

The tool supports interactive prompts, command-line flags, and environment variables. Flags take priority over environment variables, which take priority over interactive prompts.

./sitemapExport

Example session:

Enter the Sitemap or RSS feed URL or file path (required): https://example.com/sitemap.xml
Enter the CSS selector to extract content (default: body):
Enter the output filename (default: output):
Enter the URL filter pattern (default: *):
Enter the output file type (txt, json, jsonl, md) (default: txt): jsonl

Export data with the following settings:
  Input:           https://example.com/sitemap.xml
  CSS Selector:    body
  URL Filter:      *
  Output Filename: output
  Output Filetype: jsonl
  Format:          txt

Do you want to proceed with these settings? (y/n): y

./sitemapExport --input="https://example.com/sitemap.xml" --css="article" --filename="output" --type="jsonl" --filter="blog/*"

Short flags are also supported:

./sitemapExport -i "https://example.com/sitemap.xml" -c "article" -n "output" -t "jsonl"

All inputs can also be set via environment variables, which are checked when a flag is not explicitly provided:

• txt — plain text, one page per section
• json — pretty-printed JSON array
• jsonl — one JSON object per line (ideal for AI ingestion pipelines)
• md — Markdown with headings and content

JSON (output.json):

[
  {
    "Title": "Home",
    "URL": "https://example.com",
    "Description": "Welcome to our homepage",
    "Content": "Welcome to our site!"
  },
  {
    "Title": "About Us",
    "URL": "https://example.com/about",
    "Description": "Learn more about our company",
    "Content": "We are a company..."
  }
]

JSON Lines (output.jsonl):

{"Title":"Home","URL":"https://example.com/","Description":"Welcome to our homepage","Content":"Welcome to our site!"}
{"Title":"About Us","URL":"https://example.com/about","Description":"Learn more about our company","Content":"We are a company..."}

sitemapExport/
├── main.go           # CLI entry point, flag definitions, orchestration
├── crawler/          # Sitemap and RSS crawling, page extraction
│   └── crawler.go
├── formatter/        # Formats extracted pages into output formats
│   └── formatter.go
├── writer/           # Writes formatted content to disk
│   └── writer.go
├── feed/             # Feed type detection (sitemap vs RSS)
│   └── feed.go
├── html2text/        # HTML to plain text conversion
│   └── html2text.go
├── go.mod
└── go.sum

• github.com/PuerkitoBio/goquery — HTML parsing and CSS selector queries
• github.com/JohannesKaufmann/html-to-markdown — HTML to Markdown conversion
• github.com/spf13/pflag — CLI flag parsing with short and long flag support
• fortio.org/progressbar — Zero-dependency progress bar
• github.com/rlnorthcutt/cmdkit — Interactive prompts, env var resolution, and colored logging

Feel free to open issues or submit pull requests for new features, bug fixes, or general improvements.

This project is licensed under the MIT License.