Aggregate, de-duplicate and republish RSS feeds.

The problem I'm solving here is this; if I subscribe to the main Guardian RSS feed, I see a great many articles I'm not interested in¹. But if instead I subscribe to the feeds for individual tags, while I don't see the things I'm not interested in, I do see a great many duplicates - articles with multiple tags show up in multiple feeds. This little app allows me to have the best of both worlds - I can see only² the articles I'm interested in my reader³, and only once.

Local development requires:

• uv for all things Python
• xc as a task runner
• colima⁴ for running the docker images we need for our integration tests
• Node.js for pyright (version pinned in .tool-versions)

Some tasks may require additional tools:

• gh for controlling GitHub actions
• 1Password CLI for secure secret usage
• AWS CLI for controlling & querying AWS
• terraform for deployment (version pinned in .tool-versions)
• libxml2 for testing our RSS output
• fzf and gum for command line shenanigans.

On a Mac, you can install most of these with homebrew and asdf.

brew install uv xc gh 1password-cli awscli asdf colima libxml2 fzf gum # And follow any additional setup instructions brew gives you
asdf plugin add terraform
asdf plugin add nodejs
asdf install

The application is a Flask async web app deployed as an AWS Lambda function running within the Lambda Web Adapter.

Good places to start investigating the code are:

• web.py, the application entry point.
• routes.py, the Flask route definitions.

On each request, RSSService orchestrates the full pipeline:

• A FeedsService implementation reads a list of feed paths and constructs full Guardian RSS URLs.
  • FileFeedsService reads from from feeds.txt, OR
  • S3FeedsService reads from an S3 object using boto3.
• Fetcher retrieves all feeds concurrently using httpx with HTTP/2 and connection pooling.
• RSSParser parses the responses with defusedxml and de-duplicates items by GUID.
• RSSGenerator sorts by date, applies a configurable item limit, and emits a fresh RSS feed.

Services are wired together with wireup for dependency injection, with configuration sourced from environment variables as per the twelve-factor app. AWS API Gateway provides the public HTTP endpoint, backed by Terraform-managed infrastructure-as-code located in terraform/.

Configuration is sourced from environment variables:

The following are additionally required when FEEDS_SERVICE=S3FeedsService:

These tasks can be run using xc.

Precommit tasks

Requires: test, lint, audit

RunDeps: async

#!/usr/bin/env python
import this

Run CLI - outputs RSS to stdout

uv run cli -vv

Run web server

./run.sh

Run all tests

Requires: unit, integration

RunDeps: async

Unit tests

uv run pytest tests/unit/ --durations=10 --cov-report term-missing --cov-fail-under 100 --cov src

Integration tests

if command -v colima > /dev/null; then colima status || colima start; fi
uv run pytest tests/integration/ -s --durations=10

Format code

uv run ruff format .
uv run ruff check . --fix-only

Code quality & security checks

Requires: lint-code, type-checking

RunDeps: async

Lint code

uv run ruff format . --check
uv run ruff check .

Type checking

uv run pyright

Audit for known vulnerabilities

Requires: audit-py, audit-gha

RunDeps: async

Audit Python dependencies for known vulnerabilities

uv audit

Scan GitHub Actions for vulnerabilities

uvx zizmor -o .

Build lambda image

Inputs: IMAGE_NAME

Environment: IMAGE_NAME=deployment_package.zip

rm -rf build/ terraform/"$IMAGE_NAME"
uv export --no-dev --python 3.14 --format requirements-txt --output-file requirements.txt
uv pip install -r requirements.txt --target build --python 3.14
cp -r src/rss_agg build/
cp run.sh build/
cp feeds.txt build/
chmod +x build/run.sh
cd build
zip -r ../terraform/"$IMAGE_NAME" .
cd ..

Initialise terraform

Directory: ./terraform

terraform init

Plan infrastructure changes

Requires: build, terraform-init

RunDeps: async

Directory: ./terraform

terraform plan

Push to origin, and monitor CI workflow

Requires: pc

Inputs: WORKFLOW

Environment: WORKFLOW=ci.yml

git push
sleep 5
RUN_ID=$(gh run list --workflow="$WORKFLOW" --limit=1 --json databaseId --jq '.[0].databaseId')
gh run watch "$RUN_ID" --exit-status

Run deployment workflow

Inputs: WORKFLOW

Environment: WORKFLOW=cd.yml

gh workflow run "$WORKFLOW"
sleep 5
RUN_ID=$(gh run list --workflow="$WORKFLOW" --limit=1 --json databaseId --jq '.[0].databaseId')
gh run watch "$RUN_ID" --exit-status

Check feed is running and returning XML

Inputs: API_URL

Environment: API_URL=http://0.0.0.0:8080

set +x
echo "Testing API at: $API_URL"

# curl will retry up to 5 times with 5 second delays, fail on non-200 status
if curl -fsSL --retry 5 --retry-delay 5 "$API_URL" | xmllint --noout - 2>/dev/null; then
    echo "✓ API returned valid XML"
else
    echo "✗ API check failed (ensure xmllint is installed: brew install libxml2)"
    exit 1
fi

Query CloudWatch logs for recent Lambda activity

Inputs: DURATION

Environment: DURATION=1h

aws logs tail /aws/lambda/rss_aggregator --since "$DURATION" --format short

One-off commands to set up the AWS S3 bucket that terraform will use to store infrastructure state. Run aws configure first to authenticate if necessary.

aws s3 mb s3://brunns-rss-agg-terraform-state --region eu-west-2
aws s3api put-bucket-versioning --bucket brunns-rss-agg-terraform-state --versioning-configuration Status=Enabled

Upload feeds.txt (by default) to the S3 feeds bucket

Inputs: FEEDS_FILE, BUCKET, OBJECT

Environment: FEEDS_FILE=feeds.txt

Environment: BUCKET=brunns-rss-agg-feeds

Environment: OBJECT=feeds.txt

aws s3 cp "$FEEDS_FILE" s3://"$BUCKET"/"$OBJECT"

Use brunns-python-template or similar.