WebDiver is a versatile Python script for crawling websites, extracting internal and external links, titles, and descriptions. It's useful for tasks such as web analysis, OSINT (Open Source Intelligence) gathering, and competitive analysis.

A tool for search engine based competitive analysis. AWS Amplify project with many web crawlers for different search engines, and a simple web frontend for displaying historical results in Next.js

.net-based Crawlers for news of Taiwan (.net 台灣新聞爬蟲，數據物件化，方便使用)

A simple WebCrawler for exploring and downloading content from web pages within a given domain/url.

Generate a list of file links you can feed to wget for easy downloading! Mainly used for spidering web folders with lots of files. Can even generate a sitemap.txt or XML file for your website!

Web Crawler written in C# that parses all urls from a specific page then recursively visits them while parsing all links available on that webpage

Simple WebCrawler

SpiderBolt is a fast and efficient Python web scraping script that extracts links from websites using multi-threading and random user agents. It categorizes links into HTML and other types, groups them by paths, and saves them in an organized file. Customizable settings ensure flexibility for various scraping needs.

Various webcrawler programs made to gain a deeper understandings of webcrawling concepts.

A high-performance, concurrent web crawler in Go that extracts URLs, downloads content, and converts tens of thousands of web pages to Markdown in minutes

🚫🤖 Override /robots.txt to disallow all web crawlers, regardless settings stored in the database. Compatible with Liferay 7.0, 7.1, 7.2, 7.3 and 7.4.

🕷️ Crawl websites efficiently with this Bash script, producing a clean list of URLs while respecting `robots.txt` and staying within specified domains.