Web crawling projects and notes

Stuff I build with

55 projects

Web crawling · 3

Show all

Projects

routelint

routelint

TypeScript Version 0.5.1

A command-line tool for finding broken links, incorrect redirects and indexing conflicts on a live or preview website. It discovers URLs, checks what your server returns and gives you findings tied to specific pages, with the evidence behind them.

View tool
crawlcontract - scan result
crawlcontract - scan result

crawlcontract

Rust Version 0.4.0

A command-line tool for catching crawl and indexing problems when a website changes. It checks built files or pages fetched from a live or preview site, and compares scans so you can spot problems before a release.

View tool
ILCrawler crawl issues screenshot
ILCrawler crawl issues screenshot

ILCrawler

Go Version 0.41.0

ILCrawler is the technical SEO crawler I use for my own websites and client audits. It brings page checks, links, screenshots, performance samples and reporting into one place. It’s also the project that repeatedly convinced me I was only one rewrite away from being happy with it.

View project

No published notes for this tag yet.