Web crawling projects and notes

Stuff I build with

53 projects

Web crawling · 3

crawlcontract

Rust Version 0.4.0

A command-line tool for catching crawl and indexing problems when a website changes. It checks built files or pages fetched from a live or preview site, and compares scans so you can spot problems before a release.

View tool

routelint

routelint

TypeScript Version 0.4.0

A command-line tool for finding broken links, incorrect redirects and indexing conflicts on a live or preview website. It discovers URLs, checks what your server returns and gives you findings tied to specific pages, with the evidence behind them.

View tool

ILCrawler

Go Version 0.41.0

ILCrawler is the technical SEO crawler I use for my own websites and client audits. It brings page checks, links, screenshots, performance samples and reporting into one place. It’s also the project that repeatedly convinced me I was only one rewrite away from being happy with it.

View project

Notes

All notes

No published notes for this tag yet.