ILCrawler

ILCrawler is the technical SEO crawler I use for my own websites and client audits. It brings page checks, links, screenshots, performance samples and reporting into one place. It’s also the project that repeatedly convinced me I was only one rewrite away from being happy with it.

Images

About ILCrawler

It started in Python. It worked, but it was painfully slow. So I began moving the crawling work to Go, which helped enough to make the next bottleneck very obvious: moving large amounts of crawl data around.

That turned into a slightly ridiculous amount of benchmarking. Across the different parts of the project, I tried Go, Rust, Python, Node.js, C, C++, C#, Zig and several others. Even PHP got a turn. Apparently, "make the crawler faster" had become a separate project.

I measured throughput, CPU usage and memory consumption on the workloads I actually needed to run. For the heavy import work, Rust was the clear winner in my tests. That became its job: getting large volumes of crawl data into PostgreSQL without the import becoming the thing I spent all my time waiting for.

Then there were screenshots.

I went through Python with Playwright, Rust browser libraries and more combinations than I particularly want to remember. Go with Rod finally gave me the results I wanted. Fonts, images, lazy-loaded content and full-page captures all needed attention; successfully opening a page was only the beginning.

That choice escaped this project. Go + Rod is now my default whenever another project needs website screenshots. After that much experimenting, I’m quite happy to stop having the same argument with myself.

The admin went through its own rewrite. I moved it to Elixir, Phoenix and LiveView because I wanted crawl progress and job updates to arrive naturally over a live connection. Elixir now also handles the API, scheduling and workflow state, while the workers do the crawling, importing, rendering and other heavy work.

Underneath all those rewrites, the purpose has stayed fairly consistent. I want to crawl a site, inspect the pages and links, find technical problems and keep enough evidence to understand what happened. That includes redirects, canonicals, indexing rules, duplicate content, broken links and pages that are difficult to reach. Selected pages can get browser-rendered checks, screenshots and Lighthouse performance audits, with findings and reports available from the same dashboard.

It runs as a private Docker-based application for my own work and client audits. I can follow a run, investigate affected pages and export the findings without assembling everything from separate tools.

The mixed stack grew out of that process. Go does much of the crawling and worker work, Rust handles bulk imports, Elixir keeps the application and live dashboard coordinated, and Node.js runs Lighthouse.

I do like a rewrite. This project has given me plenty of opportunities. Usually, I finish making one part faster and discover that the next slow part has been patiently waiting for its turn.

Release history

Version 0.41.06 Aug 2026

ILCrawler

Images

About ILCrawler

Release history

More software