crawlcontract
A command-line tool for catching crawl and indexing problems when a website changes. It checks built files or pages fetched from a live or preview site, and compares scans so you can spot problems before a release.
DownloadVersion 0.3.1crawlcontract
About crawlcontract
A site can look perfectly fine while telling search engines several different things. The sitemap includes a page, but that page says not to index it. A canonical points somewhere unexpected. Internal links still take the scenic route through old redirects. The build passed, though, so everything must be fine. Probably.
CrawlContract checks how those pieces fit together: sitemaps, robots rules, canonical URLs, redirects, internal links and language references. When it finds a contradiction, it shows the affected URLs and the evidence behind the finding, so you have somewhere useful to start fixing it.
You can also save a scan and compare it with the next one. That helps catch changes that are easy to miss during a redesign or template update: fewer pages marked as indexable, missing headings, less content or disappearing internal links. Some of those changes might be intentional. The point is to make them visible before they become a surprise.
It’s written in Rust and can run locally or as part of your automated build checks. You choose which findings should fail a check, set thresholds for changes, and add exceptions where they make sense.
It’s still early in development. Right now, it works with built files and the HTML a server returns, without running JavaScript. Particularly useful when you’re changing how a site is generated and want to check that the less visible parts still make sense.
Release history
Version 0.3.1
- Fail live scans instead of reporting success when the entry response is not successful HTML, a request fails, robots.txt blocks the entry point, or the page budget is exhausted.
- Map preview-origin sitemap entries, HTML and HTTP canonicals, hreflang, and internal links to one consistent public identity.
- Build static routes and recognize sitemap files from native path components on Windows as well as Unix.
- Reject unknown policy fields, empty failure gates, invalid exclusions, and unsupported rule IDs instead of silently accepting ineffective policy.
- Reject crawl delays above 60 seconds without panicking, and wait only between page requests rather than before the first page.
External download
Download 0.3.1
Version 0.3.0
Added
- Explicit failures for unresolved sitemap, canonical, and internal-link targets and redirect cycles.
- Standards-aware XML sitemap parsing, multiple static sitemap files, preview origin mapping, and bounded concurrent live fetching.
- Policy validation and a dedicated word-loss threshold.
- Tag-triggered Linux, macOS, and Windows release archives with SHA-256 checksums.
Fixed
- Preserve all URL discovery sources and stop treating sitemap files as page URLs.
- Compute reachability from the real entry point so orphan pages are not hidden.
- Combine robots directives restrictively and support wildcard/end-anchored
robots.txtrules with allow-wins ties. - Record real scan timestamps and report the package version consistently.
- Reject invalid output selections, snapshot schemas, stale snapshot statistics, and incompatible snapshot base URLs.
Changed
- Parse output formats and failure severities as validated CLI values.
- Retire duplicate
CC-CANONICAL-CONSISTENCY-002; sitemap/canonical mismatch is reported once asCC-SITEMAP-INDEXABILITY-003.
External download
Download 0.3.0
Version 0.2.2
- Applied
cargo fmtformatting to source files that were inconsistent with the project's default rustfmt style.
External download
Download 0.2.2
Version 0.2.1
- GitHub Actions CI workflow for check, test, clippy, and format.
- CLI smoke tests covering
scan,diff, output formats, snapshot file writing, policy application, and error handling.
External download
Download 0.2.1
Version 0.2.0
- Live site scanning via HTTP with
crawlcontract scan https://example.com. Supports bounded concurrency, configurable--max-pages, redirect chain tracking,robots.txtcrawl-delay respect, and sitemap discovery fromrobots.txtsitemap directives. --max-pagesand--concurrencyCLI flags for thescansubcommand.- Async runtime (tokio) for both static and live scanning paths.
External download
Download 0.2.0
Version 0.1.0
- Initial release of crawlcontract.
- Static directory scanner that walks
./distand extracts HTML signals. - Signal extractors for
<link rel="canonical">,<meta name="robots">,<link rel="hreflang">, internal links, and redirect targets. robots.txtparser with allow/disallow evaluation.- XML sitemap and sitemap-index parser.
- URL-state graph builder that composes per-URL effective directives.
- Reachability analysis from entry points via internal link graph.
- Cross-artifact invariant rules engine with 18 deterministic rule IDs covering sitemap indexability, canonical consistency and resolution, effective robots directives, hreflang reciprocal alignment, internal link targets, and orphan detection.
- Snapshot diff engine that compares baseline and candidate scans, detecting unexpected losses in indexable URLs, links, and content.
- Output formatters for terminal (colored), JSON, Markdown, and SARIF 2.1.0.
- TOML policy file for exclusions and diff thresholds.
- CLI with
scananddiffsubcommands and CI-friendly exit codes. - Neutral
example.testfixtures covering all rule scenarios.
External download
Download 0.1.0
More software
IndexLane Safe WebP Queue
WordPress Version 1.0.1
A WordPress plugin that creates WebP copies of your images while keeping the originals. Convert on the server or use the optional browser mode when your host cannot write WebP, with progress reporting and checks before converted files are saved.
Classifieds & Advertising Platform — Go Rebuild
Go Version 1.17.10
An ongoing Go rebuild of my Laravel classifieds platform, with a React dashboard, incremental listing updates and background jobs handled through NATS JetStream. It preserves the existing platform’s behaviour while changing how the application processes and serves its data.
pgvector-filterbench
pgvector-filterbench
Go Version 0.1.2
A command-line tool for checking how pgvector search behaves once you add filters. It measures speed, compares approximate results with exact search, and checks query plans to see whether PostgreSQL is using your HNSW index.
wp-migrate-core
wp-migrate-core
Typescript Version 0.3.0
A command-line tool for getting a WordPress-to-Astro migration started. It reads a WordPress XML export, inventories content, images and links, and can generate an Astro project with a clear list of what still needs rebuilding.
hueframe
TypeScript Version 1.0.3
A tool for turning a photo into a website color theme. Drop in an image, see its colors applied to a page, adjust the result and export it as CSS variables or JSON design tokens.
SSRWire
Typescript Version 0.5.0
A command-line tool for inspecting the HTML your server sends, including content and metadata that arrive in a stream. It records when important elements appear and compares responses for different browser, search-crawler and social-preview user agents.
routeplay
routeplay
TypeScript Version 0.4.0
A command-line tool for checking that a page delivers the right content and metadata however you reach it. It compares the HTML your server sends, a fresh browser load and the result of clicking through from another page.
ReachLog
Typescript Version 0.2.1
ReachLog is a small outreach and contact manager for keeping track of who you’ve contacted, what you’ve discussed and when to follow up. It brings contacts, companies and email conversations together in a local application that works with your existing email account.
IndexLane Redirect & Internal Link Auditor
Wordpress Version 0.7.0
A WordPress plugin for finding broken internal links, redirects and old URLs left behind after a migration. It shows you where those links are used and takes you to the place where you can edit them.
routelint
routelint
TypeScript Version 0.3.1
A command-line tool for finding broken links, incorrect redirects and indexing conflicts on a live or preview website. It discovers URLs, checks what your server returns and gives you findings tied to specific pages, with the evidence behind them.
Data-Driven Publishing Platform
Data-Driven Publishing Platform
Laravel Version 0.14.1
A publishing platform for running several websites from structured source data. It handles imports, page generation, scheduled updates and search monitoring through one administration panel. Four domains are live; getting traffic to them remains the unfinished part.
Classifieds & Advertising Platform
Laravel Version 1.3.6
A classifieds and advertising platform I’ve been building and maintaining for roughly two years. It combines location-based browsing, advertiser dashboards, paid listings, invoicing, live chat and an administration panel for running the site.
Private Organic Search & LLM Activity Tracker
Elixir Version 0.39.19
A private analytics tool for tracking website traffic, organic search performance, and activity from search engines and LLM crawlers. It brings ordinary visitor statistics, crawler requests, Google Search Console and Bing Webmaster Tools into one dashboard.
ILCrawler
Go Version 0.41.0
ILCrawler is the technical SEO crawler I use for my own websites and client audits. It brings page checks, links, screenshots, performance samples and reporting into one place. It’s also the project that repeatedly convinced me I was only one rewrite away from being happy with it.
crawlreplay
crawlreplay
Rust Version 0.1.0
A command-line tool for testing robots.txt changes and new deployments against URLs found in your server logs. It helps you catch newly blocked pages, broken responses and unexpected indexing changes before putting those changes live.
Data-Driven Publishing Platform — Go Rebuild
Data-Driven Publishing Platform — Go Rebuild
Go Version 0.226.2
An unlaunched Go rebuild of my publishing platform, using PostgreSQL for application data and background jobs, React for administration and Astro for public websites. It includes source processing, publishing checks and migration tools for the existing sites.
IndexLane Crawl Fetch Inspector
WordPress Version 0.2.0
A WordPress plugin for checking redirects, indexing rules, sitemaps and structured data from your dashboard. Particularly handy when you’re launching a site or moving one.
IndexLane Asset Weight Snapshot
WordPress Version 0.2.0
A WordPress plugin for checking the CSS and JavaScript your pages reference, measuring the files it can fetch, and comparing the results over time. Useful when you want to see what a plugin update or site change has added.
IndexLane Site Atlas
WordPress Version 0.1.0
A WordPress plugin for figuring out how a site fits together before you start changing things. It looks at the connections between plugins, content and stored data, which is useful when you’re planning a cleanup, rebuild or handover.
Food Cost & Profit Tracker
TypeScript Version 2.2.2
A private app for tracking food costs, stock and estimated profit across direct sales and food delivery. It connects ingredient purchases, recipes, orders and platform fees, helping a seller understand what each dish costs and how much money is actually left.
Comfy Money - Private Multi-Currency Ledger
Go Version 0.14.5
Comfy Money is a private money tracker I built for myself and my family. It handles wallets in different currencies, income, expenses, transfers, budgets and debts, with automatically updated exchange rates and a mobile interface that installs as a PWA.
pgvector-index-manager
pgvector-index-manager
Go Version 0.1.1
A command-line tool for keeping your pgvector indexes in line with a configuration you control. It checks what exists, shows you what needs changing, and can build and replace HNSW indexes when you apply those changes.
Private setup for GitHub Actions runners
GitHub Actions Version 0.1.2
GHA is my private setup for running GitHub Actions runners for multiple repositories on one VDS. It handles runner registration, a shared job queue and resource limits, letting me run CI jobs on my own server while keeping the workflows in GitHub.
Bug Squasher: DOM Defense
TypeScript Version 0.1.0
I was bored and wanted a break from client projects, so I made a little TypeScript game. Naturally, it ended up being about bugs taking down a server.
DG6 — Multilingual pSEO Publishing
PHP Version 1.0.0
A Laravel publishing application for my programmatic SEO projects. DG6 turns generated content into multilingual websites, with separate domains, themes and publishing schedules, all running from one codebase.
DG6 Swarm — AI Content & Media Pipeline
Python Version 0.17
A Python pipeline that turns a domain and starting keyword into site copy, articles and visual assets. Specialized GPU workers handle each stage, starting and stopping according to queued work.