IndexLane Crawl Fetch Inspector
A WordPress plugin for checking redirects, indexing rules, sitemaps and structured data from your dashboard. Particularly handy when you’re launching a site or moving one.
DownloadVersion 0.2.0
Images
About IndexLane Crawl Fetch Inspector
A website can look perfectly fine while quietly pointing Google towards the wrong URL. Or blocking a page you wanted indexed. Or still mentioning the staging domain you thought you’d removed everywhere. That last one has a habit of hiding.
Crawl Fetch Inspector puts these checks in one place. Give it a few URLs, include recently updated posts and pages, or let it take a sample from your sitemap. There’s also a launch and migration check that brings those checks together and looks for references to old domains.
For each page, you can follow the redirects, check its canonical URL and indexing instructions, and see whether robots.txt allows Googlebot to crawl it. It also checks sitemap membership and looks through the structured data for things such as invalid JSON or duplicate IDs.
When it finds an old or staging-domain reference, the report shows the value and where it came from. You get something specific to investigate.
The checks are read-only, so you can look around without changing your content or settings. Incomplete checks are marked as such, and you can export the results to CSV when you need to work through them or pass them to someone else.
Release history
Version 0.2.0
Crawl Fetch Inspector 0.2.0
This release rebuilds Crawl Fetch Inspector around one bounded, same-site WordPress fetch engine and expands the plugin into a broader crawler-facing evidence tool.
Highlights
- Three scan modes: manual/recent URLs, distributed sitemap samples, and launch/migration checks.
- Googlebot-specific robots.txt evaluation with crawler group precedence and longest-match Allow/Disallow behavior.
- Automatic child-sitemap expansion, sampling across child sitemaps, resumable AJAX batches, and conservative Yes, No, or Unknown—incomplete membership.
- JSON-LD block counts, raw JSON validity, @type inventory, and duplicate @id evidence.
- Administrator-supplied old domains plus exact URL evidence from href, src, canonical/Open Graph fields, CSS URLs, and JSON-LD properties.
- Complete, Partial, and Failed evidence states, sectioned reports, and expanded CSV export.
- Behavioral fixtures for redirects, response truncation, robots precedence, sitemap partial evidence, JSON-LD, and migration residue.
The plugin remains read-only and same-site. External redirect destinations are recorded but not fetched.
External download
Download 0.2.0
Version 0.1.2
Fix crawl fetch correctness
External download
Download 0.1.2
Version 0.1.0
Initial release
External download
Download 0.1.0
More software
IndexLane Safe WebP Queue
WordPress Version 1.0.1
A WordPress plugin that creates WebP copies of your images while keeping the originals. Convert on the server or use the optional browser mode when your host cannot write WebP, with progress reporting and checks before converted files are saved.
Classifieds & Advertising Platform — Go Rebuild
Go Version 1.17.10
An ongoing Go rebuild of my Laravel classifieds platform, with a React dashboard, incremental listing updates and background jobs handled through NATS JetStream. It preserves the existing platform’s behaviour while changing how the application processes and serves its data.
pgvector-filterbench
pgvector-filterbench
Go Version 0.1.2
A command-line tool for checking how pgvector search behaves once you add filters. It measures speed, compares approximate results with exact search, and checks query plans to see whether PostgreSQL is using your HNSW index.
wp-migrate-core
wp-migrate-core
Typescript Version 0.3.0
A command-line tool for getting a WordPress-to-Astro migration started. It reads a WordPress XML export, inventories content, images and links, and can generate an Astro project with a clear list of what still needs rebuilding.
hueframe
TypeScript Version 1.0.3
A tool for turning a photo into a website color theme. Drop in an image, see its colors applied to a page, adjust the result and export it as CSS variables or JSON design tokens.
SSRWire
Typescript Version 0.5.0
A command-line tool for inspecting the HTML your server sends, including content and metadata that arrive in a stream. It records when important elements appear and compares responses for different browser, search-crawler and social-preview user agents.
routeplay
routeplay
TypeScript Version 0.4.0
A command-line tool for checking that a page delivers the right content and metadata however you reach it. It compares the HTML your server sends, a fresh browser load and the result of clicking through from another page.
ReachLog
Typescript Version 0.2.1
ReachLog is a small outreach and contact manager for keeping track of who you’ve contacted, what you’ve discussed and when to follow up. It brings contacts, companies and email conversations together in a local application that works with your existing email account.
IndexLane Redirect & Internal Link Auditor
Wordpress Version 0.7.0
A WordPress plugin for finding broken internal links, redirects and old URLs left behind after a migration. It shows you where those links are used and takes you to the place where you can edit them.
routelint
routelint
TypeScript Version 0.3.1
A command-line tool for finding broken links, incorrect redirects and indexing conflicts on a live or preview website. It discovers URLs, checks what your server returns and gives you findings tied to specific pages, with the evidence behind them.
Data-Driven Publishing Platform
Data-Driven Publishing Platform
Laravel Version 0.14.1
A publishing platform for running several websites from structured source data. It handles imports, page generation, scheduled updates and search monitoring through one administration panel. Four domains are live; getting traffic to them remains the unfinished part.
Classifieds & Advertising Platform
Laravel Version 1.3.6
A classifieds and advertising platform I’ve been building and maintaining for roughly two years. It combines location-based browsing, advertiser dashboards, paid listings, invoicing, live chat and an administration panel for running the site.
Private Organic Search & LLM Activity Tracker
Elixir Version 0.39.19
A private analytics tool for tracking website traffic, organic search performance, and activity from search engines and LLM crawlers. It brings ordinary visitor statistics, crawler requests, Google Search Console and Bing Webmaster Tools into one dashboard.
ILCrawler
Go Version 0.41.0
ILCrawler is the technical SEO crawler I use for my own websites and client audits. It brings page checks, links, screenshots, performance samples and reporting into one place. It’s also the project that repeatedly convinced me I was only one rewrite away from being happy with it.
crawlreplay
crawlreplay
Rust Version 0.1.0
A command-line tool for testing robots.txt changes and new deployments against URLs found in your server logs. It helps you catch newly blocked pages, broken responses and unexpected indexing changes before putting those changes live.
Data-Driven Publishing Platform — Go Rebuild
Data-Driven Publishing Platform — Go Rebuild
Go Version 0.226.2
An unlaunched Go rebuild of my publishing platform, using PostgreSQL for application data and background jobs, React for administration and Astro for public websites. It includes source processing, publishing checks and migration tools for the existing sites.
IndexLane Asset Weight Snapshot
WordPress Version 0.2.0
A WordPress plugin for checking the CSS and JavaScript your pages reference, measuring the files it can fetch, and comparing the results over time. Useful when you want to see what a plugin update or site change has added.
IndexLane Site Atlas
WordPress Version 0.1.0
A WordPress plugin for figuring out how a site fits together before you start changing things. It looks at the connections between plugins, content and stored data, which is useful when you’re planning a cleanup, rebuild or handover.
Food Cost & Profit Tracker
TypeScript Version 2.2.2
A private app for tracking food costs, stock and estimated profit across direct sales and food delivery. It connects ingredient purchases, recipes, orders and platform fees, helping a seller understand what each dish costs and how much money is actually left.
Comfy Money - Private Multi-Currency Ledger
Go Version 0.14.5
Comfy Money is a private money tracker I built for myself and my family. It handles wallets in different currencies, income, expenses, transfers, budgets and debts, with automatically updated exchange rates and a mobile interface that installs as a PWA.
crawlcontract
crawlcontract
Rust Version 0.3.1
A command-line tool for catching crawl and indexing problems when a website changes. It checks built files or pages fetched from a live or preview site, and compares scans so you can spot problems before a release.
pgvector-index-manager
pgvector-index-manager
Go Version 0.1.1
A command-line tool for keeping your pgvector indexes in line with a configuration you control. It checks what exists, shows you what needs changing, and can build and replace HNSW indexes when you apply those changes.
Private setup for GitHub Actions runners
GitHub Actions Version 0.1.2
GHA is my private setup for running GitHub Actions runners for multiple repositories on one VDS. It handles runner registration, a shared job queue and resource limits, letting me run CI jobs on my own server while keeping the workflows in GitHub.
Bug Squasher: DOM Defense
TypeScript Version 0.1.0
I was bored and wanted a break from client projects, so I made a little TypeScript game. Naturally, it ended up being about bugs taking down a server.
DG6 — Multilingual pSEO Publishing
PHP Version 1.0.0
A Laravel publishing application for my programmatic SEO projects. DG6 turns generated content into multilingual websites, with separate domains, themes and publishing schedules, all running from one codebase.
DG6 Swarm — AI Content & Media Pipeline
Python Version 0.17
A Python pipeline that turns a domain and starting keyword into site copy, articles and visual assets. Specialized GPU workers handle each stage, starting and stopping according to queued work.