crawlreplay

A command-line tool for testing robots.txt changes and new deployments against URLs found in your server logs. It helps you catch newly blocked pages, broken responses and unexpected indexing changes before putting those changes live.

DownloadVersion 0.1.0

crawlreplay

About crawlreplay

When checking a new version of a site, you’ll probably open the homepage, a few important pages and whatever you’ve just changed. Meanwhile, crawlers are still requesting an old URL from three migrations ago and a page you’d completely forgotten existed. Conveniently, there’s a record of that.

CrawlReplay uses those logs as the starting point. Give it your current and proposed robots.txt files, and it shows which logged crawler requests would become blocked or allowed under the new rules. Handy when a small cleanup turns out to cover rather more URLs than intended.

For a deployment check, it takes a sample of logged URLs and requests them from both the current site and your preview environment. It compares responses, redirects, canonical URLs and indexing directives, and flags changes to language links, structured-data types and page text.

The question is fairly simple: these URLs are already being requested—what happens to them after this change?

It works with Nginx and Cloudflare Logpush logs, with reports you can read locally or use in an automated workflow. You can set thresholds for newly blocked URLs, new noindex directives and worsening HTTP responses, so those findings can fail a check before a release goes ahead.

It’s written in Rust and is still in early development. The checks cover the logged URLs you feed it and the sample you choose for a deployment comparison, so the logs you use matter.

Release history

Version 0.1.01 Aug 2026

Added

Workspace with crawlreplay-core, crawlreplay-import, and crawlreplay-report crates, plus a binary crate with CLI entry point.
policy subcommand that evaluates logged crawler requests against old and new robots.txt files and reports policy transitions (newly blocked, newly allowed) with per-crawler severity.
release subcommand that fetches baseline and candidate URLs for each sampled log entry, then compares status codes, redirect chains, canonical links, indexability directives, hreflang alternates, schema.org types, and content fingerprints through the comparison pipeline.
Explicit --baseline-origin and --origin allowlists, so log host fields do not control outbound network destinations.
Streaming Nginx combined-log parser with bounded record handling.
Streaming Nginx JSON log parser with configurable field mappings.
Streaming Cloudflare Logpush HTTP request log parser.
Shared parse_line() function in crawlreplay-import with automatic format detection or explicit --log-format selection, used by both policy and release commands.
Input from plain text, .gz (auto-detected by magic bytes), and stdin.
Crawler classification for Google, Bing, OpenAI, Anthropic, and Perplexity with User-Agent pattern matching.
verified, claimed, spoofed, and unknown identity verdicts. Supported Google common crawlers and Bingbot are checked against exact bundled CIDR snapshots with documented source timestamps and checksums.
robots.txt matching for Allow, Disallow, wildcards, end anchors ($), longest-match-wins semantics, and empty-disallow-means-allow.
Policy evaluation engine with URL/crawler deduplication, request counting, and deterministic finding ordering.
Release comparison engine covering 13 finding rules: status_regression, baseline_failure, candidate_failure, redirect_changed, redirect_chain_grew, canonical_changed, canonical_leak, newly_noindex, newly_blocked, newly_allowed, hreflang_changed, schema_type_removed, and content_changed.
Async HTTP fetcher with bounded task concurrency, separate baseline and candidate request-start limits, configurable timeouts, exact-origin redirects, streaming response limits, proxy isolation, and DNS-aware special-address blocking.
--requests-per-second flag on the release subcommand (default 10.0) controlling the fetcher rate limit.
HTML extraction for canonical links, robots meta tags, X-Robots-Tag headers, hreflang alternates, JSON-LD @type values, and SHA-256 content fingerprints over visible text.
URL normalization (lowercase scheme and host, fragment removal, default-port stripping).
Query-safe replay by default: parameter names and duplicate order are retained while values are sent empty. --replay-query-values is an explicit privacy-sensitive opt-in.
Query-value redaction in reports, diagnostics, redirects, canonicals, and hreflang evidence.
Client addresses used for crawler verification remain in memory and are suppressed from reports by default.
Deterministic sampling for release replay with --max-urls and --seed.
Terminal output with severity coloring, grouped findings, and request counts; respects NO_COLOR and TERM=dumb.
JSON report with schema version, severity/rule summaries, findings array, and exit-decision field.
Markdown report with severity summary table and per-rule finding tables (collapsible overflow at 20 entries).
SARIF 2.1.0 report with rule definitions, severity mapping, artifact locations, and deduplication fingerprints for GitHub code scanning.
CI threshold configuration for newly_blocked, status_regression, and newly_noindex with unique_urls and weighted_requests gates.
--thresholds <path> flag on both policy and release subcommands to load threshold configuration from a JSON file.
JSON schemas for threshold configuration (docs/threshold-schema.json) and report output (docs/report-schema.json), plus an example threshold file (docs/threshold-example.json).
Six deterministic exit codes from 0 through 5, separating invalid arguments, imports, network comparison failure, and internal errors.
Configuration struct with defaults for concurrency (10), timeout (30 s), max response size (5 MiB), max redirects (10), and rate limit (10 req/s).
Error types using thiserror for core, import, and report layers.
Unit tests for identity classification, IP verification, robots parsing, policy evaluation, URL normalization, query redaction, HTML extraction, content fingerprinting, private-host detection, and shared parse_line auto-detection.
Integration tests covering policy replay with all three log formats, all four output formats, verified-only filtering, invalid-argument handling, output determinism, threshold config from file, and invalid threshold file handling.
Test fixtures: sample Nginx combined, Nginx JSON, and Cloudflare logs plus old/new robots.txt files.
Multi-stage Dockerfile with digest-pinned Rust and Debian bases, a non-root runtime user, and the project license included in the runtime image.
GitHub Action composite wrapper (action/action.yml) supporting both policy and release subcommands.
CI checks for formatting, Clippy, tests, known dependency vulnerabilities, crate assembly, the local composite action, and the container image.
README, fixture provenance, threat model, security policy, and contribution guide with privacy-safe examples and operator-facing wording.
Apache-2.0 license.
MSRV 1.86, edition 2021, resolver 2.

External download

Download 0.1.0

crawlreplay

About crawlreplay

Release history

More software