crawlreplay
Rust Version 0.1.0
A command-line tool for testing robots.txt changes and new deployments against URLs found in your server logs. It helps you catch newly blocked pages, broken responses and unexpected indexing changes before putting those changes live.