🤖 robots.txt Tester for Large Scale Testing

For a given robots.txt file check which URLs can be fetched for all user-agents in the file

Data source: advertools robotstxt module
Tags: google, SEO, robots.txt, search engine optimization,
Code: Questions, bugs, suggestions more than welcome!

About
A collection of dashboards and apps, made by Elias Dabbas.
For data exploration, learning, fun, prototyping, and sharing.
Mostly made with Plotly's Dash (Python).