robotparser
robotparser — Parser for robots.txt
Note
The robotparser module has been renamed urllib.robotparser in Python 3. The 2to3 tool will automatically adapt imports when converting your sources to Python 3.
This module provides a single class, RobotFileParser, which answers questions about whether or not a particular user agent can fetch a URL on the W
