#

robotstxt

Here are 2 public repositories matching this topic...

nicholasbergesen / robots-parser

Parse robots.txt and traverse sitemaps.

sitemap spider crawl robots-txt robots useragent robots-parser useragentparser robotstxt sitemap-url

Updated Feb 7, 2025
C#

toimik / RobotsProtocol

Parsers for robots.txt (aka Robots Exclusion Standard / Robots Exclusion Protocol), Robots Meta Tag, and X-Robots-Tag

robots-txt robots meta-tags robots-parser robots-exclusion-standard robotstxt robots-tag robots-exclusion-protocol x-robots-tag

Updated May 2, 2024
C#

Improve this page

Add a description, image, and links to the robotstxt topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the robotstxt topic, visit your repo's landing page and select "manage topics."