#rfc-9116

Fetch and evaluate any website's robots.txt. Pass a URL and a user-agent and the check endpoint tells you whether that URL is crawlable — selecting the most-specific user-agent group and applying the RFC 9309 longest-match Allow/Disallow rules (with * and $ wildcards, where Allow wins ties), and returning the matched rule, the group's crawl-delay and the sitemaps the site declares. The parse endpoint returns the whole file structured into per-user-agent groups (their allow and disallow lists and crawl-delay) plus the list of sitemaps. A missing robots.txt (404/403) means everything is allowed, exactly as the spec requires. The request is made server-side and private or internal targets are refused (SSRF-guarded). Built for SEO audits, crawler and scraper compliance, sitemap discovery and pre-flight "am I allowed to fetch this?" checks. A robots.txt evaluator — distinct from the on-page SEO audit (seo), the XML toolkit (xml) and link unfurling/preview (url). No upstream key, no cache.

api.oanor.com/robots-api

Fetch and parse any domain's RFC 9116 security.txt — the machine-readable file at /.well-known/security.txt that tells security researchers how to report vulnerabilities. Pass a domain and the service locates the file (the canonical .well-known path with a legacy root fallback), parses every field — Contact, Expires, Encryption, Acknowledgments, Preferred-Languages, Canonical, Policy, Hiring and CSAF — and reports whether it is valid (has at least one Contact and a single, non-expired Expires), whether it is PGP-signed, whether it has expired (with the number of days remaining) and a list of issues with concrete advice. A companion endpoint returns the raw file. The request is made server-side; private and internal targets are refused (SSRF-guarded). Built for security audits, vendor and third-party risk assessment, attack-surface reviews and vulnerability-disclosure-policy compliance checks. A security.txt parser and validator — distinct from the HTTP security-header grader (secheaders), the SSL/TLS certificate check (sslcheck) and host reachability (hostcheck). No upstream key, no cache.

api.oanor.com/securitytxt-api