Spec

Code snippets

curl "https://api.oanor.com/robots-api/v1/meta" \
  -H "x-oanor-key: oanor_test_..."

await fetch("https://api.oanor.com/robots-api/v1/meta", {
  headers: { "x-oanor-key": "oanor_test_..." }
});

$ch = curl_init("https://api.oanor.com/robots-api/v1/meta");
curl_setopt($ch, CURLOPT_HTTPHEADER, ["x-oanor-key: oanor_test_..."]);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$out = curl_exec($ch);

import requests
requests.get(
    "https://api.oanor.com/robots-api/v1/meta",
    headers={"x-oanor-key": "oanor_test_..."}
)

Example response

A real response from this endpoint, captured by the latest health check.

{
    "data": {
        "note": "Fetch and evaluate a website's robots.txt. /v1/check?url=https://example.com/path&user_agent=Googlebot tells you whether that URL is crawlable by that user-agent — selecting the most-specific user-agent group and applying RFC 9309 longest-match Allow/Disallow rules with * and $ wildcards (Allow wins ties), and returning the matched rule, the group's crawl-delay and the declared sitemaps. /v1/parse returns the whole file parsed into per-user-agent groups (allow/disallow lists, crawl-delay) and the sitemap list. A missing robots.txt (404/403) means everything is allowed, per spec. The request is made server-side; private/internal targets are refused (SSRF-guarded). Ideal for SEO audits, crawler and scraper compliance, sitemap discovery and pre-flight 'can I fetch this?' checks. A robots.txt evaluator — distinct from the on-page SEO audit (seo), XML tooling (xml) and link unfurling (url). No key, no cache.",
        "spec": "RFC 9309 (robots.txt)",
        "endpoints": [
            "/v1/check",
            "/v1/parse",
            "/v1/meta"
        ]
    },
    "meta": {
        "timestamp": "2026-06-01T23:40:40.518Z",
        "request_id": "a4cbd306-e093-4e36-84eb-a2e1d3ab7eca"
    },
    "status": "ok",
    "message": "Meta retrieved",
    "success": true
}