# robots.txt API
> Fetch and evaluate any website's robots.txt. Pass a URL and a user-agent and the check endpoint tells you whether that URL is crawlable — selecting the most-specific user-agent group and applying the RFC 9309 longest-match Allow/Disallow rules (with * and $ wildcards, where Allow wins ties), and returning the matched rule, the group's crawl-delay and the sitemaps the site declares. The parse endpoint returns the whole file structured into per-user-agent groups (their allow and disallow lists and crawl-delay) plus the list of sitemaps. A missing robots.txt (404/403) means everything is allowed, exactly as the spec requires. The request is made server-side and private or internal targets are refused (SSRF-guarded). Built for SEO audits, crawler and scraper compliance, sitemap discovery and pre-flight "am I allowed to fetch this?" checks. A robots.txt evaluator — distinct from the on-page SEO audit (seo), the XML toolkit (xml) and link unfurling/preview (url). No upstream key, no cache.

## Authentication
All requests require your oanor API key in the `x-oanor-key` header. Get one at https://www.oanor.com/developer/keys.

```bash
curl -H "x-oanor-key: oanor_live_…" "https://api.oanor.com/robots-api/..."
```

## Pricing
- **Free** (Free) — 2,000 calls/Mo, 2 req/s
- **Starter** ($6/Mo) — 37,500 calls/Mo, 8 req/s
- **Pro** ($20/Mo) — 202,000 calls/Mo, 20 req/s
- **Mega** ($52/Mo) — 785,000 calls/Mo, 50 req/s

## Endpoints

### robots.txt

#### `GET /v1/check` — Is a URL crawlable?

**Parameters:**
- `url` (query, required, string) — The URL to check Example: `https://www.google.com/`
- `user_agent` (query, optional, string) — User-agent (default *) Example: `Googlebot`

**Example:**
```bash
curl -H "x-oanor-key: $KEY" \
  "https://api.oanor.com/robots-api/v1/check?url=https%3A%2F%2Fwww.google.com%2F&user_agent=Googlebot"
```

#### `GET /v1/parse` — Parse a site's robots.txt

**Parameters:**
- `url` (query, required, string) — Site URL or domain Example: `https://www.google.com`

**Example:**
```bash
curl -H "x-oanor-key: $KEY" \
  "https://api.oanor.com/robots-api/v1/parse?url=https%3A%2F%2Fwww.google.com"
```

### Meta

#### `GET /v1/meta` — Spec

**Example:**
```bash
curl -H "x-oanor-key: $KEY" \
  "https://api.oanor.com/robots-api/v1/meta"
```


---
Marketplace page: https://www.oanor.com/api/robots-api
OpenAPI spec: https://www.oanor.com/api/robots-api/openapi.json