unfurlr-bot

unfurlr-bot is the fetcher behind the unfurlr link-preview API. It identifies itself with this User-Agent:

unfurlr-bot/1.0 (+https://unfurlr.onrender.com/bot)

What it does

• It fetches a single page only when an API user asks for a preview of that exact URL, the same way Slack, Discord and iMessage fetch a link someone pastes. It doesn't crawl, follow links, or index sites.
• It reads only the page metadata: title, description, Open Graph and Twitter tags, favicon and icons, canonical link, JSON-LD and oEmbed links. If the page advertises a JSON oEmbed endpoint, it fetches that too. For X (Twitter) post links it doesn't fetch anything: it builds X's standard embed from the link itself.
• It doesn't run JavaScript or download images. It reads at most the first 512 KB of a page, times out after 12 seconds, and retries once at most.
• It always sends its own User-Agent above. It never pretends to be a browser or another company's bot.
• Results are cached for up to 24 hours, so the same URL is fetched rarely. Per-site rate limits keep traffic to any one site low.
• It never fetches private or internal network addresses.

robots.txt

Single-link previews requested with an API key are user-triggered, not crawling, so unfurlr-bot doesn't apply robots.txt to them. That's the same approach other link unfurlers take. Batch requests (several URLs at once) and the anonymous demo on our homepage do honour robots.txt rules for unfurlr-bot. If you don't want your site previewed at all, use the opt-out below. It covers every kind of request.

Opting out

To stop unfurlr-bot from fetching your site, email smaran+unfurlr@mail.grokbot.com from an address at that domain, or with some other proof that you control it, such as a TXT record or a file we ask you to place. We'll add the domain and all its subdomains to our opt-out list, once we have verified it. From then on, API users get a site_opted_out error and your site isn't fetched, even from cache. You can also block the User-Agent unfurlr-bot at your server or CDN.

Abuse

To report abuse, such as excessive requests or the API being used against your site, email smaran+unfurlr@mail.grokbot.com. Include the URLs and times (with timezone) you saw.