Clarify /crawl changelog (#28914)

* Clarify /crawl changelog

* minor changes

Author: rita3ko

src/content/changelog/browser-rendering/2026-03-10-br-crawl-endpoint.mdx

@@ -6,7 +6,9 @@ products:
 date: 2026-03-10
 ---
 
-You can now crawl an entire website with a single API call using [Browser Rendering](/browser-rendering/)'s new [`/crawl` endpoint](/browser-rendering/rest-api/crawl-endpoint/), available in open beta. Submit a starting URL, and pages are automatically discovered, rendered in a headless browser, and returned in multiple formats, including HTML, Markdown, and structured JSON. This is great for training models, building RAG pipelines, and researching or monitoring content across a site.
+_Edit: this post has been edited to clarify crawling behavior with respect to site guidance._
+
+You can now crawl an entire website with a single API call using [Browser Rendering](/browser-rendering/)'s new [`/crawl` endpoint](/browser-rendering/rest-api/crawl-endpoint/), available in open beta. Submit a starting URL, and pages are automatically discovered, rendered in a headless browser, and returned in multiple formats, including HTML, Markdown, and structured JSON. The endpoint is a [signed-agent](https://developers.cloudflare.com/bots/concepts/bot/signed-agents/) that respects robots.txt and [AI Crawl Control](https://www.cloudflare.com/ai-crawl-control/) by default, making it easy for developers to comply with website rules, and making it less likely for crawlers to ignore web-owner guidance. This is great for training models, building RAG pipelines, and researching or monitoring content across a site.
 
 Crawl jobs run asynchronously. You submit a URL, receive a job ID, and check back for results as pages are processed.
 
@@ -33,7 +35,9 @@ Key features:
 - **Static mode** - Set `render: false` to fetch static HTML without spinning up a browser, for faster crawling of static sites
 - **Well-behaved bot** - Honors `robots.txt` directives, including `crawl-delay`
 
-Available on both the Workers Free and Paid plans.
+Available on both the Workers Free and Paid plans. 
+
+**Note**: the /crawl endpoint cannot bypass Cloudflare bot detection or captchas, and self-identifies as a bot.
 
 To get started, refer to the [crawl endpoint documentation](/browser-rendering/rest-api/crawl-endpoint/). 
 If you are setting up your own site to be crawled, review the [robots.txt and sitemaps best practices](/browser-rendering/reference/robots-txt/).