[Various] Bots/Rules discoverability (#28931)

* bots / rules clarification

* edits

* edits

* title

* pm feedback

* pcx feedback

Author: patriciasantaana

src/content/docs/bots/additional-configurations/custom-rules.mdx

@@ -0,0 +1,81 @@
+---
+title: Custom rules
+pcx_content_type: concept
+sidebar:
+  order: 7
+description: >-
+  Understand when to use the built-in bot protection settings in Security
+  Settings versus creating WAF custom rules for bot management.
+---
+
+import { Render } from "~/components";
+
+Bot protection on Cloudflare works through two complementary mechanisms: built-in settings configured through toggles in **Security Settings**, and [WAF custom rules](/waf/custom-rules/) that you write using [bot management fields](/bots/reference/bot-management-variables/). Understanding when to use each approach helps you avoid creating duplicate rules and simplifies your security configuration.
+
+The following features are configured through toggles and dropdowns in [Security Settings](/security/settings/). They do not require you to write any rule expressions.
+
+| Feature | What it does | Availability |
+| --- | --- | --- |
+| [Block AI bots](/bots/additional-configurations/block-ai-bots/) | Blocks AI crawlers (GPTBot, ClaudeBot, Bytespider, and others) using an auto-updating managed rule | All plans |
+| [AI Labyrinth](/bots/additional-configurations/ai-labyrinth/) | Feeds non-compliant AI crawlers into a maze of generated content | All plans |
+| [Managed robots.txt](/bots/additional-configurations/managed-robots-txt/) | Prepends AI crawler disallow directives to your `robots.txt` | All plans |
+| Super Bot Fight Mode > **Definitely automated** | Blocks or challenges traffic with a [bot score](/bots/concepts/bot-score/) of 1 | Pro, Business, Enterprise |
+| Super Bot Fight Mode > **Likely automated** | Blocks or challenges traffic with a bot score of 2-29 | Business, Enterprise |
+| [Verified bots](/bots/concepts/bot/verified-bots/) | Managed category of high-trust bots (Googlebot, Bingbot, and others) | Pro, Business, Enterprise |
+| [Static resource protection](/bots/additional-configurations/static-resources/) | Extends bot actions to cover static file types | Pro, Business, Enterprise |
+| [Optimize for WordPress](/bots/troubleshooting/wordpress-loopback-issue/) | Allows WordPress loopback requests through bot protection | Pro, Business, Enterprise |
+| [JavaScript detections](/cloudflare-challenges/challenge-types/javascript-detections/) | Injects a lightweight script to identify clients that cannot execute JavaScript | All plans (automatic on Free) |
+
+Bot settings update automatically as Cloudflare identifies new bot signatures and AI crawlers, while custom rules require manual updates. They do not count toward your [custom rule limits](/waf/custom-rules/#availability), and apply uniformly across your domain without the risk of expression errors.
+
+## Custom rules use cases
+
+Custom rules are valuable when you need capabilities that built-in settings do not offer. The following scenarios require [WAF custom rules](/waf/custom-rules/) with [bot management fields](/bots/reference/bot-management-variables/). Bot management fields are available to customers with a [Bot Management](/bots/get-started/bot-management/) subscription.
+
+### Path-specific protection
+
+Since Bot settings apply to all traffic across your domain, you may need an alternative approach to bot handling for different paths using custom rules — for example, stricter protection on `/login/` than on `/public/`.
+
+#### Example 
+
+Block likely automated traffic only on your login endpoint:
+
+```txt
+(cf.bot_management.score lt 30 and not cf.bot_management.verified_bot and http.request.uri.path eq "/login")
+```
+
+### Custom score thresholds
+
+The **Definitely automated** and **Likely automated** settings in Super Bot Fight Mode use fixed bot score groupings (1 and 2-29). If you need a different threshold, for example, challenging all traffic with a score below 20, you need a custom rule.
+
+### Conditional logic
+
+If you need to combine bot score with other request fields, such as country, ASN, URI path, JA3/JA4 fingerprint, or user agent, you need custom rules. Bot settings do not support compound conditions.
+
+#### Example
+
+Challenge likely automated traffic only from specific ASNs:
+
+```txt
+(cf.bot_management.score lt 30 and not cf.bot_management.verified_bot and ip.src.asnum in {64496 65536})
+```
+
+### Custom actions
+
+Bot settings offer **Block**, **Managed Challenge**, and **Allow** as actions. 
+
+If you need other actions, such as **Log** (for testing rules before enforcement), **Interactive Challenge**, or **Skip** (to bypass other rules), you need custom rules.
+
+### Detection ID targeting
+
+To act on specific bot heuristic detections, such as [account takeover](/bots/additional-configurations/detection-ids/account-takeover-detections/) or [scraping](/bots/additional-configurations/detection-ids/scraping-detections/) patterns, you need custom rules using the `cf.bot_management.detection_ids` field. Bot settings do not expose individual detection IDs.
+
+### Forwarding bot data to origin
+
+To send bot scores, verified bot status, or JA3/JA4 fingerprints to your origin server, use [Transform Rules](/rules/transform/) (including [Managed Transforms](/rules/transform/managed-transforms/)) or [Snippets](/rules/snippets/). These are not part of the built-in bot settings.
+
+## Execution order
+
+Custom rules execute before Super Bot Fight Mode managed rules. If a custom rule takes a terminating action (such as **Block** or **Managed Challenge**), the request does not reach bot settings. 
+
+Refer to [Security features interoperability](/waf/feature-interoperability/) for more information.

src/content/docs/bots/additional-configurations/managed-robots-txt.mdx

@@ -2,7 +2,7 @@
 pcx_content_type: reference
 title: robots.txt setting
 sidebar:
-  order: 7
+  order: 8
   label: robots.txt setting
 ---
 

src/content/docs/bots/additional-configurations/static-resources.mdx

@@ -2,7 +2,7 @@
 pcx_content_type: reference
 title: Static resource protection
 sidebar:
-  order: 8
+  order: 9
 ---
 
 import {

src/content/docs/bots/concepts/bot/index.mdx

@@ -66,12 +66,7 @@ This behavior remains the same if the setting for verified, definitely automated
 
 For self-serve non-Bot Management customers, all rules for verified, definitely automated, and likely bots run in the phase following the AI bots rule.
 
-```mermaid
-flowchart LR
-accTitle: AI bots rule phases diagram
-accDescr: This diagram details the phases in which AI bots rules run.
-A[Custom rules] --> B[Block AI bots<br>managed rule] --> C[Allow verified bots rule]
-```
+<Render file="execution-order" product="bots" />
 
 This feature is available on all Cloudflare plans.
 

src/content/docs/bots/get-started/bot-management.mdx

@@ -77,9 +77,15 @@ Refer to [Block AI bots](/bots/additional-configurations/block-ai-bots/).
 You can view blocked AI bot traffic via [Security Analytics](/waf/analytics/security-analytics/).
 :::
 
-### Deploy default templates
+### Deploy custom rule templates
 
-Cloudflare has [default templates](https://dash.cloudflare.com/?to=/:account/:zone/security/security-rules?template=bot_traffic) for definite bots, which have a [bot score](/bots/concepts/bot-score/) of 1, and likely bots which have a bot score of 2 to 29. In our templates, Cloudflare recommends to allow verified bots such as Google SEO Crawler and access to cached static resources.
+The **Security Settings** toggles you configured above already provide baseline protection against definitely automated and likely automated traffic. 
+
+If you need additional control, such as path-specific protection, custom score thresholds, or combining bot score with other fields, Cloudflare provides [rule templates](https://dash.cloudflare.com/?to=/:account/:zone/security/security-rules?template=bot_traffic) to get started.
+
+:::note
+Custom rules created from these templates execute before the managed rules configured in **Security Settings**. For more details on this execution order, refer to [Security features interoperability](/waf/feature-interoperability/).
+:::
 
 - [Definite Bots template](https://dash.cloudflare.com/?to=/:account/:zone:/security/security-rules/custom-rules/create?template=Definitely%20Bots): Targets malicious bot traffic while ignoring verified bots and routes delivering static content.
 
@@ -93,7 +99,7 @@ Cloudflare has [default templates](https://dash.cloudflare.com/?to=/:account/:zo
   (cf.bot_management.score ge 2 and cf.bot_management.score le 29 and not cf.bot_management.verified_bot and not cf.bot_management.static_resource)
   ```
 
-- (Optional) [JavaScript detections template](https://dash.cloudflare.com/?to=/:account/:zone/security/security-rules/custom-rules/create?template=JavaScript%20Verified%20URLs): If you enabled JavaScript detections, then set up a [managed challenge](/cloudflare-challenges/challenge-types/challenge-pages/#managed-challenge), make sure to add a method and URI path. JavaScript detections improves security for URLs that should only expect JavaScript-enabled clients.
+- (Optional) [JavaScript detections template](https://dash.cloudflare.com/?to=/:account/:zone/security/security-rules/custom-rules/create?template=JavaScript%20Verified%20URLs):  You must first enable JavaScript Detections from Security Settings, then set up a [managed challenge](/cloudflare-challenges/challenge-types/challenge-pages/#managed-challenge). Make sure to add a method and URI path. JavaScript detections improves security for URLs that should only expect JavaScript-enabled clients.
 
   ```txt wrap
   (not cf.bot_management.js_detection.passed and http.request.method eq "" and http.request.uri.path in {""})

src/content/docs/bots/plans/biz-and-ent.mdx

@@ -84,6 +84,8 @@ import { Render } from "~/components";
 
 [^1]: When users purchase Bot Management for Enterprise, Cloudflare automatically replaces and disables other bot products to prevent overlap.
 
+<Render file="bot-settings-vs-custom-rules" product="bots" />
+
 ## How do I get started?
 
 <Render file="plan-get-started" product="bots" />

src/content/docs/bots/plans/bm-subscription.mdx

@@ -99,6 +99,24 @@ import { Render } from "~/components";
 Zones that have [Enterprise Bot Management](/bots/get-started/bot-management/) enabled will not see Bot Fight Mode or Super Bot Fight Mode under **Security** > **Bots**.
 :::
 
+## Bot settings vs. custom rules
+
+Bot Management customers have both bot settings (configured in **Security Settings**) and the ability to create custom rules using bot score fields. Start with the bot settings for baseline protection, then add custom rules only when you need additional control.
+
+| Feature | Handled by bot settings | When to use custom rules instead |
+| --- | --- | --- |
+| Block or challenge definitely automated traffic | No | Path-specific rules, custom thresholds, or combining with other fields |
+| Block or challenge likely automated traffic | No | Path-specific rules, custom thresholds, or combining with other fields |
+| Allow or block verified bots | No | Granular control by verified bot category |
+| Block AI crawlers | Yes | Target individual AI crawlers using detection IDs |
+| Protect static resources | No | Exclude static resources from specific rules |
+| Optimize for WordPress | No | No |
+| Forward bot data to origin | No | Use [Transform Rules](/rules/transform/) or [Snippets](/rules/snippets/) |
+| Detection ID targeting | No | Use `cf.bot_management.detection_ids` in [custom rules](/waf/custom-rules/) |
+| JA3/JA4 fingerprint rules | No | Use `cf.bot_management.ja3_hash` or `cf.bot_management.ja4` in [custom rules](/waf/custom-rules/) |
+
+For more details on when custom rules are needed, refer to [custom rules](/bots/additional-configurations/custom-rules/).
+
 ## How do I get started?
 
 <Render file="plan-get-started" product="bots" />

src/content/docs/bots/plans/pro.mdx

@@ -80,6 +80,8 @@ import { Render } from "~/components";
 	</tbody>
 </table>
 
+<Render file="bot-settings-vs-custom-rules" product="bots" />
+
 ## How do I get started?
 
 <Render file="plan-get-started" product="bots" />

src/content/docs/waf/custom-rules/use-cases/challenge-bad-bots.mdx

@@ -9,6 +9,21 @@ Cloudflare's Bot Management feature scores the likelihood that a request origina
 Access to [Bot Management](/bots/plans/bm-subscription/) requires a Cloudflare Enterprise plan with Bot Management enabled.
 :::
 
+## Bot settings
+
+Before creating custom rules for bot protection, review the settings on your [Security Settings](/security/) page under **Bot traffic**. Built-in features auto-update with new bot signatures, do not count toward your custom rule limits, and are simpler to manage.
+
+| Use case | Bot setting |
+| --- | --- |
+| Block AI crawlers (GPTBot, ClaudeBot, etc.) | **Block AI bots** |
+| Block definitely automated traffic (bot score of 1) | **Definitely automated** |
+| Challenge likely automated traffic (bot score 2-29) | **Likely automated** |
+| Allow verified bots (Googlebot, Bingbot, etc.) | **Verified bots** |
+| Extend bot protection to static resources | **Static resource protection** | **Security Settings** > **Bot traffic** |
+| Allow WordPress loopback requests | **Optimize for WordPress** | **Security Settings** > **Bot traffic** |
+
+Custom rules are still valuable when you need path-specific protection (different handling for `/api/` vs. `/login/`), custom score thresholds (for example, score below 20 instead of 30), conditional logic combining bot score with other fields, or custom actions not available in the built-in settings.
+
 Bot score ranges from 1 through 99. A low score indicates the request comes from a script, API service, or an automated agent. A high score indicates that a human issued the request from a standard desktop or mobile web browser.
 
 These examples use:
@@ -28,6 +43,10 @@ Your rules may also vary based on the [nature of your site](/bots/get-started/bo
 
 ### General protection
 
+:::note
+Custom rules execute before [Super Bot Fight Mode](/bots/get-started/super-bot-fight-mode/#waf-custom-rules). If you already configured actions for **Definitely automated** and **Likely automated** traffic in **Security Settings**, deploying these custom rules creates additional rules that take priority over those settings on matching traffic.
+:::
+
 The following three custom rules provide baseline protection against malicious bots:
 
 **Rule 1:**

src/content/docs/waf/get-started.mdx

@@ -109,15 +109,30 @@ If you are an Enterprise customer, do the following:
 
 If you are on a Business plan, create a custom rule as mentioned above but use the [WAF Attack Score Class](/waf/detections/attack-score/#available-scores) field instead. For example, you could use the following rule expression: `WAF Attack Score Class equals Attack`.
 
-## 3. Create custom rule based on bot score
+## 3. Configure bot protection
 
 :::note
-Bot score is only available to Enterprise customers with [Bot Management](/bots/get-started/bot-management/). Customers on Pro and Business plans may enable [Super Bot Fight mode](/bots/get-started/super-bot-fight-mode/) instead.
+Bot score is only available to Enterprise customers with [Bot Management](/bots/get-started/bot-management/). Customers on Pro and Business plans should turn on [Super Bot Fight Mode](/bots/get-started/super-bot-fight-mode/) instead, which provides built-in bot protection without creating custom rules.
 :::
 
-Customers with access to [Bot Management](/bots/get-started/bot-management/) can block automated traffic (for example, from [bots scraping online content](https://www.cloudflare.com/learning/bots/what-is-content-scraping/)) using a custom rule with bot score, preventing this traffic from hitting your application.
+Enterprise customers with Bot Management should first configure bot protection using **Security Settings**, which provide baseline protection without creating custom rules:
 
-[Create a custom rule](/waf/custom-rules/create-dashboard/) using the <GlossaryTooltip term="bot score">Bot Score</GlossaryTooltip> and <GlossaryTooltip term="verified bot">Verified Bot</GlossaryTooltip> fields:
+<Steps>
+
+1. In the Cloudflare dashboard, go to the **Security Settings** page and filter by **Bot traffic**.
+
+   <DashButton url="/?to=/:account/:zone/security/settings" />
+
+2. Configure the **Definitely automated**, **Likely automated**, and **Verified bots** settings according to your needs.
+3. Turn on **Block AI bots** if you want to block AI crawlers.
+
+</Steps>
+
+These built-in settings auto-update with new bot signatures and do not count toward your custom rule limits. For more details, refer to [Bot Management](/bots/get-started/bot-management/).
+
+### Create a custom rule for additional control
+
+Optionally, if you need more granular control — for example, a different score threshold or rules that combine bot score with other fields — [create a custom rule](/waf/custom-rules/create-dashboard/) using the <GlossaryTooltip term="bot score">Bot Score</GlossaryTooltip> and <GlossaryTooltip term="verified bot">Verified Bot</GlossaryTooltip> fields:
 
 - **When incoming requests match**:
 
@@ -128,11 +143,13 @@ Customers with access to [Bot Management](/bots/get-started/bot-management/) can
 
 - **Choose action**: Managed Challenge
 
-For a more comprehensive example of a baseline protection against malicious bots, refer to [Challenge bad bots](/waf/custom-rules/use-cases/challenge-bad-bots/#general-protection).
+This rule uses a threshold of 20 (instead of the default threshold of 30 used by the settings), providing stricter protection for traffic in the 20-29 score range.
+
+For a more comprehensive example of baseline protection against malicious bots, refer to [Challenge bad bots](/waf/custom-rules/use-cases/challenge-bad-bots/#general-protection).
 
 For more information about the bot-related fields you can use in expressions, refer to [Bot Management variables](/bots/reference/bot-management-variables/).
 
-Once you have deployed the Cloudflare Managed Ruleset and rules based on attack score and bot score you will have achieved substantial protection, limiting the chance of false positives.
+Once you have deployed the Cloudflare Managed Ruleset and rules based on attack score and bot score, you will have achieved substantial protection, limiting the chance of false positives.
 
 ## 4. Optional - Deploy the Cloudflare OWASP Core Ruleset