AI Search Docs Refactor (#28290)

aninibread · web-flow · commit 156405da71ed · 2026-02-23T12:33:33.000-05:00
* random space

* small changes

* limit change
diff --git a/src/content/docs/ai-search/concepts/how-ai-search-works.mdx b/src/content/docs/ai-search/concepts/how-ai-search-works.mdx
@@ -5,7 +5,7 @@ sidebar:
   order: 2
 ---
 
-AI Search (formerly AutoRAG) is Cloudflare’s managed search service. You can connect your data such as websites or unstructured content, and it automatically creates a continuously updating index that you can query with natural language in your applications or AI agents. 
+AI Search is Cloudflare’s managed search service. You can connect your data such as websites or unstructured content, and it automatically creates a continuously updating index that you can query with natural language in your applications or AI agents. 
 
 AI Search consists of two core processes:
 
diff --git a/src/content/docs/ai-search/configuration/chunking.mdx b/src/content/docs/ai-search/configuration/chunking.mdx
@@ -20,9 +20,7 @@ This way, chunks are easy to embed and retrieve, without cutting off thoughts mi
 
 AI Search exposes two parameters to help you control chunking behavior:
 
-- **Chunk size**: The number of tokens per chunk.
-  - Minimum: `64`
-  - Maximum: `512`
+- **Chunk size**: The number of tokens per chunk. The option range may vary depending on the model.
 - **Chunk overlap**: The percentage of overlapping tokens between adjacent chunks.
   - Minimum: `0%`
   - Maximum: `30%`
@@ -33,16 +31,6 @@ These settings apply during the indexing step, before your data is embedded and
 
 Chunking affects both how your content is retrieved and how much context is passed into the generation model. Try out this external [chunk visualizer tool](https://huggingface.co/spaces/m-ric/chunk_visualizer) to help understand how different chunk settings could look.
 
-For chunk size, consider how:
-
-- **Smaller chunks** create more precise vector matches, but may split relevant ideas across multiple chunks.
-- **Larger chunks** retain more context, but may dilute relevance and reduce retrieval precision.
-
-For chunk overlap, consider how:
-
-- **More overlap** helps preserve continuity across boundaries, especially in flowing or narrative content.
-- **Less overlap** reduces indexing time and cost, but can miss context if key terms are split between chunks.
-
 ### Additional considerations:
 
 - **Vector index size:** Smaller chunk sizes produce more chunks and more total vectors. Refer to the [Vectorize limits](/vectorize/platform/limits/) to ensure your configuration stays within the maximum allowed vectors per index.
diff --git a/src/content/docs/ai-search/configuration/data-source/index.mdx b/src/content/docs/ai-search/configuration/data-source/index.mdx
@@ -5,7 +5,7 @@ sidebar:
   order: 2
 ---
 
-AI Search can directly ingest data from the following sources:
+AI Search can directly ingest data from the following sources: 
 
 | Data Source   | Description |
 |---------------|-------------|
diff --git a/src/content/docs/ai-search/configuration/data-source/r2.mdx b/src/content/docs/ai-search/configuration/data-source/r2.mdx
@@ -24,10 +24,7 @@ Refer to [Path filtering](/ai-search/configuration/path-filtering/) for pattern
 
 ## File limits
 
-AI Search has different file size limits depending on the file type:
-
-- **Plain text files:** Up to **4 MB**
-- **Rich format files:** Up to **4 MB**
+AI Search has a file size limit of **up to **4 MB**.
 
 Files that exceed these limits will not be indexed and will show up in the error logs.
 
diff --git a/src/content/docs/ai-search/configuration/data-source/website.mdx b/src/content/docs/ai-search/configuration/data-source/website.mdx
@@ -146,9 +146,9 @@ If you have Security rules configured to block bot activity, you can add a rule
 
 You can configure parsing options during onboarding or in your instance settings under **Parser options**.
 
-### Sitemap
+### Specific sitemap
 
-By default, AI Search crawls all sitemaps listed in your `robots.txt` in the order they appear (top to bottom). If you do not want the crawler to index everything, you can specify a single sitemap URL to limit which pages are crawled.
+By default, AI Search crawls all sitemaps listed in your `robots.txt` in the order they appear (top to bottom). If you do not want the crawler to index everything, you can specify a single sitemap URL to limit which pages are crawled. You can add up to 5 specific sitemaps.
 
 ### Rendering mode
 
@@ -157,7 +157,7 @@ You can choose how pages are parsed during crawling:
 - **Static sites**: Downloads the raw HTML for each page.
 - **Rendered sites**: Loads pages with a headless browser and downloads the fully rendered version, including dynamic JavaScript content. Note that the [Browser Rendering](/browser-rendering/pricing/) limits and billing apply.
 
-## Access protected content
+## Extra headers for access protected content
 
 If your website has pages behind authentication or are only visible to logged-in users, you can configure custom HTTP headers to allow the AI Search crawler to access this protected content. You can add up to five custom HTTP headers to the requests AI Search sends when crawling your site.
 
diff --git a/src/content/docs/ai-search/configuration/models/index.mdx b/src/content/docs/ai-search/configuration/models/index.mdx
@@ -22,10 +22,7 @@ All AI Search instances support models from [Workers AI](/workers-ai). You can u
 
 To use AI Search with other model providers:
 
-1. Add provider keys to AI Gateway
-- Go to **AI > AI Gateway** in the dashboard.
-- Select or create an AI gateway.
-- In **Provider Keys**, choose your provider, click **Add**, and enter the key.
+1. Add provider keys to [AI Gateway](/ai-gateway/configuration/bring-your-own-keys/)
 2. Connect the gateway to AI Search
 - When creating a new AI Search, select the AI Gateway with your provider keys.
 - For an existing AI Search, go to **Settings** and switch to a gateway that has your keys under **Resources**.
diff --git a/src/content/docs/ai-search/configuration/path-filtering.mdx b/src/content/docs/ai-search/configuration/path-filtering.mdx
@@ -13,7 +13,7 @@ Path filtering works with both [website](/ai-search/configuration/data-source/we
 
 You can configure path filters when creating or editing an AI Search instance. In the dashboard, open **Path Filters** and add your include or exclude rules. You can also update path filters at any time from the **Settings** page of your instance.
 
-When using the API, specify `include_items` and `exclude_items` in the `source_params` of your configuration:
+When using the REST API, specify `include_items` and `exclude_items` in the `source_params` of your configuration:
 
 | Parameter       | Type       | Limit               | Description                                              |
 | --------------- | ---------- | ------------------- | -------------------------------------------------------- |
diff --git a/src/content/docs/ai-search/configuration/reranking.mdx b/src/content/docs/ai-search/configuration/reranking.mdx
@@ -72,3 +72,6 @@ To update reranking for an existing instance:
 4. Under **Reranking**, toggle reranking on.
 5. Select the reranking model.
 
+### Considerations
+
+Adding reranking will include an additional step to the query request, as a result, there may be an increase in the latency of the request.
diff --git a/src/content/docs/ai-search/configuration/service-api-token.mdx b/src/content/docs/ai-search/configuration/service-api-token.mdx
@@ -15,13 +15,16 @@ Service API tokens are required during the AI Search beta. This requirement may
 
 When you create an AI Search instance, it needs to interact with other Cloudflare services on your behalf, such as [R2](/r2/), [Vectorize](/vectorize/), and [Workers AI](/workers-ai/). The service API token authorizes AI Search to perform these operations. Without it, AI Search cannot index your data or respond to queries.
 
+This token requires the AI Search Index Engine permission (`9e9b428a0bcd46fd80e580b46a69963c`) which grants access to run AI Search Index Engine.
+
+
 ## Service API token vs. AI Search API token
 
 AI Search uses two types of API tokens for different purposes:
 
 | Token type          | Purpose                                                             | Who uses it          | When to create                                   |
 | ------------------- | ------------------------------------------------------------------- | -------------------- | ------------------------------------------------ |
-| Service API token   | Grants AI Search permission to access R2, Vectorize, and Workers AI | AI Search (internal) | Once per account, during first instance creation |
+| Service API token   | Grants AI Search permission to access R2, Vectorize, Browser Rendering and Workers AI | AI Search (internal) | Once per account, during first instance creation |
 | AI Search API token | Authenticates your requests to query or manage AI Search instances  | You (external)       | When calling the AI Search REST API              |
 
 The **service API token** is used internally by AI Search to perform background operations like indexing your content and generating responses. You create it once and AI Search uses it automatically.
diff --git a/src/content/docs/ai-search/configuration/system-prompt.mdx b/src/content/docs/ai-search/configuration/system-prompt.mdx
@@ -58,39 +58,6 @@ The system prompt for your AI Search can be set after it has been created:
 3. Go to the **Settings** tab.
 4. Go to **Query rewrite** or **Generation**, and edit the **System prompt**.
 
-## Query rewriting system prompt
-
-If query rewriting is enabled, you can provide a custom system prompt to control how the model rewrites user queries. In this step, the model receives:
-
-- The query rewrite system prompt
-- The original user query
-
-The model outputs a rewritten query optimized for semantic retrieval.
-
-### Example
-
-```text
-You are a search query optimizer for vector database searches. Your task is to reformulate user queries into more effective search terms.
-
-Given a user's search query, you must:
-1. Identify the core concepts and intent
-2. Add relevant synonyms and related terms
-3. Remove irrelevant filler words
-4. Structure the query to emphasize key terms
-5. Include technical or domain-specific terminology if applicable
-
-Provide only the optimized search query without any explanations, greetings, or additional commentary.
-
-Example input: "how to fix a bike tire that's gone flat"
-Example output: "bicycle tire repair puncture fix patch inflate maintenance flat tire inner tube replacement"
-
-Constraints:
-- Output only the enhanced search terms
-- Keep focus on searchable concepts
-- Include both specific and general related terms
-- Maintain all important meaning from original query
-```
-
 ## Generation system prompt
 
 If you are using the AI Search API endpoint, you can use the system prompt to influence how the LLM responds to the final user query using the retrieved results. At this step, the model receives:
@@ -128,3 +95,36 @@ Important:
 - If documents contradict each other, note this and explain your reasoning for the chosen answer
 - Do not repeat the instructions
 ```
+
+## Query rewriting system prompt
+
+If query rewriting is enabled, you can provide a custom system prompt to control how the model rewrites user queries. In this step, the model receives:
+
+- The query rewrite system prompt
+- The original user query
+
+The model outputs a rewritten query optimized for semantic retrieval.
+
+### Example
+
+```text
+You are a search query optimizer for vector database searches. Your task is to reformulate user queries into more effective search terms.
+
+Given a user's search query, you must:
+1. Identify the core concepts and intent
+2. Add relevant synonyms and related terms
+3. Remove irrelevant filler words
+4. Structure the query to emphasize key terms
+5. Include technical or domain-specific terminology if applicable
+
+Provide only the optimized search query without any explanations, greetings, or additional commentary.
+
+Example input: "how to fix a bike tire that's gone flat"
+Example output: "bicycle tire repair puncture fix patch inflate maintenance flat tire inner tube replacement"
+
+Constraints:
+- Output only the enhanced search terms
+- Keep focus on searchable concepts
+- Include both specific and general related terms
+- Maintain all important meaning from original query
+```
diff --git a/src/content/docs/ai-search/get-started/api.mdx b/src/content/docs/ai-search/get-started/api.mdx
@@ -56,20 +56,14 @@ Use the [Create token API](/api/resources/user/subresources/tokens/methods/creat
              "com.cloudflare.api.account.<ACCOUNT_ID>": "*"
            },
            "permission_groups": [
-             { "id": "9e9b428a0bcd46fd80e580b46a69963c" },
-             { "id": "bf7481a1826f439697cb59a20b22293e" }
+             { "id": "9e9b428a0bcd46fd80e580b46a69963c" }
            ]
          }
        ]
      }'
    ```
 
-   This creates a token with the following permissions:
-
-   | Permission ID                      | Name                     | Description                                  |
-   | ---------------------------------- | ------------------------ | -------------------------------------------- |
-   | `9e9b428a0bcd46fd80e580b46a69963c` | AI Search Index Engine   | Grants access to run AI Search Index Engine  |
-   | `bf7481a1826f439697cb59a20b22293e` | Workers R2 Storage Write | Grants write access to Cloudflare R2 Storage |
+   This creates a token with the AI Search Index Engine permission (`9e9b428a0bcd46fd80e580b46a69963c`) which grants access to run AI Search Index Engine.
 
 2. Save the `id` (`<CF_API_ID>`) and `value` (`<CF_API_KEY>`) from the response. You will need these values in the next step.
 
@@ -94,10 +88,6 @@ Use the [Create token API](/api/resources/user/subresources/tokens/methods/creat
    					{
    						"id": "9e9b428a0bcd46fd80e580b46a69963c",
    						"name": "AI Search Index Engine"
-   					},
-   					{
-   						"id": "bf7481a1826f439697cb59a20b22293e",
-   						"name": "Workers R2 Storage Write"
    					}
    				]
    			}
diff --git a/src/content/docs/ai-search/get-started/dashboard.mdx b/src/content/docs/ai-search/get-started/dashboard.mdx
@@ -24,13 +24,12 @@ AI Search integrates with R2 for storing your data. You must have an active R2 s
 <Steps>
 1. In the Cloudflare Dashboard, go to **Compute & AI** > **AI Search**.
 2. Select **Create**.
-3. In **Create a RAG**, select **Get Started**.
-4. Choose how you want to connect your [data source](/ai-search/configuration/data-source/).
-5. Configure [chunking](/ai-search/configuration/chunking/) and [embedding](/ai-search/configuration/models/) settings for how your content is processed.
-6. Configure [retrieval settings](/ai-search/configuration/retrieval-configuration/) for how search results are returned.
-7. Name your AI Search instance.
-8. Create a [service API token](/ai-search/configuration/service-api-token/).
-9. Select **Create**.
+3. Choose how you want to connect your [data source](/ai-search/configuration/data-source/).
+4. Configure [chunking](/ai-search/configuration/chunking/) and [embedding](/ai-search/configuration/models/) settings for how your content is processed.
+5. Configure [retrieval settings](/ai-search/configuration/retrieval-configuration/) for how search results are returned.
+6. Name your AI Search instance.
+7. Create a [service API token](/ai-search/configuration/service-api-token/).
+8. Select **Create**.
 </Steps>
 
 ## Try it out
diff --git a/src/content/docs/ai-search/how-to/bring-your-own-generation-model.mdx b/src/content/docs/ai-search/how-to/bring-your-own-generation-model.mdx
@@ -21,7 +21,7 @@ import {
 
 When using `AI Search`, AI Search leverages a Workers AI model to generate the response. If you want to use a model outside of Workers AI, you can use AI Search for `search` while leveraging a model outside of Workers AI to generate responses.
 
-Here is an example of how you can use an OpenAI model to generate your responses. This example uses [Workers Binding](/ai-search/usage/workers-binding/), but can be easily adapted to use the [REST API](/ai-search/usage/rest-api/) instead.
+Here is an example of how you can use an OpenAI model to generate your responses. This example uses [Workers Binding](/ai-search/usage/workers-binding/).
 
 :::note
 AI Search now supports [bringing your own models natively](/ai-search/configuration/models/). You can attach provider keys through AI Gateway and select third-party models directly in your AI Search settings. The example below still works, but the recommended way is to configure your external model through AI Gateway.  
diff --git a/src/content/docs/ai-search/how-to/brower-rendering-autorag-tutorial.mdx b/src/content/docs/ai-search/how-to/brower-rendering-autorag-tutorial.mdx
diff --git a/src/content/docs/ai-search/how-to/nlweb.mdx b/src/content/docs/ai-search/how-to/nlweb.mdx
diff --git a/src/content/docs/ai-search/index.mdx b/src/content/docs/ai-search/index.mdx
diff --git a/src/content/docs/ai-search/platform/limits-pricing.mdx b/src/content/docs/ai-search/platform/limits-pricing.mdx
diff --git a/src/content/docs/ai-search/usage/rest-api.mdx b/src/content/docs/ai-search/usage/rest-api.mdx
