🕸️ How Crawling Works
This page answers the most common question about the Knowledge Base: "why didn't it index that page?"
Choosing what to crawl
Select New, then Add a source, then Website, and paste the address into Website URL. Fren One previews the site before anything is indexed.
Pages to crawl. The preview shows the site as a tree of sections with their page counts: /docs, /blog, /pricing. Open a section to see what's inside it, and tick or untick whole sections or single paths. Select all and Select none set everything at once.
Sections that look machine-generated arrive unticked and flagged Auto-generated. These are paginated archives, per-item pages, and numeric-ID URLs: a converter page per currency pair, a page per model, page 47 of a blog index. They add bulk without adding knowledge, and they measurably degrade answers, which is why they are opt-in rather than opt-out.
Other pages, always crawled collects pages that fall into no named section. To leave some of them out, add a pattern under Custom rules.
Pasting a URL with a path (for example https://example.com/docs) scopes the crawl to that section.
Custom rules holds the raw controls: include and exclude path patterns, the page and link-depth caps, and Only crawl pages on the same domain. What you tick in the tree is added on top of whatever you type there. Leave the caps on Auto unless you have a reason not to: the size is derived from what you ticked.
Before the crawl starts you see how many pages are selected. A crawl is charged per page actually indexed; see Credits & Limits.
Re-crawling a website from its row menu on the Sources tab offers the same choices as a list of sections under What to crawl, with the raw controls under Advanced filters.
How pages are found
Two discovery paths run before any crawling:
sitemap.xml, if the site publishes one.- RSS or Atom feeds: the feeds the page links to, then feeds under the section you are crawling (such as
/blog/feedor/blog/rss), then common site-wide paths such as/feedand/rss.xml.
Discovered pages are ordered shallow-first, with paginated URLs last, so that if a limit is reached it is the least useful pages that get dropped. Anything not found this way is reached by following links from the pages that were.
What is skipped, and why
A crawl deliberately does not index everything it encounters.
| Skipped | Reason |
|---|---|
| Login, signup, logout, search, cart and checkout paths | Never contain knowledge, and often loop forever. |
| Links to social and video platforms | Out of scope for a site crawl. Add X and YouTube as their own source types. |
| Links to files, including linked PDFs | Crawling reads pages. Add a PDF with Upload files. |
| Pages in another language | A crawl indexes one language. Translated variants are recognised and left out. |
| Pages that declare a different canonical URL | The canonical page is queued instead, so the same content is not indexed twice. |
| URLs already indexed by another source | Prevents paying twice and returning the same passage twice. |
| Very short pages | Below roughly 120 characters there is nothing to answer from. |
| Other domains | A crawl stays on the site you entered, subdomains included. |
Two more adjustments happen quietly:
- Tracking parameters are stripped from URLs (
utm_*,gclid,fbclid, and similar), so the same page arriving from three campaigns is indexed once. Meaningful parameters such as?page=2are kept. - Very long pages are truncated. Past roughly 80,000 characters the tail is dropped.
The preview reports how many pages were seen, not how many will be indexed. Language variants, canonical duplicates, already-indexed URLs and unticked sections all come off that number. A site reporting "~5,000 pages seen" routinely indexes a few hundred, and that is the intended outcome.
Sync now and re-crawl
They are not the same operation. Both are in the source's row menu on the Sources tab.
- Sync now fetches the site again with its saved settings. When the source keeps in sync looking for New content, it only adds pages that aren't indexed yet and keeps what you have.
- Re-crawl lets you change the sections and filters first, then replaces the source's content using the new settings.
When a crawl fails
The site blocks automated readers. Some sites sit behind bot protection. If the page is fine in your browser but returns nothing here, this is the usual cause. Upload the content as a file, or write it as an article, instead.
The page needs a login. Anything behind authentication cannot be crawled.
Nothing was extracted. The crawl finished but produced no readable text. Sites that render entirely in the browser sometimes expose none.