Website

Website brings your company's published knowledge into Pensieve: product pages, documentation, policies and other public material that should inform the map.

What you choose

Add one or more website addresses. Each address covers that page and every page beneath its path, so example.com/docs includes the documentation without pulling in the rest of the site.

When you add an address, Pensieve discovers the pages it covers and shows them as a tree of paths. Tick the sections you want — ticking a section takes everything beneath it, including pages published there later — and leave the rest. If an address turns out to be a single page, there's nothing to choose and no tree appears. Each ticked section is saved as its own address, up to 200 pages each, and the dialog shows an ingest cost estimate for the whole selection where pricing applies.

Discovery uses the site's sitemaps, index and crawl history together, so sites without a sitemap work too. Pensieve re-checks the selected roots monthly and removes pages that are no longer published. Saving your selection syncs straight away, so you never wait for the schedule to see a site you have just connected.

Removing an address, or part of one

Because you added each address yourself, you can disconnect one on its own — and you're asked whether to keep the pages it brought in or delete them, exactly as you would be for the whole connector. Keep, and they stay in your map and can be deleted later from Data → Sources.

Narrowing an address instead — unticking sections beneath it and saving — deletes the pages you drop, with a warning naming them first. Widening never deletes anything.

Authentication

There is no account to authorise and no credential to store — no consent screen, no OAuth popup, nothing to reconnect later. An admin or owner adds the public address directly from the Connectors page, or while creating the context: because there's no credential, a website is the one source you can put in place before anything else exists. An address added during context creation takes the whole site and is connected as soon as the context's trial or plan starts, so the context has a picture of your company from its first day of work; narrow it any time from the Connectors page, bearing in mind that narrowing deletes the pages you drop.

What lands in your map

The readable body of each public page as clean text, with navigation, headers and footers removed. Links arrive as full addresses and citations point back to the original page.

What it can't see

• Pages outside the website addresses you add, or beyond the page count you chose.
• Pages that require a login or are not publicly reachable over HTTPS.