What it connects
A website crawl turns pages you already publish into knowledge Neutron can answer from. It works well for help centers, product documentation, and internal sites that change often.
You choose the starting URL and how far the crawl goes. A page limit caps the number of pages, and a depth limit controls how many links away from the start page the crawl can go. Include and exclude patterns keep the crawl focused, for example on /docs/ while skipping /blog/.
Before committing to a full crawl, run a test crawl. It shows the pages the crawl reaches so you can adjust the settings. Then pick a schedule: manual, daily, weekly, or monthly. Each scheduled crawl picks up new and changed pages, so answers stay close to what your site says today.
Access and permissions
If the pages sit behind a sign-in, add credentials for the crawl. askneutron supports basic auth, bearer tokens, custom request headers, and cookies. Credentials are stored for the crawl and are not shown again after you save them.
Like uploaded documents, a crawled site has an access policy. You decide which groups or people can get answers from it, so an internal wiki can stay internal while a public help center powers a customer-facing widget.
Good to know
The crawl reads the pages your settings allow. Pages that need a browser login flow, such as single sign-on, may need cookies or headers from a signed-in session.