Training from a Website or Sitemap

The fastest way to give your bot real knowledge is pointing it at content you've already written. This article covers training from web pages: single URLs, whole sitemaps, and bulk lists of URLs. By the end, your site's help content will be searchable knowledge your bot answers from.

You'll do all of this from your bot's Sources tab (or the Training Data step of the new-bot wizard — it's the same input in both places).

The Training URL input

At the top of the Sources tab is a single smart input labeled Training URL, with the hint: "Enter a WordPress site, sitemap, or any webpage — we'll detect the type automatically."

Type or paste a URL and click Add (or press Enter). You don't need to include https:// - HelpJet adds it for you, and strips query strings and fragments so the same page isn't added twice under different URLs.

What auto-detection checks

When you add a URL, HelpJet probes it in this order:

1. WordPress — if the site answers on the WordPress REST API, it's added as a WordPress source with post-type selection. See Training from WordPress Content.
2. Sitemap — if the URL itself is a sitemap, or the site serves one at /sitemap.xml, it's added as a sitemap source and the pages it lists are pulled in.
3. Single webpage — anything else is added as a one-page URL source.

Detection takes a few seconds. If it can't reach the site in time, the URL is added as a single page and you'll see a notice ("Added as single URL") — the page still trains normally.

Training from a sitemap

A sitemap is the best way to bulk-train on a non-WordPress site: one URL in, your whole site out.

1. Enter your sitemap URL (or your homepage — HelpJet checks /sitemap.xml for you) and click Add.
2. HelpJet parses the sitemap and lists the pages it found — up to 1,000 URLs per sitemap. Expand the sitemap entry to review them, and use the search box to find and untick pages that shouldn't train the bot.
3. Click Start training. The button shows the item count, e.g. "Start training (42 items)".

Each page becomes its own child source under the sitemap parent, with its own status. Failed pages don't block the rest — you can retry them individually later with Retry failed.

Adding multiple URLs at once

To train on a specific set of pages rather than a whole sitemap, upload a file listing them:

1. In the add area, open the Upload menu and choose CSV of URLs.
2. Pick a .csv, .txt, or .tsv file. List one URL per line (or per cell in a CSV).

HelpJet extracts every valid URL from the file, skips duplicates you've already added, and routes any sitemap links through sitemap ingestion automatically. A toast confirms how many URLs were added and how many duplicates were skipped.

A few rules:

• The file must be under 1MB.
• URLs without a protocol (example.com/pricing) are accepted - https:// is added.
• Invalid lines are skipped silently, so a messy export is fine.

What happens during processing

After you click Start training, each URL goes through the same pipeline in the background:

1. Fetch — HelpJet loads the page, rendering it the way a browser would.
2. Extract — the main content is pulled out; navigation, headers, footers, and other boilerplate are stripped.
3. Chunk and embed — the text is split into small chunks and stored as searchable embeddings in your bot's knowledge base.

Statuses update live in the source list: Processing → Processed (or Failed, after up to 3 automatic retries). Pages are fetched with deliberate spacing rather than all at once, so a large sitemap takes minutes, not seconds — you can leave the page and come back.

Once a page shows Processed, the bot can answer from it immediately.

Keeping web sources current

• Processed sources are automatically re-fetched when they're more than 30 days old.
• Re-adding a URL that's already a source refreshes it in place instead of duplicating it — useful right after you update a page.
• Refreshing a sitemap parent re-discovers its pages: new pages are added, changed pages re-fetched, and pages that have disappeared are flagged (never auto-deleted).

See Training Sources Overview for the full refresh and stale-source behavior.

Choosing what to train on

More pages isn't automatically better. The bot answers from the most relevant chunks it finds, so noisy sources can crowd out good ones.

Prioritize:

• Help center and documentation pages
• FAQs
• Pricing, policies, shipping/returns — anything customers ask about verbatim
• Product pages with concrete details

Skip or deselect:

• Blog posts that are opinion or news rather than reference
• Legal boilerplate, careers pages, press releases
• Landing pages that repeat the same marketing copy — near-duplicate content adds noise without adding knowledge
• Anything outdated you wouldn't want quoted back to a customer

After deploying, the highest-leverage habit is checking the Activity tab for questions the bot couldn't answer, then adding exactly those pages — see Reviewing Conversations.

Common questions

Can the bot train on pages behind a login?

The standard URL flow fetches pages the way an anonymous visitor sees them, so a page that redirects to a login form won't train. There are two ways around that:

• Private WordPress content — authenticate with an Application Password or JWT token; see Training from WordPress Content.
• Sites behind HTTP authentication — pages protected by HTTP Basic auth or a bearer token can be trained; the credentials are stored encrypted, like WordPress credentials (see Security & Privacy). Contact support to set this up for your sources.

A page failed with "Couldn't load this page" — what now?

Check that the URL is publicly reachable and doesn't redirect to a login or an error page. Fix or remove the URL, then use Refresh or Retry failed on the row.

Does JavaScript-rendered content work?

Yes — pages are rendered before extraction, so content that only appears after JavaScript runs is included.