> For the complete documentation index, see [llms.txt](https://help.botpenguin.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://help.botpenguin.com/ai-agents/creating-ai-agents-in-botpenguin/training-your-ai-agent/website.md).

# Website

### What is this?

The **Website** tab under *Agent Knowledge Base* allows you to import public website content and train your AI Agent on it. You can structure the training by categories, set import preferences, and manage sync/visibility settings to keep your agent’s knowledge current and accurate.

***

### Why is it beneficial?

* **Multi-source training**: Ingest content directly from your public website(s)
* **Knowledge grouping**: Organize URLs into categories for better control
* **Real-time updates**: Retrain or sync data anytime to reflect site changes

***

### How to configure it

1. Go to [**BotPenguin**](https://app.botpenguin.com/) and log in to your account.
2. Click on **Agents** from the sidebar and select the agent you want to configure.
3. Navigate to the **Agent Knowledge Base** section and open the **Website** tab (it loads by default).

<figure><img src="/files/PTL2gzrM3Mz808YcPXmI" alt=""><figcaption></figcaption></figure>

4. Use the **Search Bar** to find specific website categories you've already added.
5. Review the table, which includes:
   * **Category**: Label grouping URLs like "chatbot-pricing" or "General"
   * **Web Pages**: Number of web URLs under each category
   * **Last Trained**: Timestamp of the last training session
   * **Visibility Toggle**: Enable/disable knowledge availability for agent responses
   * **Actions**: Edit category name, sync/retrain content, or delete the entire category
6. To add new website data:
   * Click the **Add Website** button at the top.

<figure><img src="/files/RhUY8LXpKVki1f43gWwQ" alt=""><figcaption><p>Click on Add website to add the website as data source</p></figcaption></figure>

* Enter your **Website URL** in the field provided.
* Choose an **Import Preference**:
  * *Scan Full Page*: Includes all linked subpages
  * *Scan Single Page*: Only the direct URL’s content
  * *Scan JSON Only*: Focuses only on structured JSON content
*

```
<figure><img src="/files/VtqgatE5dqPJ2JwW4lnq" alt=""><figcaption></figcaption></figure>
```

* Optionally, click **Show Advanced Options** to toggle **Train on Images as well**. This allows your agent to also learn from images on the page, though it increases time and cost.

4. Once set, click **Save** to trigger training.

***

### Need Help?

If your URLs fail to import, content doesn’t train correctly, or the agent doesn't use the expected website knowledge, reach out to [**support@botpenguin.com**](mailto:support@botpenguin.com). Our team will assist you within 48 business hours.