---
name: browser-automation
title: "Web Research & Scraping"
description: "Search the web and scrape pages into KB inboxes for automated source collection"
category: core
status: active
triggers:
  - "scrape"
  - "research and collect"
  - "find articles about"
  - "search and scrape"
  - "collect sources"
  - "web scrape"
  - "find information online"
  - "browser"
---

## Instructions

Three tools for web-powered knowledge collection:

### 1. Research & Scrape (`research_scrape`) — Primary Tool

Use when the user wants to **build knowledge on a topic**. This is the main tool — it autonomously searches, filters, and scrapes.

**When to use:**
- "Find articles about quantum computing for my physics KB"
- "Research competitor pricing strategies"
- "Collect sources about climate change mitigation"
- Any time the user wants to add web knowledge to a KB without specifying exact URLs

**How:** Call `research_scrape` with the topic and KB slug. It searches DuckDuckGo, filters out social media/PDFs/videos, scrapes the top results via agent-browser, and saves each as a `.md` file in the KB inbox.

### 2. Single URL Scrape (`scrape_url`) — Targeted Collection

Use when the user has a **specific URL** they want to add.

**When to use:**
- "Scrape this article: https://..."
- "Add this page to my KB"
- Research gaps where you know the exact source URL

### 3. Web Search (`web_search`) — Discovery Only

Use when the user wants to **find information** without scraping. Returns titles, URLs, and snippets.

**When to use:**
- "Search for recent papers on transformer efficiency"
- When you need to find URLs before deciding what to scrape
- Answering questions that need current web info

### Workflow

1. **Topic research** → `research_scrape` (searches + scrapes + saves, all-in-one)
2. **After scraping** → suggest `run_wiki_job` to process new content through the pipeline
3. **Targeted additions** → `scrape_url` for specific URLs
4. **Just searching** → `web_search` for discovery without saving

## Rules

- **research_scrape is the default** — when user says "find articles about X", use it, not web_search + manual scrape
- **Always identify the target KB** — match topic to the most relevant KB. Ask if unclear
- **Suggest KB creation** — if no KB fits the topic, suggest creating one first
- **Report results** — tell the user what was found and scraped, suggest running the wiki pipeline
- **Respect rate limits** — research_scrape already limits to 5 pages by default
- **Source attribution** — all scraped files include `source_url` and `search_query` in frontmatter
- **No login scraping** — skip pages behind authentication
- **Content quality** — if most pages return <200 chars, suggest a more specific search query
