---
name: browser-automation
title: "Web Research & Scraping"
description: "Create knowledge bases, search the web, scrape pages, and run the full research pipeline autonomously"
category: core
status: active
triggers:
  - "scrape"
  - "research and collect"
  - "find articles about"
  - "search and scrape"
  - "collect sources"
  - "web scrape"
  - "find information online"
  - "browser"
  - "create a kb"
  - "build a knowledge base"
  - "research topic"
tools_granted:
  - create_kb
  - web_search
  - research_scrape
  - scrape_url
  - run_wiki_job

# Scraping strategy — consumed by integrations/browser.py at startup.
# Edit this block to change how the agent fetches web pages without
# touching Python. Per CLAUDE.md Rule #12, strategy lives in skill YAML,
# not in code. Change one value and restart the server to apply.
#
# backend_preference: ordered list of backends to try. Remove
#   "agent-browser" entirely to force pure-httpx scraping (fastest and
#   most reliable when Device Guard / EDR blocks the Playwright CLI).
# agent_browser_timeout: seconds per subprocess call (open + get text).
#   Previously 30s which meant 120s worst-case per URL on blocked setups.
# httpx_timeout: seconds per httpx request.
# concurrency: parallel URL fetches for research_scrape batches.
# min_content_length: reject extracted text shorter than this (filters
#   cookie walls and redirect stubs).
# retry_with_alternate_user_agent: retry once with a different Chrome UA
#   when the first attempt hits 403/429/anti-bot walls.
# default_pages: how many pages research_scrape fetches when the user does
#   NOT specify a number. Used as the silent-default in the tool executor.
# max_pages: hard upper clamp. If the user asks for more, the executor
#   clamps to this value and logs the clamp. Raise this if you trust your
#   DDGS rate limit and disk; lower it if you want tighter safety.
scraping:
  backend_preference: ["httpx", "agent-browser"]
  agent_browser_timeout: 8
  httpx_timeout: 15
  concurrency: 5
  min_content_length: 200
  retry_with_alternate_user_agent: true
  default_pages: 5
  max_pages: 30

# Per-page quality gate. Pages that pass scrape but produce too little real
# text (cookie walls, JS-only pages, redirect stubs that html_to_text squeezed
# down to a single sentence) are NOT saved as KB sources — they would just
# generate fabricated wiki content downstream. Tune the floor here, NOT in
# Python (Rule #12).
quality:
  # Minimum word count after html_to_text extraction. 150 = roughly one
  # paragraph of meaningful prose. 200+ = a substantive article. Below
  # 150 the LLM has nothing useful to summarise and tends to hallucinate.
  min_words_per_source: 150

# Per-page topic-relevance gate. Pages that scraped fine but turn out to be
# OFF-TOPIC (sidebar widgets, related-articles pages, generic site landing
# pages that pretended to be about the query) are dropped before they pollute
# the KB. The check is keyword-overlap based — fast, deterministic, and works
# offline. The semantic-embedding upgrade is gated behind a separate flag.
#
# Algorithm: tokenise the topic (>=3-char words, lowercased), tokenise the
# scraped text the same way, count how many UNIQUE topic words appear at
# least once. The page is RELEVANT iff
#     (matched_topic_words / total_topic_words) >= min_keyword_overlap
# AND matched_topic_words >= min_unique_matches.
relevance:
  # Fraction of the topic's distinct keywords that must appear at least once
  # in the page text. 0.3 = "at least 30% of meaningful topic words show up".
  # Lower = more permissive (more off-topic noise), higher = stricter (more
  # legit pages dropped because they used synonyms).
  min_keyword_overlap: 0.3
  # Minimum number of distinct topic keywords that must hit, regardless of
  # the overlap ratio. Guards single-word topics where 0.3*1 == 0.3 → 0
  # always passes. With min_unique_matches=2, single-word topics auto-fail
  # the gate (which is a good safety — single-word queries are too vague to
  # judge relevance reliably).
  min_unique_matches: 1
  # Words shorter than this are not used for matching ("a", "the", "in"
  # would match every page). 4+ chars filters most stopwords without a
  # per-language stopword list.
  min_word_length: 4

# Per-page image policy. The previous behavior re-fetched every scraped page
# and bulk-downloaded up to 5 of its images into the KB inbox, where they
# became standalone "sources" the vision LLM described and compile turned
# into wiki articles. Result: NASA nebula photos and random DSC* camera
# dumps appearing as Colombia-floods wiki concepts. Default is now OFF —
# images can be referenced INLINE in the text source via [Image: alt text]
# markers, but they are not saved as separate files. Set to true ONLY for
# domains where the visual content IS the knowledge (e.g. art / design KBs).
images:
  download_during_scrape: false
  max_per_page: 0
---

## Instructions

You have five tools for autonomous knowledge research and collection. The key principle: **you do not suggest — you act.** When the user asks to research a topic, you create the KB, scrape sources, and compile the wiki in one flow.

### Full Research Workflow

When the user asks to "research X", "build a KB about X", "find out about X", or similar:

1. **Create KB** → `create_kb` — set up the knowledge base with focus questions, description, and source types that fit the domain
2. **Research & Scrape** → `research_scrape` — search the web and scrape top results into the KB inbox. Run multiple rounds with different search angles for breadth
3. **Compile Wiki** → `run_wiki_job` — process scraped sources into wiki articles, cross-link, reflect, lint

Do ALL three steps. Do not stop after creating the KB and ask the user what to do next. Do not stop after scraping and forget to compile.

### Tools

#### 1. `create_kb` — Create Knowledge Base
Use FIRST when no suitable KB exists for the topic. Sets up folder structure, SCHEMA.md with focus questions, and source type expectations.

**Parameters:**
- `topic` — name/slug (e.g., "floods-in-colombia", "competitor-analysis")
- `description` — what the KB covers and what questions it should answer
- `focus_questions` — key questions to guide research (3-5 recommended)
- `source_types` — expected source types (e.g., "news articles", "academic papers")
- `notes` — domain-specific guidance for the agent

#### 2. `research_scrape` — Search + Scrape (Primary Research Tool)
Searches DuckDuckGo for the topic, filters results, scrapes top pages, saves to KB inbox. This is the main research tool — it handles search, filter, scrape, and save autonomously.

**When to use:**
- "Find articles about quantum computing for my physics KB"
- "Research competitor pricing strategies"
- Any time the user wants web knowledge without specifying exact URLs

**For thorough research, run multiple rounds** with different search angles:
- Round 1: broad overview query
- Round 2: specific subtopic or angle
- Round 3: recent events / current data

#### 3. `scrape_url` — Single URL Scrape
Use when the user has a specific URL to add, or when `web_search` found a good result to capture.

#### 4. `web_search` — Discovery Only
Returns titles, URLs, and snippets. Does NOT save to KB. Use for quick lookups or to find URLs before targeted scraping.

#### 5. `run_wiki_job` — Process & Compile
Runs the full wiki pipeline on a KB: ingest → compile → reflect → lint. Call this AFTER scraping to turn raw sources into structured wiki articles.

## Rules

- **Act, don't suggest** — when the user says "research X", execute the full workflow: create → scrape → compile. Do not ask permission for each step
- **Create KB first** — if no KB exists for the topic, create one before scraping. Never tell the user to create it manually
- **research_scrape is the default** — when user says "find articles about X", use it, not web_search + manual scrape
- **Multiple search angles** — for thorough research, run 2-3 rounds of research_scrape with different queries targeting different aspects of the topic
- **Always compile** — after scraping, run `run_wiki_job` to process sources into wiki articles
- **Match or create KB** — if an existing KB fits the topic, use it. If not, create a new one. Ask only if genuinely ambiguous
- **Report results** — summarize what was found, how many sources scraped, how many articles compiled
- **Honour user-specified counts** — if the user says "scrape 10 articles", "find 20 sources", "grab 15 pages", pass that number as `max_pages`. Default 5 only applies when the user is silent about count. Cap is 30 — if they ask for more, clamp and tell them you clamped
- **Source attribution** — all scraped files include `source_url` and `search_query` in frontmatter
- **No login scraping** — skip pages behind authentication
- **Content quality** — if most pages return empty, suggest a more specific search query or try different angles
