Metadata-Version: 2.4
Name: bdistill
Version: 0.1.1
Summary: Behavioral X-Ray for AI models — probe behavior, extract domain knowledge, no API key needed
Author: bdistill contributors
License: MIT
Project-URL: Homepage, https://github.com/FrancyJGLisboa/bdistill
Project-URL: Repository, https://github.com/FrancyJGLisboa/bdistill
Project-URL: Issues, https://github.com/FrancyJGLisboa/bdistill/issues
Keywords: ai,llm,behavioral-analysis,mcp,fine-tuning,knowledge-extraction,model-evaluation
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: click>=8.1
Requires-Dist: httpx>=0.27
Requires-Dist: mcp[cli]>=1.0
Requires-Dist: pydantic>=2.5
Requires-Dist: rich>=13.0
Requires-Dist: tqdm>=4.66
Requires-Dist: pyyaml>=6.0
Provides-Extra: train
Requires-Dist: torch>=2.1; extra == "train"
Requires-Dist: transformers>=4.40; extra == "train"
Requires-Dist: peft>=0.10; extra == "train"
Requires-Dist: datasets>=2.18; extra == "train"
Requires-Dist: accelerate>=0.30; extra == "train"
Requires-Dist: bitsandbytes>=0.43; extra == "train"
Provides-Extra: retrieve
Requires-Dist: chromadb>=0.5; extra == "retrieve"
Requires-Dist: sentence-transformers>=2.7; extra == "retrieve"
Provides-Extra: mcp
Requires-Dist: mcp[cli]>=1.0; extra == "mcp"
Provides-Extra: export
Requires-Dist: openpyxl>=3.1; extra == "export"
Provides-Extra: dev
Requires-Dist: ruff>=0.4; extra == "dev"
Requires-Dist: pytest>=8.0; extra == "dev"
Dynamic: license-file

# bdistill — Turn AI into a Structured Knowledge Base

![bdistill overview](docs/images/bdistill-overview.png)

> *"You do not need more knowledge. You need a cleaner workflow."*

You already pay for Claude, GPT-4, or Copilot. These models know your domain — but every answer disappears into a chat log. bdistill extracts that knowledge systematically and gives you a structured, searchable, quality-scored reference you can reuse. It follows the same workflow senior professionals use — [decompose, gather evidence, challenge the thesis, track accuracy](docs/why-this-design.md) — without the anchoring bias, confirmation bias, or bus factor.

**Any topic. No API key. Works with your existing AI subscription.**

```
You: /bdistill-events --custom "quarterly earnings season, FOMC rate decision"

→ For each event, bdistill extracts what happens at T-14d, T-7d, T-day, T+1d, T+7d
→ At each time step: local, regional, national, and global effects
→ Plus causal chains, historical analogs, and conditional rules
→ Output is temporally ordered and spatially layered — not flat Q&A
→ Export as Excel, CSV, Markdown, or fine-tuning JSONL
```

Or extract structured domain knowledge on any topic:

```
You: /bdistill-distill --custom "soybean basis risk, crush margins, CBOT contango"

→ 50 targeted questions, adversarially validated, quality-scored
→ Contradictions auto-detected and resolved
→ Knowledge compounds across sessions into a searchable reference
```

Or make structured predictions — for Polymarket, Kalshi, or any scenario analysis:

```
You: /bdistill-predict "Will urea exceed $600/t by June?" --binary --market_price 0.45 --grounded --deep

→ Decomposes into 6 sub-questions (rule, event, knowledge, causal)
→ Deep distills 36 knowledge probes from LLM training data (thresholds, mechanisms, precedents)
→ Recalls matching entries from your existing knowledge base
→ Web searches for current prices, news, and market data
→ Adversarially challenges its own reasoning
→ Outputs: Prediction: YES / Probability: 72% / Market: 45% / Edge: +27%
→ Shareable HTML card with evidence, causal chain, and track record
→ Resolve later: /bdistill-predict --resolve {id} --actual_outcome yes → Brier score: 0.08
```

Don't know where to start? Just describe what you want:

```
You: /bdistill-start "I trade grain futures"

→ Routes you to the right mode automatically
→ Matches preset domains or maps your domain from scratch
```

### Install (2 commands)

```bash
pipx install bdistill    # or: uv tool install bdistill
bdistill setup           # connects to Claude Code, Cursor, VS Code, Codex
```

Start a new session. Type `/bdistill-start`. Done.

### Browse your knowledge

```bash
bdistill dashboard       # opens local web UI at localhost:8741
```

## Why bdistill

You already use AI for domain research. But you do it by chatting — and every answer vanishes into a thread you'll never find again. Next time the same question comes up, you start from scratch.

bdistill automates what you already do manually: asking the right questions, organizing the answers, and building a reference you can search later. It adds quality scoring, adversarial validation, deduplication, and structured export so the output is immediately usable — not another chat log.

**For any knowledge worker**: commodities analysts, compliance officers, financial researchers, engineers, lawyers, consultants — anyone who needs structured domain knowledge from the AI they already pay for.

## Install

### Step 1: Install the Python package globally

Use `pipx` or `uv tool install` — these install `bdistill` and `bdistill-mcp` as global commands available from any directory:

```bash
# Recommended (puts bdistill + bdistill-mcp on PATH automatically)
pipx install bdistill
# or
uv tool install bdistill
```

If you don't have `pipx` or `uv`:

```bash
# macOS
brew install pipx && pipx install bdistill

# Linux
python3 -m pip install --user pipx && pipx install bdistill
```

> **Do NOT use `pip install bdistill` or `uv pip install bdistill`.** These install into the current virtualenv only — the `bdistill` and `bdistill-mcp` commands won't be on your PATH and won't work from other directories or from your AI tool's MCP config.
>
> If you already ran `pip install` or `uv pip install -e .` inside the repo, that's fine for development, but you still need `pipx install bdistill` or `uv tool install bdistill` for global access.

### Step 2: Verify it works

```bash
bdistill --version         # should print: bdistill, version 0.1.0
which bdistill-mcp         # should print a path like ~/.local/bin/bdistill-mcp
```

If `which bdistill-mcp` prints nothing, the binary isn't on your PATH. Fix:

```bash
# Find where it was installed
python3 -c "import shutil; print(shutil.which('bdistill-mcp') or 'Run: pip show -f bdistill | grep bin')"

# Or use the full path in Step 3 (replace the bare command with the path)
```

### Step 3: Connect to your AI tool

#### Claude Code

```bash
# If bdistill-mcp is on PATH:
claude mcp add bdistill -- bdistill-mcp

# If it's NOT on PATH, use the full path:
claude mcp add bdistill -- $(python3 -c "import shutil; print(shutil.which('bdistill-mcp'))")
```

Start a **new** Claude Code session (MCP servers load at startup). Then type:

```
/bdistill-xray
```

#### VS Code + Copilot

Add to `.vscode/mcp.json` in your project root:

```json
{
  "servers": {
    "bdistill": { "command": "bdistill-mcp" }
  }
}
```

If `bdistill-mcp` isn't on PATH, use the full path:

```json
{
  "servers": {
    "bdistill": { "command": "/full/path/to/bdistill-mcp" }
  }
}
```

Find the path with: `python3 -c "import shutil; print(shutil.which('bdistill-mcp'))"`

Reload VS Code. Switch Copilot Chat to **Agent mode**, then ask: *"X-ray your behavioral patterns"*

#### Cursor

Add to `.cursor/mcp.json` in your project root:

```json
{
  "mcpServers": {
    "bdistill": { "command": "bdistill-mcp" }
  }
}
```

Same PATH note as above — use full path if `which bdistill-mcp` returns nothing.

Restart Cursor. In chat, ask: *"@bdistill x-ray your behavioral patterns"*

#### Codex CLI (OpenAI)

```bash
mkdir -p .codex
curl -fsSL https://raw.githubusercontent.com/FrancyJGLisboa/bdistill/main/.codex/AGENTS.md -o .codex/AGENTS.md
```

Then ask Codex: *"X-ray your behavioral patterns"*

### Troubleshooting

| Symptom | Cause | Fix |
|---------|-------|-----|
| `pip install` blocked | PEP 668 (modern macOS/Linux) | Use `pipx install bdistill` or `uv tool install bdistill` |
| `No module named 'mcp'` | Old version (0.1.0) didn't include `mcp` as a core dependency | Upgrade: `pipx upgrade bdistill` or reinstall: `pipx install --force bdistill` |
| `Error: No such command 'setup'` | Old version (0.1.0) didn't have the `setup` command | Upgrade: `pipx upgrade bdistill` — version 0.1.1+ has `setup` |
| `bdistill` works inside project but not elsewhere | Installed with `pip install` or `uv pip install` (venv-only) | Reinstall with `pipx install bdistill` or `uv tool install bdistill` for global access |
| `bdistill-mcp: command not found` | Binary not on PATH | Use full path in MCP config (see Step 3) |
| MCP server doesn't appear in tool | Session started before MCP was added | Start a **new** session |
| `/bdistill-setup` not recognized | It's a Claude Code slash command, not a terminal command | Run it inside a Claude Code session, or use `bdistill setup` in your terminal |
| `bdistill check` shows warnings | Partial install | Follow the warnings — they tell you exactly what to fix |

## The prediction loop

bdistill isn't just extraction — it's a prediction system. Extract an event calendar from any domain, get notified when events approach, run structured predictions, and track your accuracy over time.

```
/bdistill-calendar extract crypto --grounded
  → Model extracts 12 events: ETH upgrades, BTC halving, FOMC, CPI, options expiry...
  → Each has dates, recurrence patterns, and prediction scenarios

/bdistill-calendar check crypto
  → "ETH Pectra upgrade in 8 days. 2 scenarios ready."
  → "FOMC rate decision in 12 days. 3 scenarios ready."

/bdistill-predict "What happens to ETH price if Pectra upgrade goes smoothly?"
  → Prediction card: up/moderate, confidence 0.71
  → Evidence: 3 rules, 1 event pattern, 1 causal chain
  → Assumptions, failure modes, confidence boundary

/bdistill-predict --resolve {card_id} --actual up --magnitude moderate
  → Correct! Ledger updated.

/bdistill-predict --ledger
  → "15 predictions, 9 resolved, 78% directional accuracy"
```

**Works with any domain.** Agriculture, macro rates, crypto, elections, biotech FDA dates, earnings — the model knows the event calendars, the mechanisms, and the rules. bdistill makes it systematic and trackable.

## Modes

### 1. Behavioral X-Ray (`/bdistill-xray`)

The agent probes *itself* — 30 questions across 6 behavioral dimensions. You get a visual HTML report showing how the model actually behaves.

```
You: /bdistill-xray

Agent: Starting self-probe... 30 questions across 6 dimensions.
       [answers each question, auto-tagged with behavioral metadata]

       Done! Report: data/self-probe/report.html

       Key findings:
       - Refusal rate: 12% (offers alternatives 78% of the time)
       - Shows reasoning: 85% (you don't need "think step by step")
       - Admits ignorance: 41% on fabrication traps
       - Defaults to prose, not lists
```

**6 probe dimensions:**

| Dimension | What it tests |
|-----------|--------------|
| `tool_use` | When to call tools vs. answer from knowledge, chaining, ambiguity |
| `refusal` | Safety boundaries, false-positive refusals, refusal style |
| `formatting` | Lists vs. prose, code blocks, length calibration |
| `reasoning` | Chain-of-thought, trick questions, self-correction |
| `persona` | Identity, tone matching, composure under hostility |
| `grounding` | Hallucination resistance, fabrication traps, knowledge limits |

**No API key.** Uses your existing subscription.

### 2. Event-Centered Extraction (`/bdistill-events`)

Extract what happens **before, during, and after** recurring events — temporally ordered and spatially layered. Every domain has events that drive decisions: earnings releases, regulatory deadlines, seasonal cycles, report publications.

```
You: /bdistill-events --custom "quarterly earnings season, FOMC rate decision"

Agent: [extracts 15-17 prompts per event across 5 dimensions]
       Timeline:   T-14d → T-7d → T-day → T+1d → T+7d (what happens when)
       Spatial:    local → regional → national → global (what happens where)
       Causal:     how effects propagate across markets and time
       Historical: 3 past instances with surprise, reaction, and lesson
       Conditional: IF pre-condition THEN impact is amplified/dampened

       Done! Sorted by event → time → space
```

**Preset events:** 12 agriculture (WASDE, planting, harvest, CFTC), 5 finance (FOMC, NFP, CPI, earnings, opex). Or provide any recurring event with `--custom`.

**No API key.**

### 3. In-Session Knowledge Extraction (`/bdistill-distill`)

Extract structured domain knowledge from the model you're already talking to. bdistill generates targeted questions, the agent answers from its training knowledge, and bdistill structures the responses into a reference knowledge base.

```
You: /bdistill-distill agriculture

Agent: [bdistill generates 50 targeted agriculture questions]
       Q1: "What numeric thresholds determine soybean basis risk levels?"
       [answers with specific basis points, regional spreads, seasonal patterns]
       Q2: "How does crush margin optimization work mechanistically?"
       [answers with GPM calculation, board crush spread, processor economics]
       ...
       Done! 50 entries exported to data/knowledge/agriculture-a1b2c3.jsonl

       Avg quality: 0.78
       Categories: commodity markets, agronomy, logistics, policy
```

**Output format** — structured reference data, not training data:

```json
{
  "question": "What numeric thresholds determine soybean basis risk levels?",
  "answer": "Soybean basis risk is measured as the difference between local cash price and CBOT futures. Basis wider than -50 cents signals elevated risk...",
  "domain": "agriculture",
  "category": "commodity markets",
  "tags": ["quantitative", "threshold", "evidence-based"],
  "quality_score": 0.82,
  "source_model": "Claude Sonnet 4",
  "extracted_at": "2026-03-21T00:30:00Z"
}
```

Use this for lookup tables, Q&A reference datasets, research corpora, or as structured input to your own projects.

**Works with any domain.** 6 presets (agriculture, climate-commodity, medical, legal, cybersecurity, finance) or provide your own terms on any topic: `--custom "insurance claim triage, subrogation, total loss valuation"`. Don't know where to start? Use `/bdistill-discover` — describe your work and bdistill maps the territory for you.

**No API key.** Uses your existing subscription.

### 4. Chat History Analysis (`/bdistill-analyze-chats`)

Analyze your existing conversations from ChatGPT or Claude exports:

```bash
bdistill analyze conversations.json
```

Supports ChatGPT exports (`conversations.json`), Claude exports, generic JSON, and markdown chat logs. Generates the same visual behavioral report based on *your real usage* — not synthetic probes.

**No API key.**

### 5. Local Model Extraction (`/bdistill-extract`)

Extract domain knowledge from open-source models running locally via Ollama:

```bash
bdistill extract --domain agriculture --model qwen3:4b
bdistill extract --terms "soybean basis,crush margin,CBOT contango" --model mistral:7b
bdistill extract --list-domains
```

Generates diverse prompts, harvests completions, deduplicates, quality-scores, and exports structured JSONL.

**No API key. Runs entirely on your machine.**

### 6. Decision Rules Extraction (`/bdistill-rules`)

Extract structured IF-THEN decision rules from the model's embedded knowledge — thresholds, combinatorial conditions, temporal windows, exception rules, and priority hierarchies.

```
You: /bdistill-rules agriculture

Agent: [bdistill generates rule-extraction prompts]
       Q1: "What numeric thresholds determine soybean basis risk levels?"
       → IF basis < -50 cents AND carry > 15 cents THEN basis_risk = elevated
       → IF CBOT futures contango > 20 cents THEN storage_play = profitable
       ...
       Done! 45 rules exported to data/rules/agriculture-rules-a1b2c3.jsonl
```

**Output**: Structured JSONL with conditions, actions, confidence scores, and exception lists. Ready for rule engines, decision trees, or audit trails.

**No API key.**

### 7. Prediction Assembly (`/bdistill-predict`)

Compose structured predictions from the model's domain knowledge, web search data, and your existing knowledge base. The prediction pipeline decomposes your question, optionally deep-distills 36 knowledge probes, recalls matching KB entries, grounds with current web data, adversarially challenges the reasoning, and produces a shareable Prediction Card.

**Four tiers:**

| Mode | Flags | Time | Evidence | Best for |
|------|-------|------|----------|----------|
| Quick | (default) | ~5 min | 6 extractions from model | Exploration, "what if" scenarios |
| Medium | `--medium_depth` | ~10-12 min | 12-18 probes (top 3 strategies per sub-question) | Serious analysis, good depth/speed balance |
| Grounded | `--grounded` | ~10 min | 6 extractions + KB recall + web search | Current data, real analysis |
| Deep + Grounded | `--deep --grounded` | ~25 min | 36 deep probes + KB recall + web search | High-stakes predictions, trading |

**Directional predictions:**
```
/bdistill-predict "What happens to fertilizer prices if Hormuz closes?" --grounded
  → Direction: UP / Magnitude: MAJOR / Confidence: 65%
  → 6 evidence citations with quality scores and source tags [kb]/[web]/[model]
  → Shareable HTML card auto-generated
```

**Binary predictions for prediction markets (Polymarket/Kalshi):**
```
/bdistill-predict "Will urea exceed $600/t by June?" --binary --market_price 0.45 --grounded
  → Prediction: YES / Probability: 72% / Market: 45% / Edge: +27%
  → Brier score computed on resolution
```

**Medium-depth — the sweet spot between fast and thorough:**
```
/bdistill-predict "Impact of FOMC rate decision on 10Y yields" --medium_depth --grounded
  → 12-18 targeted probes (top 3 strategies per sub-question, ~10-12 min)
  → All auto-merged into KB → next prediction on same domain is richer
```

**Deep mode — mine the LLM's embedded knowledge before predicting:**
```
/bdistill-predict "Impact of FOMC rate decision on 10Y yields" --deep --grounded
  → 36 targeted probes (thresholds, mechanisms, precedents, edge cases, ~25 min)
  → All auto-merged into KB → next prediction on same domain is richer
```

**Track outcomes over time:**
```
/bdistill-predict --resolve {card_id} --actual down --magnitude major     # directional
/bdistill-predict --resolve {card_id} --actual_outcome yes                # binary → Brier score
/bdistill-predict --ledger
  → 15 predictions, 9 resolved, 78% accuracy, Brier: 0.18
```

**No API key.** See [docs/prediction-pipeline.md](docs/prediction-pipeline.md) for the full architecture and [docs/prediction-markets-playbook.md](docs/prediction-markets-playbook.md) for the Kalshi/Polymarket trading playbook.

### 8. Event Calendar (`/bdistill-calendar`)

Extract a domain event calendar from model knowledge — recurring reports, seasonal windows, and conditional triggers. The calendar surfaces approaching events with ready-to-run prediction scenarios.

```
You: /bdistill-calendar extract us-elections --grounded

Agent: [extracts 15 events: primaries, debates, convention, election day...]
       [each with dates, recurrence, prediction scenarios]
       [verifies dates via web search]

       Calendar saved! 15 events (10 exact, 3 approximate, 2 conditional)

You: /bdistill-calendar check us-elections
  → "New Hampshire primary in 12 days. 3 scenarios ready."
  → /bdistill-predict "What if candidate X wins NH by >10 points?"
  → /bdistill-predict "What if result is within 3 points?"
```

**Add your own events:**
```
/bdistill-calendar add crypto "My Conference" --date 2026-06-15
```

**Staleness detection:** Events extracted >90 days ago are flagged for re-verification.

**No API key.**

## All eleven modes at a glance

| Mode | Command | What it produces | API Key |
|------|---------|-----------------|---------|
| Prediction Assembly | `/bdistill-predict` | Structured prediction cards with evidence, uncertainty, outcome tracking | No |
| Event Calendar | `/bdistill-calendar` | Domain event calendar with approaching events + prediction scenarios | No |
| Event Extraction | `/bdistill-events` | Temporally ordered, spatially layered event timelines | No |
| Knowledge Extraction | `/bdistill-distill` | Structured Q&A reference data with adversarial validation | No |
| Rules Extraction | `/bdistill-rules` | IF-THEN decision rules with thresholds and exceptions | No |
| Grounded Time-Series | `/bdistill-timeseries` | Web-sourced dated data (t, y_t, text) with trend/anomaly/regime signals | No |
| Tabular ML Data | `/bdistill-schema` | Feature-label CSV for regression, classification, XGBoost | No |
| Behavioral X-Ray | `/bdistill-xray` | Visual HTML report of model behavior patterns | No |
| Privacy Probe | `/bdistill-xray` + `bdistill_privacy_probe` | Exposure risk audit — what the LLM knows about your org | No |
| Chat Analysis | `/bdistill-analyze-chats` | Behavioral profile from your real conversation history | No |
| Local Extraction | `/bdistill-extract` | JSONL reference data from open-source models via Ollama | No |

## MCP Tools

When connected via MCP, the agent has access to 56 tools:

| Tool | What it does |
|------|-------------|
| **Discovery** | |
| `bdistill_discover` | Map an unknown domain from a vague work description |
| `bdistill_discover_respond` | Record domain map or seed terms for discovery |
| `bdistill_discover_select` | Record which topics the user selected |
| **Behavioral Self-Probe** | |
| `bdistill_self_start` | Start behavioral self-probe |
| `bdistill_self_respond` | Record answer, get next question |
| `bdistill_self_report` | Generate visual HTML report |
| `bdistill_self_status` | Check probe progress |
| `bdistill_list_probes` | List probe dimensions |
| `bdistill_preview` | Preview sample questions |
| **Knowledge Extraction** | |
| `bdistill_distill_start` | Start in-session knowledge extraction (adversarial by default) |
| `bdistill_distill_respond` | Record knowledge answer, get next question or challenge |
| `bdistill_distill_export` | Export structured knowledge base with next-step guidance |
| **Rules Extraction** | |
| `bdistill_rules_start` | Start decision rules extraction |
| `bdistill_rules_respond` | Record rules, get next prompt or challenge |
| `bdistill_rules_export` | Export structured IF-THEN rules |
| **Event Extraction** | |
| `bdistill_event_start` | Start event-centered extraction |
| `bdistill_event_respond` | Record event timeline answer |
| `bdistill_event_export` | Export event timelines as JSONL |
| **Tabular ML Data** | |
| `bdistill_tabular_start` | Start tabular ML data generation |
| `bdistill_tabular_respond` | Submit generated rows, get next batch prompt |
| `bdistill_tabular_export` | Export tabular data as CSV |
| **Grounded Time-Series** | |
| `bdistill_grounded_start` | Start web-sourced time-series extraction |
| `bdistill_grounded_respond` | Submit web search results as JSON rows |
| `bdistill_grounded_signal` | Compute trend/anomaly/regime signals |
| `bdistill_grounded_export` | Export grounded time-series as CSV |
| **Privacy Probe** | |
| `bdistill_privacy_probe` | Generate privacy probe terms for an organization |
| `bdistill_privacy_classify` | Classify extraction results by exposure risk |
| **Knowledge Base Management** | |
| `bdistill_kb_citations` | Check citations in knowledge base |
| `bdistill_kb_cross_validate` | Cross-validate entries across models |
| `bdistill_kb_contradictions` | Detect contradictions in knowledge base |
| `bdistill_kb_resolve` | Start contradiction resolution |
| `bdistill_kb_resolve_respond` | Record resolution decision |
| `bdistill_kb_review` | Mark entries as reviewed/invalid |
| `bdistill_kb_stale` | Find stale entries |
| **Guided Start** | |
| `bdistill_start` | Single entry point — describe what you want, get routed to the right mode |
| **Prediction Assembly** | |
| `bdistill_predict_start` | Start prediction session (supports `--binary`, `--medium_depth`, `--deep`, `--grounded`, `--market_price`) |
| `bdistill_predict_respond` | Record answer, advance through decompose/deep_distill/extract/challenge/predict |
| `bdistill_predict_export` | Export Prediction Card (JSON + shareable HTML) to disk, merge evidence into KB |
| `bdistill_predict_share` | Generate/regenerate shareable HTML prediction card |
| `bdistill_predict_resolve` | Record actual outcome (directional or binary with Brier score) |
| `bdistill_predict_ledger` | Show prediction track record — accuracy + Brier score |
| **Event Calendar** | |
| `bdistill_calendar_extract` | Start calendar extraction session for a domain |
| `bdistill_calendar_respond` | Record answer, advance through seed/detail/verify |
| `bdistill_calendar_save` | Save verified calendar to disk |
| `bdistill_calendar_check` | Show upcoming events with ready-to-run prediction commands |
| `bdistill_calendar_add` | Add a manual event to an existing calendar |
| `bdistill_calendar_remove` | Remove an event by ID |
| **Export** | |
| `bdistill_export_excel` | Export knowledge as formatted Excel workbook (.xlsx) |
| `bdistill_export_checklist` | Export as audit-ready checklist with blank status/evidence columns |
| `bdistill_export_prompt` | Export curated system prompt for Claude Projects, Cursor rules, Copilot instructions, ChatGPT custom GPTs |
| `bdistill_export_harness` | Export as importable Python module or JSON for harness sub-agents (RULES + CONTEXT + `build_prompt()`) |
| `bdistill_training_export` | Export as fine-tuning JSONL (alpaca/sharegpt/openai formats, date-stamped filenames) |
| **Local Extraction** | |
| `bdistill_extract_check` | Verify Ollama is ready |
| `bdistill_extract_domains` | List preset domains |
| `bdistill_extract_run` | Run local model extraction |
| `bdistill_extract_results` | View extracted knowledge |

## Slash Commands (Claude Code)

| Command | What it does |
|---------|-------------|
| `/bdistill-xray` | Run a full behavioral self-probe |
| `/bdistill-xray-report` | Generate report from completed probe |
| `/bdistill-discover` | Don't know what to extract? Describe your work, get a domain map |
| `/bdistill-events` | Extract event timelines — what happens before, during, after recurring events |
| `/bdistill-distill` | Extract structured domain knowledge (adversarial by default) |
| `/bdistill-distill-loop` | Autonomous extraction loop — `--target 250` auto-expands topics across waves until target reached |
| `/bdistill-deep-distill` | Multi-pass deep extraction — each pass goes deeper using previous results |
| `/bdistill-rules` | Extract structured IF-THEN decision rules from domain knowledge |
| `/bdistill-schema` | Generate tabular ML training data (regression, classification) |
| `/bdistill-timeseries` | Gather real-world time-series from the web (uses WebSearch) |
| `/bdistill-start` | **Start here** — describe what you want, get routed to the right mode |
| `/bdistill-predict` | Structured predictions with `--binary`, `--medium_depth`, `--deep`, `--grounded`, `--market_price` |
| `/bdistill-calendar` | Extract event calendar, check upcoming events with prediction scenarios |
| `/bdistill-training-export` | Export knowledge base as fine-tuning JSONL (alpaca/sharegpt/openai) |
| `/bdistill-extract` | Extract knowledge from local open-source model |
| `/bdistill-analyze-chats` | Analyze exported chat history |
| `/bdistill-resume` | Continue an interrupted session |
| `/bdistill-validate` | In-session quality audit — grades entries A-F on 5 dimensions |
| `/bdistill-setup` | Configure bdistill MCP server in current project (Claude Code slash command, not a terminal command) |

### Finding your way

With 20 slash commands and 54 MCP tools, the surface area can be overwhelming. Two built-in helpers:

- **Workflow recipes** — Type `6` (or "recipes") in `/bdistill-start` to see end-to-end sequences for 6 specific outcomes: building a KB, generating LoRA training data, steering an AI tool, feeding a harness, structured predictions, and event-driven prediction loops.
- **Command reference card** — Type `7` (or "commands") in `/bdistill-start` to see every command with its parameters, output format, and file location in one card.

Or just describe what you want to do: `/bdistill-start "I want to make my Claude more reliable for compliance work"` — the router figures out the right sequence.

**Domain-specific recipes** — See [docs/domain-recipes.md](docs/domain-recipes.md) for copy-paste command sequences across 9 professional domains: agri-commodity trading, AML/KYC compliance, pharma regulatory, insurance underwriting, tax planning, construction contracts, macro trading, crypto/DeFi, and political risk.

## Who this is for

**Commodities researchers and analysts**: Extract structured knowledge on basis risk, crush margins, futures curves, trade flows, and weather-commodity linkages. Build reference datasets you can search instead of re-prompting. Export to Excel for your existing workflows.

**Researchers and academics**: Extract structured knowledge on your domain — agriculture, finance, legal, engineering, any field. Build reference datasets from the model's training knowledge. Use as lookup tables, literature review aids, or structured input to your own analysis.

**Engineers and developers**: Build domain-specific Q&A datasets for your products. Extract the niche technical knowledge you need without manually prompting 200 times and copy-pasting into spreadsheets.

**Compliance and risk professionals**: Extract regulatory requirements as structured checklists with blank status/evidence columns. Export as Excel with quality color-coding.

**Prediction market traders (Polymarket, Kalshi)**: Run binary YES/NO predictions with probability, edge calculation, and Brier score tracking. Deep mode mines 36 knowledge probes before grounding with current web data. See the [prediction market walkthrough](#prediction-market-walkthrough) below.

**AI product builders**: Extract domain knowledge from open-source models via Ollama for fine-tuning specialized small models with LoRA. Use `--target 250` for autonomous extraction of training-ready datasets.

**Non-coding professionals**: Export validated rules as system prompts for Claude Projects, Cursor, Copilot, or ChatGPT custom GPTs — no code, just paste. `bdistill_export_prompt --platform claude-project` produces ready-to-paste markdown.

**Harness engineers**: Export knowledge as importable Python modules (`bdistill_export_harness`) with `RULES`, `CONTEXT`, and `build_prompt()` for deterministic sub-agent injection. The bridge between AI-extracted knowledge and code-level reliability.

## "I can just ask ChatGPT for this"

You can. And you'll get a great answer. Once. Then it's gone.

The question isn't whether AI can give you the answer — it can. The question is what happens to that answer at 3pm next Tuesday when your colleague needs it and you're on a call. It's in a chat thread somewhere. Maybe. If you can find it. If you remember which conversation. If the AI gives the same answer again (it won't).

| | Prompting | bdistill |
|---|-----------|---------|
| Get an answer | Yes | Yes |
| Challenge the answer for accuracy | If you remember to ask | Automatic — every answer gets adversarially challenged |
| Detect hallucinated citations | You google each one | Citation checker flags every claimed study |
| Check consistency across 200 entries | You read all 200 and remember | Contradiction detection runs programmatically |
| Cross-validate with a second model | You re-ask in GPT-4, compare by hand | Cross-model validation aligns entries and measures agreement |
| Find it 4 months later | Search your chat history (good luck) | Search the knowledge base by keyword, category, or tag |
| Share with your team | Copy-paste into a doc | Export as Excel with quality scores and review status |
| Build on it next month | Start a new chat from scratch | Knowledge base merges, deduplicates, compounds |
| Know what's stale | You remember when you asked | Stale detection flags entries past verification date |
| Audit trail for compliance | None | Every entry has reviewer name, verification date, prior answer history |

**Prompting is reading. bdistill is building a library.** Nobody argues against libraries by saying "I can just read the book again."

## Prediction market walkthrough

If you trade on Polymarket or Kalshi, here's what a real session looks like:

### Session 1: Quick scan (5 min)

```
/bdistill-predict "Will the Fed cut rates at the June FOMC?" --binary --market_price 0.40
```

The market says 40% YES. You want to know if that's right. bdistill decomposes the question, extracts evidence from its training data, challenges its own reasoning, and gives you a probability with edge:

```
Prediction: YES
Probability: 52%
Market: 40%
Edge: +12%    ← there might be a trade here
Confidence: 0.61
```

But this used only model knowledge — no current data. The numbers might be stale.

### Session 2: Grounded (10 min)

```
/bdistill-predict "Will the Fed cut rates at the June FOMC?" --binary --market_price 0.40 --grounded
```

Same question, now with `--grounded`. bdistill:
1. Recalls any matching entries from your KB
2. Web searches for current CME FedWatch data, recent FOMC minutes, latest CPI
3. Each evidence piece is tagged `[kb]`, `[web]`, or `[model]`
4. Challenge phase explicitly checks for KB-vs-web conflicts

Now the probability reflects today's data, not training-time snapshots.

### Session 3: Deep conviction (25 min)

```
/bdistill-predict "Will the Fed cut rates at the June FOMC?" --binary --market_price 0.40 --grounded --deep
```

All flags. bdistill:
1. Decomposes into 6 sub-questions
2. **Deep distills 36 probes** — mining thresholds ("at what core PCE level has the Fed historically cut?"), mechanisms ("how does the yield curve inversion signal feed into FOMC decisions?"), precedents ("what happened in June 2019 and July 2023?"), edge cases ("what would make them cut despite above-target inflation?")
3. All 36 answers auto-merge into your KB
4. Recalls matching KB entries (now including the 36 fresh ones)
5. Web searches for current data
6. Challenges with source conflict detection
7. Predicts with full evidence chain

You get a prediction card with 40+ evidence entries, each tagged and quality-scored. Open the HTML card, read the failure modes, decide if the edge is real.

### Resolve and track

```
# After the June FOMC meeting:
/bdistill-predict --resolve {card_id} --actual_outcome yes

  → Result: CORRECT
  → Brier score: 0.2304 (lower is better)

/bdistill-predict --ledger
  → 15 predictions, 9 resolved
  → Accuracy: 78%
  → Brier score: 0.18 (avg over binary predictions)
```

Your Brier score over 50+ predictions is a credibility signal most Polymarket traders can't produce.

### More prediction market prompts

```
# Geopolitics
/bdistill-predict "Will Iran test a nuclear weapon before 2027?" --binary --market_price 0.08 --grounded
/bdistill-predict "Will there be a ceasefire in the Israel-Iran conflict by July 2026?" --binary --market_price 0.22 --grounded --deep

# Commodities
/bdistill-predict "Will Brent crude exceed $120/bbl before July 2026?" --binary --market_price 0.35 --grounded
/bdistill-predict "Will urea FOB Middle East exceed $900/t by June 2026?" --binary --market_price 0.30 --grounded --deep

# Crypto
/bdistill-predict "Will Bitcoin exceed $150k by end of 2026?" --binary --market_price 0.28 --grounded
/bdistill-predict "Will Ethereum flip Bitcoin in market cap by 2027?" --binary --market_price 0.05 --grounded

# Elections
/bdistill-predict "Will the incumbent party win the next US presidential election?" --binary --market_price 0.45 --grounded --deep

# Tech
/bdistill-predict "Will OpenAI IPO before end of 2026?" --binary --market_price 0.45 --grounded
```

### Parameter reference

| Parameter | What it does | When to use |
|-----------|-------------|-------------|
| `--binary` | YES/NO mode with probability (0.0-1.0) instead of direction/magnitude | Prediction markets, any yes/no question |
| `--market_price 0.45` | Current market price for edge calculation (probability - market_price) | When you want to know if the market is mispriced |
| `--grounded` | Web search for current data + KB recall for prior knowledge | When you need today's numbers, not training data |
| `--deep` | Mine 36 knowledge probes before extraction (thresholds, mechanisms, precedents) | High-stakes predictions worth 25 min of analysis |
| `--timeframe 1q` | Prediction horizon: 1d, 1w, 1m (default), 1q, 1y | Match to the contract expiry |
| `--domain energy` | Domain tag for KB organization | Keeps your knowledge base organized |

### What you see in the session

When you run a prediction, the terminal shows each phase:

```
PREDICTION SESSION STARTED
  ID: 37fd374936fd
  Mode: DEEP + GROUNDED + BINARY — market price: 45%

━━━ Decompose ━━━
[AI breaks question into 6 sub-questions]

Recorded. [deep 1/36] strategy=threshold
━━━ DEEP DISTILL ━━━
[AI answers: "European ammonia plants shut down above $15/MMBtu TTF..."]

... [deep 2/36 through 36/36] ...

KNOWLEDGE BASE RECALL
  KB entries found: 42
  Mode: GROUNDED — web search required for current data

━━━ EXTRACT ━━━
[AI answers each sub-question using KB context + web search data]

━━━ CHALLENGE ━━━
[AI critiques its own evidence, lists KB vs web conflicts]
ADJUSTMENT: -0.10

━━━ PREDICT ━━━
Prediction: YES / Probability: 72% / Market: 45% / Edge: +27%

PREDICTION CARD EXPORTED
  Card ID: abc123
  JSON: data/predictions/cards/abc123.json
  HTML: data/predictions/cards/abc123.html   ← open in browser or send to anyone
```

## Building your data moat

In the AI era, the product isn't the UI — it's the data. Any interface can be "vibed into existence" in a weekend. The only real moat is proprietary data that compounds over time and can't be replicated by someone who starts tomorrow.

You already have the tools. You pay for Claude, Copilot, Cursor, or ChatGPT. You use them daily. But every answer disappears into a chat log you'll never find again. bdistill turns those sessions into a compounding, structured knowledge asset that grows every time you work.

### The compounding loop

```
Week 1:   /bdistill-distill agriculture                   →  50 validated entries on basis, crush, futures
Week 4:   4 sessions across 2 models, custom terms         → 200 entries, cross-validated
Month 3:  Weekly extractions covering logistics, policy     → 500+ entries, deduplicated
Month 6:  Temporal snapshots + seasonal pattern data        → irreplaceable commodity research dataset
```

Each session merges into your persistent knowledge base. Duplicates are resolved by keeping the higher-confidence entry. Cross-model runs surface disagreements. Adversarial rounds add validation. The dataset gets better every time you use it — and nobody can replicate the accumulation by installing bdistill today.

### What this looks like for real professionals

**Commodities research analyst**

You spend your days tracking soybean basis, crush margins, and USDA WASDE reports. The models you use know global agricultural trade patterns, historical price drivers, and supply chain dynamics — but that knowledge is locked behind one-off chat responses.

```
/distill --custom "soybean basis risk, crush margin optimization, CBOT futures contango, Brazilian safrinha corn, freight rate seasonality"
```

After 3 months: a structured reference dataset of 500+ entries on commodity market mechanics, cross-validated across Claude and GPT-4. Export as CSV, plug into your existing spreadsheets, or search instantly when a client asks a niche question. Your junior analysts can query it. Your competitors are still copy-pasting from ChatGPT.

**Grain trader / origination desk**

You track basis across 15 elevators, monitor CBOT spreads, and need to explain carry economics to new hires. The models know commodity market mechanics — but that knowledge is locked behind one-off chat responses.

```
/bdistill-rules agriculture
```

After one session: structured IF-THEN rules for basis trading decisions, carry trade thresholds, and seasonal patterns — exported as Excel. Hand it to your junior analyst. Your competitors are still copy-pasting from ChatGPT.

**Climate-commodity risk analyst**

You model weather impact on crop yields and need to track the relationship between La Nina cycles, growing degree days, and freight rates.

```
/bdistill-distill climate-commodity
/bdistill-deep-distill --custom "La Nina soybean yield impact, drought index interpretation, Mississippi River low water freight, Brazilian cerrado rainfall"
```

After 3 passes: a deep knowledge base covering weather-yield relationships, regional production risk, and logistics bottlenecks. Export as Excel, plug into your existing risk models. The adversarial challenges caught where Claude was over-confident on drought thresholds — those corrections are the most valuable entries.

**Financial analyst**

You model risk, price derivatives, and analyze market microstructure. The models know textbook finance — but you need it structured, validated, and searchable.

```
/bdistill-distill --custom "Black-Scholes limitations, volatility surface dynamics, jump-diffusion models, counterparty credit risk, CVA/DVA calculation"
```

The adversarial mode is critical here: when Claude claims the Black-Scholes assumption of constant volatility is "generally reasonable," the challenger forces it to explain volatility smile, skew, and term structure. The corrected, validated entry is the one worth keeping.

**Engineering team lead**

Your team uses AI daily but the knowledge stays in individual chat histories. bdistill centralizes it.

```
# Each team member runs extractions in their tool of choice
/distill --custom "kubernetes horizontal pod autoscaler, gRPC load balancing, PostgreSQL query optimization, Redis cluster failover"

# Knowledge compounds into a shared base
bdistill kb list        → shows all domains your team has built
bdistill kb search "connection pooling"  → instant lookup
bdistill kb export -d infrastructure -f markdown  → team wiki
```

### Why this is a moat and not just a feature

| Asset | Can a competitor replicate it? |
|-------|-------------------------------|
| Your UI | Yes — in a weekend with AI |
| Your prompts | Yes — prompt libraries are everywhere |
| Your 6-month, multi-model, adversarially-validated, domain-specific knowledge base with temporal snapshots | No |

The knowledge base is the intersection of three things that are unique to you:
1. **Your domain expertise** — the seed terms and custom questions only you know to ask
2. **Your temporal position** — you started extracting 6 months ago; nobody can go back in time
3. **Your cross-model validation** — you ran the same domains across Claude, GPT-4, and Copilot; the disagreements and agreements are data nobody else has

This is the same pattern that made Clearbit, Waze, and Plaid valuable: the product generates proprietary data as an unavoidable byproduct of people getting value. You don't ask for the data — you produce it by simply using the tool.

### The knowledge base CLI

```bash
bdistill kb list                              # Show all domains
bdistill kb stats -d medical                  # Quality, categories, models, coverage
bdistill kb search "atrial fibrillation"      # Keyword search across domains
bdistill kb export -d medical -f csv          # Download as spreadsheet
bdistill kb export -d medical -f markdown     # Readable knowledge document
```

## Where your data lives

Everything bdistill produces goes into a `data/` directory. The `--domain` flag names the KB — all sessions with the same domain merge into one file.

```
data/
├── knowledge/base/              ← YOUR KNOWLEDGE BASES (one file per domain)
│   ├── grain-trading.jsonl      ← all --domain grain-trading sessions compound here
│   ├── aml-compliance.jsonl     ← all --domain aml-compliance sessions compound here
│   └── custom.jsonl             ← default if you don't use --domain
├── knowledge/exports/
│   ├── training/                ← fine-tuning JSONL (date-stamped, never overwrites)
│   ├── prompts/                 ← system prompts (paste into Claude/Cursor/ChatGPT)
│   └── harness/                 ← importable Python/JSON for harness code
├── predictions/cards/           ← shareable HTML prediction cards
├── predictions/ledger.jsonl     ← outcome tracking + Brier scores
├── rules/                       ← IF-THEN decision rules
└── calendar/                    ← domain event calendars
```

**How compounding works:** Every session with `--domain grain-trading` merges into the same `grain-trading.jsonl`. Deduplication is automatic. Different domains stay separate. The KB is the hub — predictions recall from it, exports curate from it, and failed predictions tell you what to extract next.

**How to check:** `bdistill kb list` (all domains), `bdistill dashboard` (visual browser at localhost:8741), or `python -c "from bdistill import DATA_ROOT; print(DATA_ROOT)"` to find the path.

See [docs/data-directory.md](docs/data-directory.md) for the full guide: how to use each output individually or collectively, how the compounding loop works, and common questions.

## Project structure

```
bdistill/
├── .claude/commands/         # Slash commands for Claude Code
├── .codex/AGENTS.md          # Codex CLI agent config
├── .cursor/rules/            # Cursor rules
├── .github/
│   ├── copilot-instructions.md  # VS Code Copilot instructions
│   └── workflows/            # Auto-publish to PyPI + MCP Registry
├── .mcp/server.json          # MCP Registry manifest
├── CLAUDE.md                 # Claude Code project instructions
├── skills/                   # Installable skill definitions
│   ├── behavioral-xray/
│   └── knowledge-extraction/
├── integrations/             # Per-tool MCP configs
├── src/bdistill/
│   ├── cli.py                # CLI entry point
│   ├── mcp_server.py         # MCP server (52 tools)
│   ├── prediction_session.py # Prediction assembly engine — structured prediction cards
│   ├── calendar_session.py   # Event calendar — domain events + prediction triggers
│   ├── self_probe.py         # Self-probe engine + report generator
│   ├── knowledge_session.py  # In-session knowledge extraction engine
│   ├── knowledge_base.py     # Compounding knowledge store (merge, search, export)
│   ├── tabular_session.py    # Tabular ML training data generation
│   ├── challenger.py         # Adversarial challenge generation + scoring
│   ├── export_analyzer.py    # Chat history analysis
│   ├── config.py             # Configuration schema
│   ├── probes/               # 6 behavioral probe dimensions
│   ├── exporters/            # Stats summary
│   └── extraction/           # Local model extraction pipeline
│       ├── seeder.py         # Domain prompt generation
│       ├── harvester.py      # Local model querying (Ollama/vLLM/transformers)
│       └── inverter.py       # Knowledge mining + quality scoring
└── install.sh                # One-command cross-tool installer
```

## Available on

- **PyPI**: `pip install bdistill`
- **Antigravity Awesome Skills**: `npx antigravity-awesome-skills install bdistill`

## License

MIT

---

*bdistill helps you get more value from the AI models you already pay for. All extraction happens within your normal subscription session. Users are responsible for compliance with their model provider's terms of service.*
