Metadata-Version: 2.4
Name: blindpath
Version: 0.1.0
Summary: BlindPath: An LLM-Navigation Benchmark for Partial-Observation Grid Worlds
Author-email: prooheckcp <vasco.soares.2001@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/prooheckcp/BlindPath
Project-URL: Repository, https://github.com/prooheckcp/BlindPath
Project-URL: Issues, https://github.com/prooheckcp/BlindPath/issues
Keywords: llm,benchmark,navigation,grid-world,partial-observability,game-ai
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.26.0
Provides-Extra: llm
Requires-Dist: anthropic>=0.25.0; extra == "llm"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"
Dynamic: license-file

# BlindPath 🗺️

**An LLM-Navigation Benchmark for Partial-Observation Grid Worlds**

BlindPath tests whether LLM agents can navigate a 2D grid world under
**partial observability** — a key challenge in real game AI that most
existing benchmarks ignore.

---

## Key Distinctions

| Property | BlindPath | Typical Benchmarks |
|---|---|---|
| Targets | Game AI agents | General reasoning |
| Observability | Partial (vision window) | Full global state |
| Input format | Non-rigid (hybrid prompting) | Fixed natural language |
| Task complexity | Long-horizon planning | Short single-step |

---

## Benchmark Metrics

| Metric | Formula | Measures |
|---|---|---|
| **Success Rate** | `Successes / Episodes × 100%` | Task completion |
| **Optimal Rate** | `OptimalPathLen / LegalActions × 100%` | Navigation efficiency |
| **Feasible Rate** | `LegalActions / TotalActions × 100%` | Spatial rule adherence |
| **Compliance Ratio** | `TotalActions / Iterations × 100%` | Output format compliance |
| **Exploration Ratio** | `SeenTiles / AccessibleVisionTiles × 100%` | Coverage |

---

## Installation

```bash
pip install -e .
```

Or install dependencies directly:

```bash
pip install anthropic numpy
```

---

## Quick Start

```python
from blindpath import BlindPathEnv, EnvConfig, RandomAgent, Session

config = EnvConfig(
    seed=42,
    env_size=(20, 20),
    num_goals=1,
    obstacle_count=20,
    vision_size=7,       # agent sees a 7×7 window around itself
)

agent = RandomAgent(seed=42)
session = Session(agent, config, verbose=True)

# Single episode
metrics = session.run()
print(metrics.summary())

# Multi-episode benchmark
results = session.run_benchmark(num_episodes=20, base_seed=42)
print(results.summary())
```

---

## Hybrid Prompting Architecture

Developers supply a **custom_prompt** that is combined with BlindPath's
pre-defined static rules before being fed to the LLM. This gives full
flexibility over how state information is presented:

```python
from blindpath import LLMAgent, Session, BlindPathEnv, StepResult

class MyAgent(LLMAgent):
    def _prompt(self, prompt: str) -> str:
        # Example: call your LLM's API here
        # return my_llm_client.generate(prompt)
        return '{"action": "Up"}'

def my_prompt(env: BlindPathEnv, result: StepResult) -> str:
    pos = result.agent_position
    return (
        f"You are at column {pos.x}, row {pos.y}. "
        f"Goals remaining: {result.goals_remaining}. "
        "Move towards any visible 'G' tile."
    )

agent = MyAgent(prompt_builder=my_prompt)
session = Session(agent, config)
```

---

## Action Space

```json
{"action": "Up"}
```

| Action | Effect |
|---|---|
| `"Up"` | Move agent by `(0, -1)` |
| `"Down"` | Move agent by `(0, +1)` |
| `"Left"` | Move agent by `(-1, 0)` |
| `"Right"` | Move agent by `(+1, 0)` |

---

## Grid Labels

| Symbol | Meaning |
|---|---|
| `A` | Agent |
| `E` | Empty |
| `O` | Obstacle (impassable) |
| `G` | Goal |
| `B` | Boundary (edge of map) |

---

## Hyperparameters

| Parameter | Type | Default | Notes |
|---|---|---|---|
| `seed` | `int \| None` | `None` | Falls back to system time; ensures reproducibility when set |
| `env_size` | `(int, int)` | `(40, 40)` | Minimum 5×5 |
| `num_goals` | `int` | `1` | ≥1 |
| `obstacle_count` | `int` | `0` | Number of obstacles |
| `vision_size` | `int` | `11` | Must be odd; ≥3 |

Each episode uses a **unique, procedurally generated map**. When running a
benchmark with `run_benchmark(num_episodes=N, base_seed=S)`, each episode
receives `seed = base_seed + i`, producing a different grid layout every time
while keeping the full run reproducible.

---

## Running the Baselines

```bash
# Random agent (10 episodes)
python experiments/run_random.py --episodes 10 --seed 42

# POMCP agent (5 episodes)
python experiments/run_pomcp.py --episodes 5 --sims 300

# LLM agent (1 episode)
export ANTHROPIC_API_KEY="sk-ant-..."
python experiments/run_llm.py --model claude-haiku-4-5 --verbose
```

---

## Project Structure

```
blindpath/
├── Constants/
│   ├── env.py        # Default/minimum config values
│   └── tiles.py      # Tile label constants (A, E, O, G, B)
├── DataClasses/
│   ├── env_config.py  # EnvConfig dataclass
│   ├── generated_map.py # GeneratedMap dataclass
│   ├── step_result.py # StepResult dataclass
│   └── vector2.py     # Vector2 dataclass
├── Util/
│   ├── astar.py       # A* pathfinding
│   └── grid_formatting.py # grid_to_ascii()
├── core/
│   ├── grid.py           # Grid class
│   ├── map_generation.py # Procedural map generation
│   └── env.py            # BlindPathEnv
├── agents/
│   ├── base.py           # BaseAgent abstract class
│   ├── random_agent.py
│   ├── pomcp_agent.py
│   └── llm_agent.py
├── eval/
│   ├── episode_metrics.py # EpisodeMetrics dataclass
│   └── benchmark_results.py # BenchmarkResults dataclass
└── session.py            # Session lifecycle + run_benchmark()
experiments/
├── arguments.py  # Shared CLI argument parsing
├── run_random.py
├── run_pomcp.py
└── run_llm.py
```

---

## Baselines

- **Random Agent** — Uniform random action selection. Lower bound.
- **POMCP Agent** — Partially Observable Monte-Carlo Planning with particle
  filters. Traditional method baseline for partial-info navigation.
- **LLM Agent** — Claude via the Anthropic API. The primary evaluation target.

---

## Iteration Limit

```
max_iterations = OptimalPathLength × (full_area / vision_area) × 3
```

Scales to allow for back-tracking proportional to how blind the agent is.

---

## Related Work

- [PPLN](https://arxiv.org/pdf/2310.03249) — Similar task, no partial observability
- [GWSOT](https://arxiv.org/pdf/2502.16690) — Spatial awareness probing, full state
- [GameTraversalBenchmark](https://arxiv.org/pdf/2410.07765) — 2D navigation, global state
- [GridRoute](https://arxiv.org/pdf/2505.24306) — Prompt engineering for pathfinding
