Metadata-Version: 2.1
Name: balarama
Version: 0.1.0
Summary: Local in-memory hybrid analytical index for Python
Home-page: https://github.com/mayankagrawalbro/ai-eng-db
Author: Mayank Agrawal
License: MIT
Project-URL: Homepage, https://github.com/mayankagrawalbro/ai-eng-db
Project-URL: Repository, https://github.com/mayankagrawalbro/ai-eng-db
Project-URL: Issues, https://github.com/mayankagrawalbro/ai-eng-db/issues
Keywords: analytics,database,in-memory,hybrid storage,python,developer tools
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: C
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Database
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: benchmark
Requires-Dist: duckdb (>=1.5.1) ; extra == 'benchmark'
Requires-Dist: lancedb (>=0.30.2) ; extra == 'benchmark'
Requires-Dist: pyarrow (>=20.0.0) ; extra == 'benchmark'

# Balarama

Balarama is a local in-memory hybrid analytical index for Python.

The core idea is simple:

- keep rows as the source of truth
- build sorted per-column access paths
- load data once
- run many repeated local queries and analytics very quickly

The primary use case now is a code corpus slice explorer for AI-assisted software development. You load function metadata once, then repeatedly filter by `language`, `repo`, `split`, `code_len_chars`, `doc_len_chars`, `code_line_count`, and similar fields for corpus prep, eval slicing, and prompt/example selection.

## What Works Today

The implemented path is centered on the original hybrid design:

- native C table engine with one row store and one sorted index array per column
- bulk-friendly load-once path that appends rows first and builds indexes once
- Python `HybridTable` API for equality/range filtering, counts, sums, row reconstruction, and value counts
- higher-level `CodeCorpusExplorer` API for the CodeSearchNet metadata workflow
- sample code corpus demo that runs locally from the repo
- public CodeSearchNet metadata benchmark path with DuckDB as the first external local comparator

The earlier vector-search path is still in the repo as an experiment, but it is no longer the main product story.

## Quick Example

```python
from pathlib import Path

import balarama as br

csv_path = Path("examples/data/code_corpus_sample.csv")
explorer = br.CodeCorpusExplorer.from_csv(csv_path)

python_tests = explorer.examples(
    language="python",
    split="test",
    has_docstring=True,
    limit=3,
)

docstring_rich_train_chars = explorer.sum_code_length(
    language="python",
    split="train",
    has_docstring=True,
    min_doc_len=40,
)

top_repos = explorer.top_repos(
    has_docstring=True,
    min_doc_len=20,
    top_n=5,
)
```

## Run It

Install the editable package into a virtualenv:

```sh
python3 -m venv .venv
. .venv/bin/activate
python -m pip install -e .
```

For a standard release install after PyPI publish:

```sh
python -m pip install balarama
```

Run the code corpus sample app:

```sh
make demo-code-corpus
```

Run the full test suite:

```sh
make test
```

Run the public CodeSearchNet metadata benchmark:

```sh
make benchmark-codesearchnet-hybrid
```

The benchmark prepares a public CodeSearchNet metadata CSV, loads it into Balarama’s hybrid table engine, and compares repeated slice/count/sum/example/top-repo workloads against DuckDB on the same corpus.

## Why This Direction

The strongest product story here is not “another database” and not “another vector store”.

It is:

- load a large structured dataset once
- keep it in memory
- slice it repeatedly with indexed filters
- reconstruct rows quickly
- do this locally from Python without running a service

That is a better match for the original architecture and for real engineering workflows around dataset prep, eval slicing, and example selection.

## Current Strengths We Intend To Preserve

- contiguous in-memory data structures
- explicit ownership and predictable behavior
- sorted per-column access paths
- stable `rowid` bridging rows and indexes
- benchmarkability
- small, understandable APIs

## Non-Goals For The Main Product Story

- no hosted service
- no distributed clustering
- no mutable operational database story
- no ANN-first or vector-first product identity
- no broad SQL engine ambitions

## Repository Guide

- [docs/product/hybrid_load_once_vision.md](docs/product/hybrid_load_once_vision.md): the reset vision for the original hybrid design
- [docs/research/hybrid_use_case_research.md](docs/research/hybrid_use_case_research.md): use-case shortlist and the recommended code-corpus story
- [docs/research/hybrid_storage_landscape.md](docs/research/hybrid_storage_landscape.md): research on related hybrid row/column systems and where Balarama fits
- [docs/product/quickstart.md](docs/product/quickstart.md): the main quick start for the hybrid Python path
- [docs/product/code_corpus_slice_explorer.md](docs/product/code_corpus_slice_explorer.md): user-facing walkthrough of the code corpus explorer
- [docs/product/python_api_reference.md](docs/product/python_api_reference.md): Python API reference for the hybrid and explorer APIs
- [docs/product/go_to_market_plan.md](docs/product/go_to_market_plan.md): sequential launch plan for the current library and product surfaces
- [docs/product/repo_launch_surface.md](docs/product/repo_launch_surface.md): recommendation for the clean launch-facing repo surface and what remains experimental
- [docs/product/codex_plugin_plan.md](docs/product/codex_plugin_plan.md): detailed design, implementation, testing, and docs plan for the Codex plugin
- [docs/product/pypi_release_runbook.md](docs/product/pypi_release_runbook.md): concrete same-day release steps for GitHub plus PyPI
- [docs/product/release_notes_v0_1_0.md](docs/product/release_notes_v0_1_0.md): release note for the first public Python launch
- [docs/product/example_output.md](docs/product/example_output.md): representative output from the user-style sample program
- [docs/benchmarking/public_dataset_strategy.md](docs/benchmarking/public_dataset_strategy.md): recommended public datasets and why they fit
- [docs/benchmarking/codesearchnet_hybrid_results.md](docs/benchmarking/codesearchnet_hybrid_results.md): latest one-million-row public benchmark results for the hybrid path
- [docs/benchmarking/duckdb_gap_plan.md](docs/benchmarking/duckdb_gap_plan.md): the concrete plan to close the performance gap with DuckDB on the right benchmark family
- [docs/research/codex_productization_strategy.md](docs/research/codex_productization_strategy.md): recommendation for packaging Balarama into Codex as a plugin plus skill plus MCP server
- [benchmarks/python/README.md](benchmarks/python/README.md): benchmark entry points and artifact locations
- [docs/product/marketing_copy.md](docs/product/marketing_copy.md): short product copy for posts and repo blurbs
- [docs/product/linkedin_post.md](docs/product/linkedin_post.md): outward-facing post for the current product story
- [docs/architecture/design_and_principles.md](docs/architecture/design_and_principles.md): hybrid-table design principles
- [docs/architecture/detailed_design.md](docs/architecture/detailed_design.md): current table/data-structure details
- [docs/architecture/hybrid_storage_layers.md](docs/architecture/hybrid_storage_layers.md): diagrams and explanation of the row store, sorted column indexes, and future hot column arrays

Experimental vector-path docs remain available here:

- [docs/product/vector_retrieval_direction.md](docs/product/vector_retrieval_direction.md)
- [docs/product/v1_collection_api.md](docs/product/v1_collection_api.md)
- [docs/architecture/vector_engine_design.md](docs/architecture/vector_engine_design.md)
