Metadata-Version: 2.4
Name: cf-datahive
Version: 0.1.5
Summary: Canonical result and measurement data storage APIs for Cogniflow
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: pyarrow>=12
Provides-Extra: pandas
Requires-Dist: pandas>=2.0; extra == "pandas"
Provides-Extra: test
Requires-Dist: pytest>=8.0; extra == "test"
Requires-Dist: pandas>=2.0; extra == "test"

# cf_datahive

`cf_datahive` is the Data Hive package boundary for Python-facing APIs/tooling around the canonical data hive root (`workspace/<data_hive>`).

## Boundary (Current Phase)

- Python package role (`sandcastle/cf_datahive`): read-oriented API/tooling/validation for pipeline-facing workflows.
- Native role (`sandcastle/cf_datahive/src/cf_datahive/cpp`): write gatekeeper and only allowed writer under `workspace/data_hive`.
- Step packages must stay thin wrappers and call the native gatekeeper instead of implementing filesystem/parquet helpers.
- Downstream first-party native consumers must discover the gatekeeper source surface through the owner package API instead of repo-relative path reach-in.

## Development workflow

- Current development mode is source-first via `scripts/fresh_install.ps1`.
- The package can now be built and published independently without changing the read/write ownership boundary above.

## Canonical layout

```
workspace/
  data_hive/
    <pipeline_id>/
      runs/
        <run_id>/
          manifest.json
          tables/
            <table_name>/
              part-0000.parquet
              part-0001.parquet
          artifacts/
            <artifact_name>
      latest.txt
```

- `latest.txt` stores the committed `run_id` and is updated atomically.
- `manifest.json` is the SOT for run metadata, table metadata, file hashes, and artifact hashes.

## Usage

```python
from pathlib import Path

from cf_datahive import (
    DataHiveClient,
    cf_datahive_cpp_consumer_cmake_path,
    cf_datahive_cpp_source_path,
)

workspace_root = Path("workspace")
client = DataHiveClient(str(workspace_root))

runs = client.list_runs("opcua_fifo_avg")
if runs:
    latest = runs[0].run_id
    manifest = client.load_manifest("opcua_fifo_avg", latest)
    table = client.read_table("opcua_fifo_avg", latest, "measurements")
    print(manifest.status, table.num_rows)
    print(cf_datahive_cpp_source_path())
    print(cf_datahive_cpp_consumer_cmake_path())
```

Native owner API:

- `cf_datahive_cpp_source_path()` returns the installed/package-owned native source root used by first-party build consumers such as `cf_basic_sinks`.
- `cf_datahive_cpp_include_path()` returns the include root inside that native source tree.
- `cf_datahive_cpp_consumer_cmake_path()` returns the owner-provided CMake helper for downstream native consumers that need runtime staging without re-encoding backend policy.

## Native consumer ownership

`cf_datahive` owns the backend-specific native build and runtime policy for `cf_datahive_cpp`.
First-party step packages should consume that owner surface instead of carrying their own DuckDB rules.

Typical consumer pattern:

```cmake
execute_process(
  COMMAND ${Python3_EXECUTABLE} -c "import cf_datahive as d; print(d.cf_datahive_cpp_source_path())"
  OUTPUT_VARIABLE CF_DATAHIVE_CPP_SOURCE_DIR
  OUTPUT_STRIP_TRAILING_WHITESPACE
)

add_subdirectory(${CF_DATAHIVE_CPP_SOURCE_DIR} ${CMAKE_CURRENT_BINARY_DIR}/cf_datahive_cpp_build)

cf_datahive_stage_consumer_runtime(
  TARGET my_step_plugin
  DESTINATIONS
    "${CMAKE_CURRENT_SOURCE_DIR}/../bin"
    "${SKBUILD_PLATLIB_DIR}/my_step_package/bin"
)
```

DuckDB configuration remains owner-controlled under `cf_datahive`:

- default mode is `static`
- shared mode can be selected with `CF_DATAHIVE_CPP_DUCKDB_LINKAGE=shared`
- owner-supported override vars are `CF_DATAHIVE_CPP_DUCKDB_INCLUDE`, `CF_DATAHIVE_CPP_DUCKDB_LIB`, `CF_DATAHIVE_CPP_DUCKDB_SOURCE`, and on Windows `CF_DATAHIVE_CPP_DUCKDB_DLL`
- when no override vars are set, `cf_datahive` searches for a repo-local `.native_deps/duckdb` by walking upward from the consuming CMake source tree before falling back to the owner package tree

## Manifest details

Each run stores a `RunManifest` (`schema_version="1.0"`) with:

- run lifecycle fields (`status`: `staged|committed|aborted`)
- table entries (`parquet`, schema fingerprint, row/file counts, optional file hashes)
- artifact entries (sha256, media type, size)
- optional `semantic_refs` placeholder map for future ontology links

Schema fingerprint is sha256 of Arrow schema serialization bytes.

## Guardrails

Run the repository guardrail check:

```
python tools/check_datahive_guardrails.py
```

The script performs C++/header scans and step-package checks that:

- use canonical `workspace/data_hive` literals outside the native gatekeeper location (hard fail)
- violate the thin-steps rule in `sandcastle/cf_basic_steps/*/src/*/cpp` (hard fail)
- reintroduce backend-specific ownership in `cf_basic_sinks` package surfaces (hard fail)

## Testing

Install test dependencies and run:

```
pip install -e "sandcastle/cf_datahive[test]"
pytest -q sandcastle/cf_datahive/tests
```

Published distribution name:

```bash
pip install cf-datahive
```

## Publishing

`cf_datahive` is published with the dedicated Windows workflow:

- Workflow: `.github/workflows/cf_datahive_windows_publish.yml`
- Package directory: `sandcastle/cf_datahive`
- PyPI tag: `cf-datahive-v<version>`
- TestPyPI tag: `cf-datahive-v<version>-test`

Local preflight:

```powershell
powershell -ExecutionPolicy Bypass -File scripts/mimic_windows_python_publish_workflow.ps1 `
  -WorkflowFile .github/workflows/cf_datahive_windows_publish.yml `
  -PackageDir sandcastle/cf_datahive `
  -PythonExe py `
  -PythonVersion 3.13
```

Queue a dry-run dispatch:

```powershell
powershell -ExecutionPolicy Bypass -File scripts/queue_windows_python_publish_workflow.ps1 `
  -WorkflowFile .github/workflows/cf_datahive_windows_publish.yml `
  -PackageDir sandcastle/cf_datahive `
  -PublishTarget testpypi `
  -Ref main `
  -RequireLocalPass `
  -DryRun
```

## Do / Don't

- Do: use `DataHiveClient` read APIs (`list_runs`, `load_manifest`, `read_table`, `open_artifact`) for inspection and validation.
- Do: route pipeline write ownership through `cf_datahive_cpp` in the sink path.
- Don't: write parquet files or artifacts directly into the canonical data hive root from pipeline steps.
- Don't: bypass manifest updates.
