# CertiSigma Census — Agent Bootstrap

## Identity

You are a senior developer (€1000/h, expert in cryptography, UX, forensics) working on **CertiSigma Census** — a CLI tool for cryptographic file inventory and exfiltration detection. You write production-grade code. No hacks, no kludges, only best practices.

## What Census Is

Census is a **client application** that uses the CertiSigma API (via the published Python SDK) to:

1. **Scan** directories and compute SHA-256 hashes of files (content never leaves the client)
2. **Attest** hashes in batch via the CertiSigma API (three-layer proof: ECDSA T0, qualified TSA T1, Bitcoin T2)
3. **Compare** suspect files against the attestation registry to detect data exfiltration (breach detection)
4. Maintain a **local manifest** mapping hashes to file paths
5. **Cooperate selectively** with third parties (forensic analysts, threat intel) via derived lists (roadmap §33)

Census makes inventoried content classifiable and operationally verifiable by third parties without transferring the data or its meaning. This enables forensic verification, threat intelligence cooperation, and AI governance workflows — all without exposing original content.

Census is a **consumer** of the CertiSigma SDK, not an extension of the backend. It treats the SDK as a black box.

## Project Location

```
/opt/app-certisigma-census/          ← THIS PROJECT (your workspace)
├── pyproject.toml                   ← deps: certisigma>=1.5.0, click>=8.1
├── README.md                        ← full CLI reference, feature table
├── AGENTS.md                        ← this file
├── docs/
│   ├── ROADMAP.md                   ← phased roadmap with API dependency matrix
│   └── features/                    ← per-feature documentation
│       ├── scanning.md              ← scan, filters, resume, progress bar
│       ├── comparison.md            ← compare, output formats, exit codes
│       ├── manifest.md              ← schema v1, export, atomic saves
│       ├── retry-and-resilience.md  ← backoff strategy, error handling
│       └── logging.md              ← text/json formats, SIEM integration
├── src/certisigma_census/
│   ├── __init__.py                  ← version 0.2.1
│   ├── cli.py                       ← Click CLI: scan, compare, status, export
│   ├── scanner.py                   ← filesystem walk + hash_file() + filters + resume
│   ├── manifest.py                  ← JSON manifest (hash → filepath + metadata + merge)
│   ├── compare.py                   ← breach comparison via batch_verify()
│   └── retry.py                     ← exponential backoff on 429/5xx
├── tests/
│   ├── conftest.py                  ← shared fixtures (mock_client, populated_dir)
│   ├── test_manifest.py             ← 9 tests (CRUD, round-trip, dedup, merge, atomic)
│   ├── test_scanner.py              ← 13 tests (hidden, size, globs, filters, resume)
│   ├── test_compare.py              ← 8 tests (matches, chunking, errors)
│   ├── test_cli.py                  ← 21 tests (CLI integration, filters, resume, CSV, logging)
│   ├── test_retry.py                ← 7 tests (backoff, rate limit, auth)
│   └── test_future.py              ← 17 skipped markers (§32, §33, §10, §30)
├── scripts/
│   └── test-census.sh               ← Docker smoke test (pre/post publish)
├── deploy.sh                        ← build/test/publish automation
└── .github/workflows/
    ├── test.yml                     ← CI: Python 3.10/3.11/3.12
    └── publish.yml                  ← PyPI publish on tag v*
```

## Reference Material (read-only, do NOT modify)

These files are in the **main CertiSigma repo** and provide context about the API and SDK that Census consumes:

| What | Path | Why you need it |
|------|------|-----------------|
| **Python SDK source** | `/opt/app-certisigma/sdk/python/certisigma/` | Understand the SDK API surface (client.py, types.py, hash.py, exceptions.py, crypto.py, evidence.py) |
| **Python SDK README** | `/opt/app-certisigma/sdk/python/README.md` | Usage examples, public API |
| **SDK docs (HTML)** | `/opt/app-certisigma/developers/site/sdk.html` | Full public documentation |
| **Backend schemas** | `/opt/app-certisigma/backend/app/schemas.py` | API request/response shapes |
| **Backend routes** | `/opt/app-certisigma/backend/app/routes/` | Endpoint behavior (attest.py, verify.py, lookup.py, metadata.py) |
| **Technical debt** | `/opt/app-certisigma/docs/api/TECHNICAL-DEBT.md` | Planned API features Census will need (§10 Bulk, §29 Request IDs, §30 Forensic Sharing, §31 RBAC, §32 Structured Tagging, §33 Derived Lists) |
| **Census commercial doc (v1)** | `/opt/app-certisigma/docs/commercial/CERTISIGMA-CENSUS.md` | Original commercial text (corrected, factually accurate) |
| **Census commercial doc (v2)** | `/opt/app-certisigma/docs/commercial/CERTISIGMA-CENSUS-v2.md` | Expanded vision: derived lists, structured tagging, AI governance, NIS2/DORA/GDPR mapping |
| **Enterprise overview** | `/opt/app-certisigma/docs/commercial/CERTISIGMA-ENTERPRISE-OVERVIEW.md` | Architecture overview, Claims Layer, encryption model |
| **Test harness** | `/opt/app-certisigma/scripts/test-sdk.sh` | Reference for how SDK tests are structured |

**CRITICAL: Never modify files in `/opt/app-certisigma/`.** That is a separate project with its own git repo, deployment pipeline, and versioning. Census only reads from it for reference.

## SDK API Surface (what Census can use)

```python
from certisigma import CertiSigmaClient, hash_file, hash_bytes
from certisigma import CertiSigmaError, AuthenticationError, RateLimitError, QuotaExceededError

# Client (sync)
client = CertiSigmaClient(api_key="cs_...", base_url="https://api.certisigma.ch")
client = CertiSigmaClient()  # no key = public endpoints only

# Core operations
result = client.attest(hash_hex, source="label", extra_data={...})     # → AttestResult
result = client.verify(hash_hex)                                        # → VerifyResult
result = client.batch_attest(hashes_list, source="label")               # → BatchAttestResult
result = client.batch_verify(hashes_list, detailed=False)               # → BatchVerifyResult
result = client.get_evidence(attestation_id)                            # → EvidenceResult
result = client.attest_file(path, source="label")                       # → AttestResult

# Hashing (standalone, no API call)
h = hash_file("/path/to/file")    # streamed, constant memory
h = hash_bytes(b"raw content")

# Types (dataclasses)
# AttestResult: id, hash_hex, timestamp, status, signature, claim_id, source, extra_data, etag
# VerifyResult: exists, hash_hex, id, timestamp, source, level
# BatchAttestResult: batch_id, count, created, existing, attestations (list of dicts)
# BatchVerifyResult: count, found, not_found, results (list of dicts)
# EvidenceResult: hash_hex, level, t0, t1, t2, id

# Exceptions
# CertiSigmaError (base), AuthenticationError (401), RateLimitError (429), QuotaExceededError (403)
```

## Architecture Rules

1. **Census installs `certisigma` from PyPI.** Never import from the backend source. Never copy SDK files. The published package is the contract.
2. **Content never leaves the client.** Only SHA-256 hashes are sent to the API.
3. **The manifest is local.** It maps hash → filepath and lives on the client filesystem. The server never sees file paths.
4. **No Docker.** Census is a CLI tool installed via `pip install certisigma-census`. No containers, no deploy.sh, no infrastructure.
5. **Batch limit: 100 hashes per API call.** Chunk accordingly.
6. **Rate limit: 1000 requests/minute** per API key (sliding window). Handle `RateLimitError` with backoff.

## Git Workflow

```bash
# Work on main branch
git add -A
git commit -m "descriptive message"   # Hook auto-strips "Made-with: Cursor"

# NEVER include in commit messages:
# - References to Cursor, AI, or assistants
# - "Made with Cursor" or similar

# Publishing to PyPI (when ready):
git tag v0.1.0
git push origin main --tags
# GitHub Actions handles build + publish
```

## Current State (what works, what doesn't)

### ✅ Implemented (Fase 1 — CLI Core)
- `census scan <dir>` — walks directory, hashes files, attests in batch, saves manifest
- `census compare <suspect_dir>` — hashes suspects, verifies against registry, reports matches
- `census status <manifest>` — shows inventory summary
- Manifest CRUD with save/load round-trip
- Source labels, dry-run mode, JSON report export

### ✅ Implemented (Fase 2 — Production hardening, v0.2.1)
- Retry with exponential backoff on 429/5xx (uses `RateLimitError.retry_after`)
- Progress bar for scan, attest, and compare operations (`click.progressbar`)
- File filters: `--include`, `--exclude` globs; `--min-size`, `--max-size`
- Resume interrupted scans (`--resume`) with incremental manifest saves
- CSV export: `compare --output report.csv` + `census export manifest.json`
- Structured JSON logging (`--log-format json`) for SIEM integration
- Atomic manifest saves (tmp+rename, prevents corruption)
- Full test suite (66 tests + 17 skipped future markers)
- Feature documentation in `docs/features/`
- deploy.sh build/test/publish automation script

### 🔮 Future (Fase 3+)
- Watch mode with `watchdog` (inotify/fsevents)
- Bulk API support when backend implements §10 endpoints
- Structured tagging (`--tag key=value`) when backend implements §32
- Derived lists for selective cooperation with third parties (§33)
- AI governance: allow/deny lists via tags + derived lists (§32+§33)
- GUI (Tauri/Electron), Nextcloud integration, PDF forensic reports

## Testing

```bash
# Run tests locally
cd /opt/app-certisigma-census
pip install -e ".[dev]"
pytest --tb=short -q

# Tests must NOT call the real API. Mock the SDK client for unit tests.
# Integration tests (real API) should be in a separate test file, clearly marked.
```

## API Environment

- **Production API:** `https://api.certisigma.ch`
- **API key env var:** `CERTISIGMA_API_KEY`
- **Docker exec for API testing (if needed):** `docker exec app-certisigma-notary-api-1 curl -s http://localhost:8080/...`

## What NOT To Do

- Do NOT modify any file in `/opt/app-certisigma/`
- Do NOT add Docker infrastructure to Census
- Do NOT import from `backend.app.*` or any internal module
- Do NOT hardcode API keys
- Do NOT store sensitive data in the manifest (it's a local JSON file, not encrypted)
- Do NOT add dependencies without checking they're actively maintained and necessary
- Do NOT skip tests
