Metadata-Version: 2.4
Name: aetherflow-core
Version: 0.0.14
Summary: Workflow operations engine (flow/job/step) with state, resume, cleanup and plugins.
Author: aetherflow contributors
Project-URL: Homepage, https://github.com/aicodedao/aetherflow
Project-URL: Source, https://github.com/aicodedao/aetherflow/tree/master/packages/aetherflow-core
Project-URL: Documentation, https://github.com/aicodedao/aetherflow/tree/master/docs
Project-URL: license, https://github.com/aicodedao/aetherflow/blob/master/LICENSE
Project-URL: Bug Tracker, https://github.com/aicodedao/aetherflow/issues
Project-URL: CI, https://github.com/aicodedao/aetherflow/blob/master/.github/workflows/ci.yaml
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic>=2.6
Requires-Dist: PyYAML>=6.0
Requires-Dist: Jinja2>=3.1
Provides-Extra: zip
Requires-Dist: pyzipper>=0.3.6; extra == "zip"
Provides-Extra: rest
Requires-Dist: httpx>=0.27; extra == "rest"
Requires-Dist: anyio>=4.0; extra == "rest"
Requires-Dist: tenacity>=8.2; extra == "rest"
Provides-Extra: sftp
Requires-Dist: paramiko>=3.4; extra == "sftp"
Provides-Extra: smb
Requires-Dist: pysmb>=1.2.9; extra == "smb"
Requires-Dist: smbprotocol>=1.11; extra == "smb"
Provides-Extra: db
Requires-Dist: sqlalchemy>=2.0; extra == "db"
Requires-Dist: oracledb>=2.0; extra == "db"
Requires-Dist: psycopg2-binary>=2.9; extra == "db"
Requires-Dist: pymysql>=1.1; extra == "db"
Requires-Dist: pyexasol>=0.25; extra == "db"
Provides-Extra: mail
Provides-Extra: duckdb
Requires-Dist: duckdb>=1.0; extra == "duckdb"
Provides-Extra: parquet
Requires-Dist: pyarrow>=16.0; extra == "parquet"
Provides-Extra: excel
Requires-Dist: openpyxl>=3.1; extra == "excel"
Provides-Extra: reports
Requires-Dist: duckdb>=1.0; extra == "reports"
Requires-Dist: pyarrow>=16.0; extra == "reports"
Requires-Dist: openpyxl>=3.1; extra == "reports"
Provides-Extra: all
Requires-Dist: pyzipper>=0.3.6; extra == "all"
Requires-Dist: httpx>=0.27; extra == "all"
Requires-Dist: anyio>=4.0; extra == "all"
Requires-Dist: tenacity>=8.2; extra == "all"
Requires-Dist: paramiko>=3.4; extra == "all"
Requires-Dist: pysmb>=1.2.9; extra == "all"
Requires-Dist: smbprotocol>=1.11; extra == "all"
Requires-Dist: sqlalchemy>=2.0; extra == "all"
Requires-Dist: oracledb>=2.0; extra == "all"
Requires-Dist: psycopg2-binary>=2.9; extra == "all"
Requires-Dist: pymysql>=1.1; extra == "all"
Requires-Dist: pyexasol>=0.25; extra == "all"
Requires-Dist: duckdb>=1.0; extra == "all"
Requires-Dist: pyarrow>=16.0; extra == "all"
Requires-Dist: openpyxl>=3.1; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-timeout>=2.2; extra == "dev"
Requires-Dist: pytest-xdist>=3.6; extra == "dev"
Requires-Dist: pytest-cov>=5.0; extra == "dev"
Requires-Dist: respx>=0.21; extra == "dev"
Requires-Dist: ruff>=0.6; extra == "dev"
Requires-Dist: openpyxl>=3.1; extra == "dev"
Requires-Dist: pyzipper>=0.3.6; extra == "dev"
Dynamic: license-file

# aetherflow-core

[![GitHub Repo](https://img.shields.io/badge/GitHub-aetherflow-blue?logo=github)](https://github.com/aicodedao/aetherflow/tree/master/packages/aetherflow-core)
[![TestPyPI Version](https://img.shields.io/badge/dynamic/json?label=TestPyPI&url=https://test.pypi.org/pypi/aetherflow-core/json&query=$.info.version&cacheSeconds=10)](https://test.pypi.org/project/aetherflow-core/)
[![PyPI Version](https://img.shields.io/pypi/v/aetherflow-core?cacheSeconds=10)](https://pypi.org/project/aetherflow-core/)

`aetherflow-core` is a **YAML-first workflow engine** for ops-style ETL/ELT/automation: run-once jobs you can schedule anywhere (cron, Kubernetes, Airflow, Nomad, systemd, etc.).

It focuses on being:
- **Deterministic**: given the same inputs, do the same thing
- **Composable**: Flow → Job → Step, with clear boundaries
- **Safe by default**: retries, timeouts, idempotency helpers, structured results
- **Extensible**: add custom steps/connectors via a stable public API

This distribution ships the CLI **`aetherflow`**.

---

## Install

```bash
pip install aetherflow-core[all]
```

Minimal install (if you want to control optional deps yourself):

```bash
pip install aetherflow-core
```

Python: **3.10+**.

---

## Quickstart (run a flow)

Create a file `flow.yaml`:

```yaml
version: 1

flows:
  hello_flow:
    jobs:
      main:
        steps:
          - id: hello
            type: external.process
            command: ["python", "-c", "print('hello from aetherflow')"]
```

Run it:

```bash
aetherflow run --flow-yaml flow.yaml --flow-job main
```

> Tip: the built-in step type is **`external.process`** (not `external_process`).

---

## Concepts (how to think about it)

- **Flow**: top-level unit (a collection of jobs)
- **Job**: a run-once unit of work (a list or graph of steps)
- **Step**: one action (shell/process, connector ops, custom plugin step, etc.)
- **Resources**: named configs used by steps/connectors (e.g., DB connection profiles)
- **State / resume**: persistent run state for reliability (if enabled/configured)
- **Locks**: prevent overlapping executions when you run in parallel environments

Core stays intentionally run-once. Scheduling is handled by `aetherflow-scheduler` (separate package).

---

## CLI

```bash
aetherflow run --help
aetherflow run --flow-yaml flow.yaml --flow-job main
```

Full command reference (in this repo):
- [Cli. Reference](https://github.com/aicodedao/aetherflow/tree/master/docs/90-cli-reference.md)

---

## Public API (for plugins / integrations)

If you are writing plugins or integrating AetherFlow programmatically, **only import from**:

```python
from aetherflow.core.api import (
    FlowSpec, JobSpec, StepSpec, 
    Step, StepResult,
    register_step, register_connector,
    Settings, RunContext,
)
```

Everything outside `aetherflow.core.api` is internal and may change without notice.

Public API and SemVer policy:
- `aetherflow/docs/25-public-api-and-semver.md`

---

## Extending AetherFlow 

High-level flow:
1) Implement a `Connector`, `Step` subclass (or a compatible callable/factory, depending on your plugin style).
2) Register it with `register_connector(...)`.
2) Register it with `register_step(...)`.
3) Reference it by `type:` in YAML.

Plugin guide:
- [Plugins](https://github.com/aicodedao/aetherflow/tree/master/docs/18-plugins.md)
- [Connectors](https://github.com/aicodedao/aetherflow/tree/master/docs/19-connectors.md)
- [Steps](https://github.com/aicodedao/aetherflow/tree/master/docs/20-steps.md)

---

## YAML spec & guides

Start here:
- [Yaml Spec.](https://github.com/aicodedao/aetherflow/tree/master/docs/06-yaml-spec.md)
- [Flow yaml guide](https://github.com/aicodedao/aetherflow/tree/master/docs/10-flow-yaml-guide.md)
- [Flow in 15-minutes](https://github.com/aicodedao/aetherflow/tree/master/docs/93-flow-in-15-minutes.md)

Built-in catalog:
- [Step - External process](https://github.com/aicodedao/aetherflow/tree/master/docs/22-external-process-step.md)
- [Builtins Catalog](https://github.com/aicodedao/aetherflow/tree/master/docs/23-builtins-catalog.md)

---

## Docs (in this repository)

Canonical docs live in [Aetherflow Doumentation](https://github.com/aicodedao/aetherflow/tree/master/docs).

Start here:
- [Home Docs.](https://github.com/aicodedao/aetherflow/tree/master/docs/index.md)

If you need scheduling, install **[aetherflow-scheduler](https://github.com/aicodedao/aetherflow/tree/master/packages/aetherflow-scheduler)**.
