Metadata-Version: 2.4
Name: airflow-dag-slot-booking-cc
Version: 0.0.1
Summary: Airflow plugin for DAG schedule slot booking — supports Airflow 2.x (Flask) and 3.x (FastAPI)
Home-page: https://github.com/jeevini/airflow-dag-slot-booking-cc
Author: Jeevini Manohar
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Framework :: Apache Airflow
Requires-Python: >=3.8
Description-Content-Type: text/markdown
Requires-Dist: apache-airflow>=2.0.0
Requires-Dist: fastapi>=0.100.0; python_version >= "3.8"
Requires-Dist: jinja2
Requires-Dist: psycopg2-binary
Dynamic: author
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# Airflow DAG Slot Booking Plugin

A visual scheduling plugin for Apache Airflow that shows **which time slots are already used and which are available** — through an interactive 7-day x 24-hour grid UI inside Airflow.

## The Problem

When you have dozens (or hundreds) of DAGs running on Airflow, scheduling conflicts become a real issue:
- Multiple DAGs scheduled at the same time compete for **workers, database connections, and API rate limits**
- This leads to **failures, timeouts, and slow runs**
- There's no built-in way in Airflow to **visualize** which time slots are crowded and which are free
- Teams end up **hardcoding cron schedules** in DAG scripts without knowing what's already running at that time

---

## Airflow Version Compatibility

> **Supports both Airflow 2.x and Airflow 3.x automatically — no configuration needed.**

The plugin detects your Airflow version at startup and registers itself using the correct interface:

| Airflow Version | Plugin Interface | URL |
|---|---|---|
| **2.x** (e.g. 2.8, 2.9, 2.10) | Flask Blueprint | `http://<host>/dag_slot_booking` |
| **3.x** (e.g. 3.0, 3.1+) | FastAPI app | `http://<host>/dag_slot_booking/` |

**You don't need to do anything differently** — just install the package and restart Airflow. The right interface is chosen automatically.

> **Note for Airflow 3.x users:** The URL has a trailing slash — use `/dag_slot_booking/` not `/dag_slot_booking`.

---

## Approach 1: View Available Slots (All Users — Zero Config)

**Just install and open.** The plugin reads your existing DAG schedules from Airflow's own metadata.

### Step-by-step setup:

```
your-airflow-project/
├── dags/
│   ├── my_etl_dag.py          # your existing DAGs (no changes needed)
│   ├── my_ml_pipeline.py
│   └── ...
├── docker-compose.yml          # or Helm chart
└── requirements.txt            # add the plugin here
```

**Step 1:** Install the plugin

```bash
# Option A: Add to requirements.txt
echo "airflow-dag-slot-booking-cc" >> requirements.txt

# Option B: Add to docker-compose.yml env
_PIP_ADDITIONAL_REQUIREMENTS: "airflow-dag-slot-booking-cc"

# Option C: Add to Dockerfile
RUN pip install airflow-dag-slot-booking-cc
```

**Step 2:** Restart your Airflow webserver

```bash
# Docker Compose (Airflow 2.x)
docker-compose restart airflow-webserver

# Kubernetes (Airflow 2.x)
kubectl rollout restart deployment/airflow-webserver -n airflow

# Kubernetes (Airflow 3.x — pod is called api-server)
kubectl rollout restart deployment/airflow-api-server -n airflow
```

**Step 3:** Open the UI

```
# Airflow 2.x
http://<your-airflow-host>/dag_slot_booking

# Airflow 3.x (note the trailing slash)
http://<your-airflow-host>/dag_slot_booking/
```

**Done.** You'll see a grid showing all your DAGs' schedules. No database, no variables, no DAG changes needed.

### Features:
- A **7-day x 24-hour grid** showing every DAG's cron schedule
- **Colored slots** = occupied, **gray slots** = available
- Click any colored slot to see **which DAGs** are scheduled at that time
- All times in **UTC** (Airflow's default), hover for your local time
- Filter by **day of week** or **DAG type**
- Stats: total DAGs, full slots, available slots

### Your existing DAGs stay the same:

```python
# Nothing changes — your DAGs work as before
from airflow import DAG
from datetime import datetime

dag = DAG(
    'my_etl_pipeline',
    schedule_interval='0 3 * * 1',  # hardcoded is fine for this approach
    start_date=datetime(2024, 1, 1),
)
```

The plugin just reads these schedules and shows them visually. Use it to **check what's free** before adding a new DAG.

---

## Approach 2: Book Slots from UI + Read Schedule from DB (Advanced)

This is how I use it in production with **150+ web scrapers**. Instead of hardcoding cron schedules in DAG scripts, you store them in a **database table** and book new slots from the UI.

### Why this approach is better:
- **No more hardcoded schedules** — DAG scripts read their cron from the DB at runtime
- **No merge conflicts** — team members book slots from the UI instead of editing DAG files
- **Capacity control** — prevents overloading a time slot (max 5 DAGs per slot)
- **Visibility** — everyone sees who booked what and when
- **Dynamic** — change a schedule in the UI, DAG picks it up automatically without code changes

### Step-by-step setup:

```
your-airflow-project/
├── dags/
│   ├── my_batch_dag.py         # reads schedule from DB (see example below)
│   └── ...
├── docker-compose.yml
└── requirements.txt            # add: airflow-dag-slot-booking-cc, psycopg2-binary
```

**Step 1:** Install the plugin (same as Approach 1)

```bash
pip install airflow-dag-slot-booking-cc
```

**Step 2:** Create the schedule table in your PostgreSQL database

```sql
CREATE TABLE "YOUR_SCHEMA"."DAG_SCHEDULE_CONFIG" (
    "DAG_NAME"       VARCHAR(255),
    "SCHEDULE_TIME"  VARCHAR(100),   -- cron expression in UTC (e.g., '0 3 * * 1')
    "OWNER"          VARCHAR(255),   -- who scheduled this DAG
    "COMMAND_NAME"   VARCHAR(255),   -- optional: spider name, script name, etc.
    "TYPE"           VARCHAR(50),    -- e.g., 'Scrapy', 'Python', 'Spark', 'ETL'
    "BATCHES"        VARCHAR(255),   -- optional: batch group name
    "SELENIUM"       VARCHAR(10),    -- optional: 'TRUE' if browser-based
    "IS_ACTIVE"      VARCHAR(10)     -- 'TRUE' or 'FALSE'
);
```

**Step 3:** Create an Airflow Variable named `EXTRACTION_CONFIG`

Go to **Airflow UI > Admin > Variables > +** and add:

- **Key:** `EXTRACTION_CONFIG`
- **Value** (JSON):

```json
{
  "DATABASE_HOST": "your-db-host",
  "DATABASE_USER": "your-db-user",
  "DATABASE_NAME": "your-db-name",
  "DATABASE_PASSWORD": "your-db-password",
  "DATABASE_PORT": "5432"
}
```

**Step 4:** Write your DAG script to read schedule from DB

Instead of hardcoding the schedule:

```python
# OLD WAY — hardcoded, no visibility, merge conflicts
dag = DAG('my_spider', schedule_interval='0 3 * * 1')
```

Read it from the database:

```python
# NEW WAY — schedule comes from DB, booked via the plugin UI
import psycopg2
from airflow import DAG
from airflow.models import Variable
from airflow.operators.bash import BashOperator
from datetime import datetime

config = Variable.get("EXTRACTION_CONFIG", deserialize_json=True)

def get_schedule(dag_name):
    """Read cron schedule from DAG_SCHEDULE_CONFIG table."""
    conn = psycopg2.connect(
        host=config["DATABASE_HOST"],
        user=config["DATABASE_USER"],
        dbname=config["DATABASE_NAME"],
        password=config["DATABASE_PASSWORD"],
        port=config["DATABASE_PORT"],
    )
    cursor = conn.cursor()
    cursor.execute("""
        SELECT "SCHEDULE_TIME"
        FROM "YOUR_SCHEMA"."DAG_SCHEDULE_CONFIG"
        WHERE "DAG_NAME" = %s AND "IS_ACTIVE" = 'TRUE'
    """, (dag_name,))
    row = cursor.fetchone()
    cursor.close()
    conn.close()
    return row[0] if row else None

schedule = get_schedule('my_spider')

if schedule:
    dag = DAG(
        'my_spider',
        schedule_interval=schedule,
        start_date=datetime(2024, 1, 1),
        catchup=False,
    )
    run_task = BashOperator(
        task_id='run_spider',
        bash_command='scrapy crawl my_spider',
        dag=dag,
    )
```

**Step 5:** Book slots from the UI

1. Open `http://<your-airflow-host>/dag_slot_booking` (or `/dag_slot_booking/` on Airflow 3.x)
2. Click a **free (gray) slot** on the grid
3. Fill in: DAG name, owner, type
4. Click **Schedule DAG**
5. The schedule is saved to your DB table
6. Your DAG script picks it up on the next scheduler refresh — **no code changes needed**

### The full workflow:

```
Team member opens plugin UI
        ↓
Sees the 7x24 grid — finds a free slot
        ↓
Clicks the free slot → booking form appears
        ↓
Fills in DAG name, owner, type → clicks "Schedule DAG"
        ↓
Schedule saved to DB table: DAG_SCHEDULE_CONFIG
        ↓
DAG script reads schedule from DB at runtime
        ↓
Airflow runs the DAG at the booked time
```

---

## How the Plugin Auto-Detects the Mode

| `EXTRACTION_CONFIG` variable | Mode | UI |
|-----|------|-----|
| **Not set** | Airflow Mode | Read-only grid (Approach 1) |
| **Set with DB credentials** | DB Mode | Grid + booking panel (Approach 2) |

No code changes needed — the plugin switches automatically based on whether the variable exists.

---

## Feature Comparison

| Feature | Approach 1 (View Only) | Approach 2 (DB + Booking) |
|---------|:---:|:---:|
| Visual 7x24 schedule grid | Yes | Yes |
| UTC times with local time on hover | Yes | Yes |
| Click slot to see DAG details | Yes | Yes |
| Filter by day / type | Yes | Yes |
| Live stats dashboard | Yes | Yes |
| Book new DAGs from UI | — | Yes |
| Slot capacity limits (max 5/slot) | — | Yes |
| Batch management | — | Yes |
| Cancel/deactivate DAGs | — | Yes |
| DAGs read schedule from DB | — | Yes |

---

## Real-World Example

I built this for my team's web scraping infrastructure:
- **150+ Scrapy spiders** scheduled across the week
- Each spider is a DAG that reads its cron schedule from `DAG_SCHEDULE_CONFIG`
- Team members use the plugin UI to **book free slots** for new spiders
- Spiders are grouped into **batches** (max 5 per batch) for coordinated execution
- No one edits DAG files to change schedules — it's all done from the UI

## Tech Stack

- **Backend:** FastAPI (Airflow 3.x) / Flask Blueprint (Airflow 2.x) — auto-detected at startup
- **Frontend:** Bootstrap 4, vanilla JavaScript
- **Database:** PostgreSQL via psycopg2 (optional, for Approach 2 only)
- **Compatibility:** Apache Airflow 2.x and 3.x, Python 3.8+

## Author

**Jeevini Manohar**

## License

MIT
