<div align="center">
  <h1>🚀 Antigravity Lite (FinOps & AWS Glue Tools)</h1>
  <p><b>AWS Financial Auditor and Smart S3 Manager for PySpark Ecosystems</b></p>
</div>

---

## 🛑 The Silent AWS Glue Killer: Spark's Catalyst Optimizer
Have you ever wondered why your massive PySpark cluster just hangs for hours, consuming 100% CPU without writing a single byte of data when processing a <b>Wide Dataframe</b>?

Many Data Engineers blame data skew or bad partitioning, panicking and upscaling AWS Glue Worker instances to expensive `G.4X` or `G.8X` tiers. But throwing money at RAM is not the solution. **The architectural solution is not buying more RAM; it's isolating the math.**

## 📦 Installation

```bash
pip install antigravity-lite
```

## 🛠 Included Open-Source Tools

### 1. Smart S3 Renamer (`S3Finalizer` - Universal API)
Tired of PySpark polluting your Datalake with `part-00000...` strings and empty `_SUCCESS` files?
`S3Finalizer` is a native Boto3 utility that scans raw outputs and renames them sequentially and cleanly without breaking cluster concurrency. It works with Apache Spark, AWS Glue DynamicFrames, and standard S3 files seamlessly.

```python
from antigravity_lite.io.s3_finalizer import S3Finalizer

finalizer = S3Finalizer(bucket_name="my-corporate-datalake")

# Automatically re-sequence and format any outputs natively
finalizer.sequence_files(
    s3_prefix="raw_zone/sales/",
    pattern="ENTERPRISE_REPORT_{seq:04d}.parquet",
    starts_with="",      # Optional: Target specific outputs (e.g. "0000_part")
    ends_with=".parquet",# Optional: Ignore non-parquet files
    contains="part"      # Optional: Filter
)
# Magic Output: ENTERPRISE_REPORT_0001.parquet
```

### 2. AWS Glue FinOps Auditor (`AgAuditor`)
Inject this standalone tool to scan your AWS CloudWatch telemetry and compute exactly how many thousands of dollars you are wasting each month on inflated AWS Worker instances just to keep Spark's Catalyst Optimizer from crashing.

```python
from antigravity_lite.auditor.finops import AgAuditor

# Scan the cluster and generate a high-fidelity FinOps report
AgAuditor.run_aws_audit(
    region="us-east-1", 
    dias_analisis=7, 
    anonimize=True  # Optional: Masks sensitive Job names for safe sharing
)
```
*   **Premium TUI Output:** Generates professional ASCII tables with ANSI colors and Unicode borders (LinkedIn-ready screenshots).
*   **Resource Anonymization:** Deterministic hashing to mask internal AWS naming conventions and project IDs.
*   **OOM Risk Detection:** Automatically flags Jobs where JVM Heap usage exceeds 85%, indicating severe architectural instability.

### 3. S3 Directory Explorer (`AgS3DirectoryLister`)
Tired of discovering that AWS S3 is a flat namespace and doesn't have real "folders"? Listing hierarchies in Boto3 using the `CommonPrefixes` property is frustrating. 
`AgS3DirectoryLister` abstracts all the pain of native pagination and returns a clean logical "folder" tree.

```python
from antigravity_lite.io import AgS3DirectoryLister

explorer = AgS3DirectoryLister()
child_folders = explorer.list_folders("s3://your-bucket/datalake/bronze/")

# Imprime un cómodo árbol en tu terminal emulando un `ls`
explorer.print_tree("s3://your-bucket/datalake/bronze/")

# Cuenta archivos exactos en toda la jerarquía
total_parquet = explorer.count_files("s3://your-bucket/datalake/bronze/", suffix=".parquet")
print(f"Total archivos Parquet: {total_parquet}")
```

### 4. Multithreaded S3 Smart Copier (`AgS3SmartCopier`)
Cloning or merging massive Datalakes in S3 using traditional iterative scripts chokes your network and takes all afternoon. Additionally, spinning up a Spark cluster just to "copy data" is a gross waste of AWS billing.
`AgS3SmartCopier` spins up an asynchronous swarm in pure Python `ThreadPoolExecutor` to transfer thousands of files applying mathematical filters, at a fraction of the time of a conventional Boto3 script.

```python
from antigravity_lite.io import AgS3SmartCopier

copier = AgS3SmartCopier()

# Ultra-fast massive copy without spinning up Spark
copier.copy_path(
    origin_path="s3://data-lake/raw/",
    dest_path="s3://data-lake/historical/",
    starts_with="SALES_2026",
    ends_with=".parquet",    # Filters to ignore hidden trash files
    max_workers=10           # CPU threads fired simultaneously
)
```

### 5. Memory Optimizing Chunker (`DataFrameChunkerLite`)
Does your Spark cluster throw `Java Heap Space / OutOfMemoryError` when saving `Wide DataFrames` with dozens of columns? 
`DataFrameChunkerLite` intercepts Spark's execution graph, truncating the mathematical lineage using Logarithmic Tree-Reduction methodologies so your cluster survives without scaling your AWS infrastructure.

**Important Note:** *This is the Community Edition and is strictly limited to a maximum of 550 columns.* If you run this on a wider dataframe, it will safely reject execution.

```python
from antigravity_lite.core import DataFrameChunkerLite

# 1. Provide the wide dataframe and the primary key
chunker = DataFrameChunkerLite(df_crashing, id_cols=["client_id"], chunk_size=20)

def business_logic(chunk_df, index, multiplier):
    # This logic now runs isolated and safe from Catalyst OOM
    # using personal parameters passed via the Chunker
    for c in chunk_df.columns:
        if c != "client_id":
            chunk_df = chunk_df.withColumn(c, chunk_df[c] * multiplier)
    return chunk_df

# 2. Slice and safely process passing your own parameters (*args, **kwargs)
results = chunker.process_chunks(business_logic, multiplier=1.5)

# 3. Merges back all the columns automatically using O(log N) Binary Trees
df_final = chunker.join_chunks(results) 
```

### 6. Massive Binary Stream Concatenator (`AgStreamConcatenator`)
Are you trying to concatenate 100 GBs of CSV fragments into a single file but Python crashes with `MemoryError` using `pandas` or `open().read()`?
This utility implements high-performance constant memory `O(1)` piping. It dynamically transfers chunks of bytes directly to the disk (or S3) keeping a hard limit on RAM usage, while intelligently evading redundant CSV headers.

```python
import glob
from antigravity_lite.io import AgStreamConcatenator, AgS3DirectoryLister

joiner = AgStreamConcatenator()
explorer = AgS3DirectoryLister()

# --- LOCAL TO LOCAL ---
# Pass 500 files dynamically by exploring the local disk
local_files = glob.glob("/Users/data/chunks/*.csv")
joiner.concat_local(
    input_paths=local_files, 
    output_path="massive_combined.csv",
    has_header=True
)

# --- S3 T0 S3 ---
# Gather 1,000 files dynamically traversing S3 without bringing them to local disk
cloud_files = explorer.list_files("s3://lake/raw/sales/", suffix=".csv")
joiner.concat_s3(
    input_s3_uris=cloud_files,
    output_s3_uri="s3://lake/gold/massive_sales.csv",
    has_header=True
)
```

---

## 💎 Commercial Licensing (Antigravity PRO)

The *Lite* version can tell you you're burning thousands of dollars... **Purchasing the Antigravity PRO Enterprise License actually fixes it.**

If your `AgAuditor` report flags an **"⚠️ AST/OOM RISK"** or your Heap spikes past 85%, you need the **DataFrameChunker** mathematical engine (Exclusive to the Pro B2B Edition).
The enterprise version intercepts Spark's low-level planner and vertically slices the execution plan using **Logarithmic Binary Trees (Tree Reduce)** to forcibly truncate the AST *Lineage*. This drops your memory footprint so drastically that you can process half-a-billion operations on tiny `G.1X` clusters at zero `OutOfMemory` risk.

💻 **Request a Proof-of-Concept or Live Architecture Demo for B2B deployment by connecting via LinkedIn.**
