ADFMentor
=========

A Python package for evaluating Azure Data Factory (ADF) submissions with comprehensive AI-driven grading.

Provides accurate, fair assessment of ADF implementations including architecture, design quality, error handling, and best practices.

## Features

- **AI-Driven Grading (Default)**: Comprehensive quality evaluation covering architecture, design, error handling, parameterization, and best practices
- ADF JSON validation for ARM templates and repo-export style files
- Safe ZIP extraction with size and file-count limits
- Recursive discovery of ADF JSON artifacts
- Optional rule-based grading for fast component checking (backward compatible)
- Clear feedback with specific recommendations
- Secure API key management via environment variables

## Installation

```bash
pip install ADFMentor
```

## Quick Start

```python
from ADFMentor import ADFMentor

# Initialize mentor
mentor = ADFMentor(api_key="your-api-key", model_name="gemini-2.0-flash-exp")

# Evaluate with comprehensive AI grading (default - most accurate and fair)
result = mentor.evaluate_adf(
    submission_path="path/to/submission.zip",
    question="Grade this ADF assignment"
)

print(f"Score: {result['score']}/100")
print(f"Feedback:\n{result['feedback']}")
print(f"Metadata:\n{result['metadata_report']}")

# Optional: Use custom grading rubric
custom_rubric = """
Evaluate focus on: error handling, parameterization, 
security practices, and performance optimization.
Score 0-100 with detailed feedback.
"""

result_custom = mentor.evaluate_adf(
    submission_path="path/to/submission.zip",
    prompt=custom_rubric,
    question="Assess ADF quality"
)

# Optional: Use fast rule-based grading (backward compatible)
result_fast = mentor.evaluate_adf(
    submission_path="path/to/submission.zip",
    use_rule_based=True
)
```

## Environment Setup

1. Get a free Google API key: https://makersuite.google.com/app/apikey
2. Copy `.env.example` to `.env`:
   ```bash
   cp .env.example .env
   ```
3. Edit `.env` and add your API key:
   ```
   GEMINI_API_KEY=your_actual_key_here
   ```
4. **Important**: Never commit `.env` to git (already protected in `.gitignore`)

Test files automatically load from `.env`.

## Supported Formats

- `.json` - ARM templates or repo-export JSON files
- `.zip` - ADF export packages (may include `.txt` notes)
- folder path - ADF project directory
- `.txt` - Optional written explanations (AI grading only)

## AI Grading Evaluation

The system uses Gemini AI (default: gemini-2.0-flash-exp) for comprehensive evaluation including:

- **Architecture & Design**: Pipeline organization, activity flow, component relationships
- **Error Handling**: Exception handling, retry logic, resilience patterns
- **Parameterization**: Runtime flexibility, hardcoded values, reusability
- **Best Practices**: Naming conventions, schema definitions, performance, security
- **Completeness**: All required components, connections, configurations
- **Code Quality**: JSON structure, documentation, clarity

Returns: Score (0-100) + detailed feedback with specific improvements.

## Security Validation

Safe ZIP extraction with:
- 200 file limit
- 50MB total size limit
- 10MB per-file limit
- Path traversal protection

## Custom Rubrics (PowerBI-Style)

For specialized evaluations, define custom prompts and pass them to `evaluate_adf()`:

```python
from ADFMentor import ADFMentor
import os

mentor = ADFMentor(api_key=os.getenv("GEMINI_API_KEY"))

# Define specialized prompts (like PowerBI DAX/Visual/Text pattern)
prompts = {
    "pipeline": """Evaluate pipeline logic correctness (40 pts), activity selection (30 pts), 
                   performance optimization (20 pts), and completeness (10 pts). 
                   Provide 50-80 word feedback.""",
    
    "architecture": """Evaluate architecture design (40 pts), error handling (30 pts), 
                        security practices (20 pts), and documentation (10 pts). 
                        Provide 50-80 word feedback.""",
    
    "configuration": """Evaluate configuration accuracy (40 pts), parameter usage (35 pts), 
                         security implementation (15 pts), and best practices (10 pts). 
                         Provide 50-80 word feedback."""
}

# Use specific prompt for different evaluation types
result_pipeline = mentor.evaluate_adf(
    submission_path="submission.zip",
    prompt=prompts["pipeline"],
    question="Evaluate the pipeline logic"
)

result_arch = mentor.evaluate_adf(
    submission_path="submission.zip",
    prompt=prompts["architecture"],
    question="Evaluate the architecture"
)

# Or use default comprehensive AI grading (no prompt needed)
result_default = mentor.evaluate_adf(submission_path="submission.zip")
```

**Key Features:**
- Define prompts in your test file (like PowerBI pattern)
- Pass different prompts for different evaluation focuses
- Default AI grading works without custom prompts
- 50-80 word concise feedback

```
ADFMentor/
  __init__.py
ADFMentorInternal/
  core.py
  models/
    gemini.py
  utils/
    adf_validator.py
    adf_processor.py
    adf_evaluator.py
```

## API

### ADFMentor

- `ADFMentor(api_key, model_name="gemini-2.0-flash-exp")`
- `evaluate_adf(submission_path, prompt=None, question=None)`

Returns:

```json
{
  "score": 0,
  "feedback": "...",
  "metadata_report": "..."
}
```

## Notes:
- For `.txt` submissions, provide `prompt` (and optionally `question`) to enable AI grading.
- For `.zip`/folder submissions, `.txt` files are treated as optional companion notes in AI mode.
## License

MIT
