Metadata-Version: 2.4
Name: automia
Version: 0.0.1
Summary: Automated Membership Inference Attacks: Discovering MIA Signal Computations using LLM Agents
Author-email: Toan Tran <vtran29@emory.edu>
License: MIT
Project-URL: Homepage, https://github.com/placeholder/automia
Project-URL: Repository, https://github.com/placeholder/automia
Project-URL: Documentation, https://github.com/placeholder/automia#readme
Project-URL: Bug Tracker, https://github.com/placeholder/automia/issues
Keywords: machine-learning,agents,membership inference attack,mia,llm
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.8
Description-Content-Type: text/markdown
Requires-Dist: pydantic>=2.0.0
Requires-Dist: langchain>=0.1.0
Requires-Dist: langchain-core>=0.1.0
Requires-Dist: PyYAML>=6.0
Requires-Dist: numpy>=1.24.1
Provides-Extra: openai
Requires-Dist: openai>=1.0.0; extra == "openai"
Requires-Dist: langchain-openai>=0.0.5; extra == "openai"
Provides-Extra: google
Requires-Dist: langchain-google-genai>=0.0.5; extra == "google"
Provides-Extra: postgres
Requires-Dist: psycopg2-binary>=2.9.0; extra == "postgres"
Provides-Extra: search
Requires-Dist: faiss-cpu>=1.7.0; extra == "search"
Requires-Dist: bm25s>=0.1.0; extra == "search"
Requires-Dist: PyStemmer>=2.0.0; extra == "search"
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: black>=23.0.0; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Provides-Extra: full
Requires-Dist: automia[google,openai,postgres,search]; extra == "full"
Provides-Extra: all
Requires-Dist: automia[dev,full]; extra == "all"

## Installation

To install `automia`:
> python -m pip install -e ".[all]"

A sandbox environment (`env.sh`) that will be used to execute the generated MIAs should be installed with common computation packages (such as numpy, torch, scipy,..) and packages required to run the MIAs. We recommend to save the model's outputs (i.e., logits) using safetensor and read load to save computational cost each trial.

## Running AutoMIA for a MIA setting:

1. Prepare a codebase template including `env.sh`, `template.py`, and `config.yaml`. Example in (`examples/bbllm/arxiv_pythia`)

2. Run vllm servers. `automia` requires two APIs (LLM and embedding) and then run automia
> python -m automia.main --experiment-dir examples/bbllm/arxiv_pythia --timeout 300 --model-name qwen --base-url http://localhost:9800/v1 --provider vllm --embedding-model-name qwen-embedding --embedding-base-url http://localhost:9700/v1  --budget 100 --output-dir results/bbllm/arxiv

3. `eval.py` and `vis.py` are available to eval the top 10 MIAs and visualize the MIAs

> python vis.py --output-dir results/bbllm/arxiv

A html file index.html will be written into results/bbllm/arxiv

> python eval.py --template examples/bbllm/arxiv_pythia/template.py --output-dir results/bbllm/arxiv

A cvs file will be written into results/bbllm/arxiv

```
Please make sure the stored data is available to reload for our example template.py (line 135 in examples/bbllm/arxiv_pythia/template.py) by running generate.py scripts.
The template.py must have an argument of `output-dir`, create this output directory, and write the results into `mia-results.json` with 4 keys: "auc_score", "tpr_1_score", "tpr_5_score", "combined_score".
A detailed documents and step-by-step to reproceduce our paper's experiments will be available soon! Feel free to reach out to me at <email> or create issues.
``` 
