Metadata-Version: 2.4
Name: clawsbench
Version: 0.2.0
Summary: ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspaces
Project-URL: Homepage, https://github.com/benchflow-ai/ClawsBench
Project-URL: Paper, https://arxiv.org/abs/2604.05172
Project-URL: Dataset, https://huggingface.co/datasets/benchflow/ClawsBench
Project-URL: Website, https://clawsbench.benchflow.ai
Project-URL: Discord, https://discord.gg/mZ9Rc8q8W3
Author-email: BenchFlow <public@benchflow.ai>
License: CC-BY-NC-SA-4.0
Keywords: agent-safety,benchmark,evaluation,google-workspace,llm-agents,productivity,slack
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown

# ClawsBench

**Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspaces**

[![arXiv](https://img.shields.io/badge/arXiv-2604.05172-b31b1b.svg)](https://arxiv.org/abs/2604.05172)
[![Website](https://img.shields.io/badge/Website-ClawsBench-blue)](https://clawsbench.benchflow.ai)
[![Dataset](https://img.shields.io/badge/Dataset-HuggingFace-yellow)](https://huggingface.co/datasets/benchflow/ClawsBench)

ClawsBench evaluates LLM agents on realistic productivity tasks across **5 high-fidelity mock services** (Gmail, Calendar, Docs, Drive, Slack), measuring both **capability** (task success) and **safety** (harmful action prevention).

- **44 tasks**: 30 single-service + 14 cross-service, including 24 safety-critical scenarios
- **6 models**: Claude Opus 4.6, Claude Sonnet 4.6, GPT-5.4, Gemini 3.1 Pro, Gemini 3.1 Flash-Lite, GLM-5
- **4 harnesses**: OpenClaw, Claude Code, Codex, Gemini CLI
- **33 conditions**, **7,224 trials**

## Links

- **GitHub**: https://github.com/benchflow-ai/ClawsBench
- **Paper**: https://arxiv.org/abs/2604.05172
- **Dataset**: https://huggingface.co/datasets/benchflow/ClawsBench
- **Website**: https://clawsbench.benchflow.ai
- **Discord**: https://discord.gg/mZ9Rc8q8W3

## Citation

```bibtex
@misc{li2026clawsbenchevaluatingcapabilitysafety,
      title={ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspaces}, 
      author={Xiangyi Li and Kyoung Whan Choe and Yimin Liu and Xiaokun Chen and Chujun Tao and Bingran You and Wenbo Chen and Zonglin Di and Jiankai Sun and Shenghan Zheng and Jiajun Bao and Yuanli Wang and Weixiang Yan and Yiyuan Li and Han-chung Lee},
      year={2026},
      eprint={2604.05172},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2604.05172}, 
}
```
