Merge pull request #1 from metauto-ai/release/initialize

release agent-as-a-judge

Author: mczhuge

.config/pre-commit-config.yaml

@@ -0,0 +1,15 @@
+repos:
+- repo: https://github.com/pre-commit/pre-commit-hooks
+  rev: v4.0.1
+  hooks:
+    - id: trailing-whitespace
+    - id: end-of-file-fixer
+    - id: check-yaml
+    - id: check-json
+- repo: https://github.com/pre-commit/mirrors-mypy
+  rev: v0.910
+  hooks:
+    - id: mypy
+      additional_dependencies: ['types-termcolor']
+      language: python
+      entry: poetry run mypy

.env.sample

@@ -0,0 +1,3 @@
+DEFAULT_LLM="gpt-4o-2024-08-06"
+OPENAI_API_KEY="sk-***"
+PROJECT_DIR="{PATH_TO_THIS_PROJECT}"

.github/.codecov.yml

@@ -0,0 +1,15 @@
+codecov:
+  notify:
+    wait_for_ci: true
+
+coverage:
+  status:
+    patch:
+      default:
+        threshold: 100% 
+    project:
+      default:
+        threshold: 5%
+comment: false
+github_checks:
+    annotations: false

.github/dependabot.yml

@@ -0,0 +1,11 @@
+version: 2
+updates:
+  - package-ecosystem: "pip"
+    directory: "/"
+    schedule:
+      interval: "weekly"
+    open-pull-requests-limit: 5
+    assignees:
+      - mczhuge
+    labels:
+      - "dependencies"

.gitignore

@@ -106,10 +106,8 @@ ipython_config.py
 #pdm.lock
 #   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
 #   in version control.
-#   https://pdm.fming.dev/latest/usage/project/#working-with-version-control
+#   https://pdm.fming.dev/#use-with-ide
 .pdm.toml
-.pdm-python
-.pdm-build/
 
 # PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
 __pypackages__/
@@ -160,3 +158,15 @@ cython_debug/
 #  and can be added to the global gitignore or merged into this file.  For a more nuclear
 #  option (not recommended) you can uncomment the following to ignore the entire idea folder.
 #.idea/
+
+# log
+workspace/
+first_test/
+
+# Cache
+Dockerfile
+.DS_Store
+
+
+# pycache
+*/__pycache__/

README.md

@@ -1,2 +1,115 @@
-# agent-as-a-judge
-🤠 Agent-as-a-Judge and DevAI dataset
+<div align="center">
+    <h1 align="center">Agents Evaluate Agents</h1>
+    <img src="assets/devai_logo.png" alt="DevAI Logo" width="150" height="150">
+    <p align="center">
+        <a href="https://devai.tech"><b>Project</b></a> | 
+        <a href="https://huggingface.co/DEVAI-benchmark"><b>Dataset</b></a> | 
+        <a href="https://arxiv.org/pdf/2410.10934"><b>Paper</b></a> 
+    </p>
+</div>
+
+> [!NOTE]
+> Current evaluation techniques are often inadequate for advanced **agentic systems** due to their focus on final outcomes and labor-intensive manual reviews. To overcome this limitation, we introduce the **Agent-as-a-Judge** framework. 
+>
+
+## 🤠 Features
+
+Agent-as-a-Judge offers two key advantages:
+
+- **Automated Evaluation**: Agent-as-a-Judge can evaluate tasks during or after execution, saving 97.72% of time and 97.64% of costs compared to human experts.
+- **Provide Reward Signals**: It provides continuous, step-by-step feedback that can be used as reward signals for further agentic training and improvement.
+
+<div align="center">
+    <img src="assets/demo.gif" alt="Demo GIF" style="width: 100%; max-width: 650px;">
+</div>
+<div align="center">
+    <img src="assets/judge_first.png" alt="AaaJ" style="width: 95%; max-width: 650px;">
+</div>
+
+
+
+## 🎮 Quick Start 
+
+### 1. install
+
+```python
+git clone https://github.com/metauto-ai/agent-as-a-judge.git
+cd agent-as-a-judge/
+conda create -n aaaj python=3.11
+conda activate aaaj
+pip install poetry
+poetry install
+```
+
+### 2. set LLM&API
+
+Before running, rename `.env.sample` to `.env` and fill in the **required APIs and Settings** in the main repo folder to support LLM calling. The `LiteLLM` tool supports various LLMs.
+
+### 3. run 
+
+> [!TIP]
+> See more comprehensive [usage scripts](scripts/README.md).
+>
+
+
+#### Usage A: **Ask Anything** for Any Workspace:
+
+```python 
+
+PYTHONPATH=. python scripts/run_ask.py \
+  --workspace $(pwd)/benchmark/workspaces/OpenHands/39_Drug_Response_Prediction_SVM_GDSC_ML \
+  --question "What does this workspace contain?"
+```
+
+You can find an [example](assets/ask_sample.md) to see how **Ask Anything** works.
+
+
+#### Usage B: **Agent-as-a-Judge** for **DevAI**
+
+
+```python
+
+PYTHONPATH=. python scripts/run_aaaj.py \
+  --developer_agent "OpenHands" \
+  --setting "black_box" \
+  --planning "efficient (no planning)" \
+  --benchmark_dir $(pwd)/benchmark
+```
+
+💡 There is an [example](assets/aaaj_sample.md) that shows the process of how **Agent-as-a-Judge** collects evidence for judging.
+
+
+
+## 🤗 DevAI Dataset 
+
+
+
+<div align="center">
+    <img src="assets/dataset.png" alt="Dataset" style="width: 100%; max-width: 600px;">
+</div>
+
+> [!IMPORTANT]
+> As a **proof-of-concept**, we applied **Agent-as-a-Judge** to code generation tasks using **DevAI**, a benchmark consisting of 55 realistic AI development tasks with 365 hierarchical user requirements. The results demonstrate that **Agent-as-a-Judge** significantly outperforms traditional evaluation methods, delivering reliable reward signals for scalable self-improvement in agentic systems.
+> 
+> Check out the dataset on [Hugging Face 🤗](https://huggingface.co/DEVAI-benchmark).
+> See how to use this dataset in the [guidelines](benchmark/devai/README.md).
+
+
+<!-- <div align="center">
+    <img src="assets/sample.jpeg" alt="Sample" style="width: 100%; max-width: 600px;">
+</div> -->
+
+## Reference
+
+Feel free to cite if you find the Agent-as-a-Judge concept useful for your work:
+
+```
+@article{zhuge2024agent,
+  title={Agent-as-a-Judge: Evaluate Agents with Agents},
+  author={Zhuge, Mingchen and Zhao, Changsheng and Ashley, Dylan and Wang, Wenyi and Khizbullin, Dmitrii and Xiong, Yunyang and Liu, Zechun and Chang, Ernie and Krishnamoorthi, Raghuraman and Tian, Yuandong and Shi, Yangyang and Chandra, Vikas and Schmidhuber, J{\"u}rgen},
+  journal={arXiv preprint arXiv:2410.10934},
+  year={2024}
+}
+```
+
+

agent_as_a_judge/__init__.py

@@ -0,0 +1,4 @@
+from .llm.provider import LLM
+from .llm.cost import Cost
+
+__all__ = ["LLM", "Cost"]