Skip to content

Commit 9aa2732

Browse files
lyarwoodlyarwood
committed
WIP test(kubevirt): Add gevals-based tests for VM toolset
Add comprehensive gevals tests for vm_create and vm_troubleshoot tools with 6 test scenarios covering basic VM creation, different workloads (Fedora, Ubuntu), instancetype selection, size hints, and performance families. These tests validate that AI agents can successfully discover and use the VM tools. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Lee Yarwood <lyarwood@redhat.com>
1 parent 79f691f commit 9aa2732

File tree

28 files changed

+483
-0
lines changed

28 files changed

+483
-0
lines changed

‎pkg/toolsets/kubevirt/vm/tests/README.md‎

Lines changed: 198 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,198 @@
1+
# KubeVirt VM Toolset Tests
2+
3+
This directory contains gevals-based tests for the KubeVirt VM toolset in the Kubernetes MCP Server.
4+
5+
## Overview
6+
7+
These tests validate the VM creation and troubleshooting tools (`vm_create` and `vm_troubleshoot`) by having AI agents complete real tasks using the MCP server.
8+
9+
## Test Structure
10+
11+
```
12+
tests/
13+
├── README.md # This file
14+
├── mcp-config.yaml # MCP server configuration
15+
├── claude-code/ # Claude Code agent configuration
16+
│ ├── agent.yaml
17+
│ └── eval.yaml
18+
└── tasks/ # Test tasks
19+
├── create-vm-basic/ # Basic VM creation test
20+
├── create-vm-with-instancetype/ # VM with specific instancetype
21+
├── create-vm-with-size/ # VM with size parameter
22+
├── create-vm-ubuntu/ # Ubuntu VM creation
23+
├── create-vm-with-performance/ # VM with performance family
24+
└── troubleshoot-vm/ # VM troubleshooting test
25+
```
26+
27+
## Prerequisites
28+
29+
1. **Kubernetes cluster** with KubeVirt installed
30+
- The cluster must have KubeVirt CRDs installed
31+
- For testing, you can use a Kind cluster with KubeVirt
32+
33+
2. **Kubernetes MCP Server** running at `http://localhost:8080/mcp`
34+
```bash
35+
# Build and run the server
36+
cd /path/to/kubernetes-mcp-server
37+
make build
38+
./kubernetes-mcp-server --http-port 8080
39+
```
40+
41+
3. **gevals binary** built from the gevals project
42+
```bash
43+
cd /path/to/gevals
44+
go build -o gevals ./cmd/gevals
45+
```
46+
47+
4. **Claude Code** installed and in PATH
48+
```bash
49+
# Install Claude Code (if not already installed)
50+
npm install -g @anthropic/claude-code
51+
```
52+
53+
5. **kubectl** configured to access your cluster
54+
55+
## Running the Tests
56+
57+
### Run All Tests
58+
59+
```bash
60+
# From the gevals directory
61+
./gevals eval /path/to/kubernetes-mcp-server/pkg/toolsets/kubevirt/vm/tests/claude-code/eval.yaml
62+
```
63+
64+
### Run a Specific Test
65+
66+
```bash
67+
# Run just the basic VM creation test
68+
./gevals eval /path/to/kubernetes-mcp-server/pkg/toolsets/kubevirt/vm/tests/tasks/create-vm-basic/create-vm-basic.yaml \
69+
--agent-file /path/to/kubernetes-mcp-server/pkg/toolsets/kubevirt/vm/tests/claude-code/agent.yaml \
70+
--mcp-config-file /path/to/kubernetes-mcp-server/pkg/toolsets/kubevirt/vm/tests/mcp-config.yaml
71+
```
72+
73+
## Test Descriptions
74+
75+
### create-vm-basic
76+
**Difficulty:** Easy
77+
**Description:** Tests basic VM creation with default Fedora workload.
78+
**Key Tool:** `vm_create`
79+
**Expected Behavior:** Agent should use `vm_create` to generate a plan and then create the VM using `resources_create_or_update`.
80+
81+
### create-vm-with-instancetype
82+
**Difficulty:** Medium
83+
**Description:** Tests VM creation with a specific instancetype (u1.medium).
84+
**Key Tool:** `vm_create`
85+
**Expected Behavior:** Agent should pass the instancetype parameter to `vm_create` and create a VM with the correct instancetype reference.
86+
87+
### create-vm-with-size
88+
**Difficulty:** Medium
89+
**Description:** Tests VM creation using a size hint ('large').
90+
**Key Tool:** `vm_create`
91+
**Expected Behavior:** Agent should use the size parameter which should map to an appropriate instancetype.
92+
93+
### create-vm-ubuntu
94+
**Difficulty:** Easy
95+
**Description:** Tests VM creation with Ubuntu workload.
96+
**Key Tool:** `vm_create`
97+
**Expected Behavior:** Agent should create a VM using the Ubuntu container disk image.
98+
99+
### create-vm-with-performance
100+
**Difficulty:** Medium
101+
**Description:** Tests VM creation with performance family ('compute-optimized') and size.
102+
**Key Tool:** `vm_create`
103+
**Expected Behavior:** Agent should combine performance and size to select an appropriate instancetype (e.g., c1.medium).
104+
105+
### troubleshoot-vm
106+
**Difficulty:** Easy
107+
**Description:** Tests VM troubleshooting guide generation.
108+
**Key Tool:** `vm_troubleshoot`
109+
**Expected Behavior:** Agent should use `vm_troubleshoot` to generate a troubleshooting guide for the VM.
110+
111+
## Assertions
112+
113+
The tests validate:
114+
115+
- **Tool Usage:** Agents must call `vm_create`, `vm_troubleshoot`, or `resources_*` tools
116+
- **Call Limits:** Between 1 and 30 tool calls (allows for exploration and creation)
117+
- **Task Success:** Verification scripts confirm VMs are created correctly
118+
119+
## Expected Results
120+
121+
**✅ Pass** means:
122+
- The VM tools are well-designed and discoverable
123+
- Tool descriptions are clear to AI agents
124+
- Schemas are properly structured
125+
- Implementation works correctly
126+
127+
**❌ Fail** indicates:
128+
- Tool descriptions may need improvement
129+
- Schema complexity issues
130+
- Missing functionality
131+
- Implementation bugs
132+
133+
## Output
134+
135+
Results are saved to `gevals-kubevirt-vm-operations-out.json` with:
136+
- Task pass/fail status
137+
- Assertion results
138+
- Tool call history
139+
- Agent interactions
140+
141+
## Customization
142+
143+
### Using Different AI Agents
144+
145+
You can create additional agent configurations (similar to the `claude-code/` directory) for testing with different AI models:
146+
147+
```yaml
148+
# Example: openai-agent/agent.yaml
149+
kind: Agent
150+
metadata:
151+
name: "openai-agent"
152+
commands:
153+
argTemplateMcpServer: "{{ .File }}"
154+
runPrompt: |-
155+
agent-wrapper.sh {{ .McpServerFileArgs }} "{{ .Prompt }}"
156+
```
157+
158+
### Adding New Tests
159+
160+
To add a new test task:
161+
162+
1. Create a new directory under `tasks/`
163+
2. Add task YAML file with prompt
164+
3. Add setup, verify, and cleanup scripts
165+
4. The test will be automatically discovered by the glob pattern in `eval.yaml`
166+
167+
## Troubleshooting
168+
169+
### Tests Fail to Connect to MCP Server
170+
171+
Ensure the Kubernetes MCP Server is running:
172+
```bash
173+
curl http://localhost:8080/mcp/health
174+
```
175+
176+
### VirtualMachine Not Created
177+
178+
Check if KubeVirt is installed:
179+
```bash
180+
kubectl get crds | grep kubevirt
181+
kubectl get pods -n kubevirt
182+
```
183+
184+
### Permission Issues
185+
186+
Ensure your kubeconfig has permissions to:
187+
- Create namespaces
188+
- Create VirtualMachine resources
189+
- List instancetypes and preferences
190+
191+
## Contributing
192+
193+
When adding new tests:
194+
- Keep tasks focused on a single capability
195+
- Make verification scripts robust
196+
- Document expected behavior
197+
- Set appropriate difficulty levels
198+
- Ensure cleanup scripts remove all resources

‎pkg/toolsets/kubevirt/vm/tests/claude-code/agent.yaml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
kind: Agent
2+
metadata:
3+
name: "claude-code"
4+
commands:
5+
useVirtualHome: false
6+
argTemplateMcpServer: "--mcp-config {{ .File }}"
7+
argTemplateAllowedTools: "mcp__{{ .ServerName }}__{{ .ToolName }}"
8+
allowedToolsJoinSeparator: ","
9+
runPrompt: |-
10+
claude {{ .McpServerFileArgs }} --strict-mcp-config --allowedTools "{{ .AllowedToolArgs }}" --print "{{ .Prompt }}"

‎pkg/toolsets/kubevirt/vm/tests/claude-code/eval.yaml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
kind: Eval
2+
metadata:
3+
name: "kubevirt-vm-operations"
4+
config:
5+
agentFile: agent.yaml
6+
mcpConfigFile: ../mcp-config.yaml
7+
llmJudge:
8+
env:
9+
baseUrlKey: JUDGE_BASE_URL
10+
apiKeyKey: JUDGE_API_KEY
11+
modelNameKey: JUDGE_MODEL_NAME
12+
taskSets:
13+
- glob: ../tasks/*/*.yaml
14+
assertions:
15+
toolsUsed:
16+
- server: kubernetes
17+
toolPattern: "(vm_create|vm_troubleshoot|resources_.*)"
18+
minToolCalls: 1
19+
maxToolCalls: 30

‎pkg/toolsets/kubevirt/vm/tests/mcp-config.yaml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
mcpServers:
2+
kubernetes:
3+
type: http
4+
url: http://localhost:8080/mcp
5+
enableAllTools: true

‎pkg/toolsets/kubevirt/vm/tests/tasks/create-vm-basic/cleanup.sh‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
#!/usr/bin/env bash
2+
kubectl delete virtualmachine test-vm -n vm-test --ignore-not-found
3+
kubectl delete namespace vm-test --ignore-not-found

‎pkg/toolsets/kubevirt/vm/tests/tasks/create-vm-basic/create-vm-basic.yaml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
kind: Task
2+
metadata:
3+
name: "create-basic-vm"
4+
difficulty: easy
5+
steps:
6+
setup:
7+
file: setup.sh
8+
verify:
9+
file: verify.sh
10+
cleanup:
11+
file: cleanup.sh
12+
prompt:
13+
inline: Please create a Fedora virtual machine named test-vm in the vm-test namespace. Use the vm_create tool to generate the creation plan and then create the VirtualMachine resource.

‎pkg/toolsets/kubevirt/vm/tests/tasks/create-vm-basic/setup.sh‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
#!/usr/bin/env bash
2+
kubectl delete namespace vm-test --ignore-not-found
3+
kubectl create namespace vm-test

‎pkg/toolsets/kubevirt/vm/tests/tasks/create-vm-basic/verify.sh‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
#!/usr/bin/env bash
2+
# Wait for VirtualMachine to be created
3+
if kubectl wait --for=jsonpath='{.metadata.name}'=test-vm virtualmachine/test-vm -n vm-test --timeout=30s 2>/dev/null; then
4+
echo "VirtualMachine test-vm created successfully"
5+
exit 0
6+
else
7+
echo "VirtualMachine test-vm not found"
8+
kubectl get virtualmachines -n vm-test
9+
exit 1
10+
fi

‎pkg/toolsets/kubevirt/vm/tests/tasks/create-vm-ubuntu/cleanup.sh‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
#!/usr/bin/env bash
2+
kubectl delete virtualmachine ubuntu-vm -n vm-test --ignore-not-found
3+
kubectl delete namespace vm-test --ignore-not-found

‎pkg/toolsets/kubevirt/vm/tests/tasks/create-vm-ubuntu/create-vm-ubuntu.yaml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
kind: Task
2+
metadata:
3+
name: "create-ubuntu-vm"
4+
difficulty: easy
5+
steps:
6+
setup:
7+
file: setup.sh
8+
verify:
9+
file: verify.sh
10+
cleanup:
11+
file: cleanup.sh
12+
prompt:
13+
inline: Create an Ubuntu virtual machine named ubuntu-vm in the vm-test namespace. Use the vm_create tool to generate the creation plan and then create the VirtualMachine resource.

0 commit comments

Comments
 (0)