Skip to content

Commit 2f8c2ab

Browse files
lyarwoodlyarwood
committed
test(kubevirt): Add results with initial toolset
Signed-off-by: Lee Yarwood <lyarwood@redhat.com>
1 parent 06c56f7 commit 2f8c2ab

17 files changed

+10096
-0
lines changed

‎pkg/toolsets/kubevirt/tests/results/gevals-claude-code-20251107-124320.log‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
Starting evaluation at Fri 7 Nov 12:43:20 GMT 2025...
2+
3+
4+
=== Starting Evaluation ===
5+
6+
Task: create-basic-vm
7+
Difficulty: easy
8+
→ Running agent...
9+
→ Verifying results...
10+
✓ Task passed
11+
12+
Task: create-ubuntu-vm
13+
Difficulty: easy
14+
→ Running agent...
15+
→ Verifying results...
16+
✓ Task passed
17+
18+
Task: create-vm-with-instancetype
19+
Difficulty: medium
20+
→ Running agent...
21+
→ Verifying results...
22+
✓ Task passed
23+
24+
Task: create-vm-with-performance
25+
Difficulty: medium
26+
→ Running agent...
27+
→ Verifying results...
28+
✓ Task passed
29+
30+
Task: create-vm-with-size
31+
Difficulty: medium
32+
→ Running agent...
33+
→ Verifying results...
34+
✓ Task passed
35+
36+
Task: troubleshoot-vm
37+
Difficulty: easy
38+
→ Running agent...
39+
→ Verifying results...
40+
✓ Task passed
41+
42+
=== Evaluation Complete ===
43+
44+
📄 Results saved to: gevals-kubevirt-vm-operations-out.json
45+
46+
=== Results Summary ===
47+
48+
Task: create-basic-vm
49+
Path: /home/lyarwood/redhat/devel/src/k8s/kubernetes-mcp-server/pkg/toolsets/kubevirt/tests/tasks/create-vm-basic/create-vm-basic.yaml
50+
Difficulty: easy
51+
Task Status: PASSED
52+
Assertions: PASSED (3/3)
53+
54+
Task: create-ubuntu-vm
55+
Path: /home/lyarwood/redhat/devel/src/k8s/kubernetes-mcp-server/pkg/toolsets/kubevirt/tests/tasks/create-vm-ubuntu/create-vm-ubuntu.yaml
56+
Difficulty: easy
57+
Task Status: PASSED
58+
Assertions: PASSED (3/3)
59+
60+
Task: create-vm-with-instancetype
61+
Path: /home/lyarwood/redhat/devel/src/k8s/kubernetes-mcp-server/pkg/toolsets/kubevirt/tests/tasks/create-vm-with-instancetype/create-vm-with-instancetype.yaml
62+
Difficulty: medium
63+
Task Status: PASSED
64+
Assertions: PASSED (3/3)
65+
66+
Task: create-vm-with-performance
67+
Path: /home/lyarwood/redhat/devel/src/k8s/kubernetes-mcp-server/pkg/toolsets/kubevirt/tests/tasks/create-vm-with-performance/create-vm-with-performance.yaml
68+
Difficulty: medium
69+
Task Status: PASSED
70+
Assertions: PASSED (3/3)
71+
72+
Task: create-vm-with-size
73+
Path: /home/lyarwood/redhat/devel/src/k8s/kubernetes-mcp-server/pkg/toolsets/kubevirt/tests/tasks/create-vm-with-size/create-vm-with-size.yaml
74+
Difficulty: medium
75+
Task Status: PASSED
76+
Assertions: PASSED (3/3)
77+
78+
Task: troubleshoot-vm
79+
Path: /home/lyarwood/redhat/devel/src/k8s/kubernetes-mcp-server/pkg/toolsets/kubevirt/tests/tasks/troubleshoot-vm/troubleshoot-vm.yaml
80+
Difficulty: easy
81+
Task Status: PASSED
82+
Assertions: PASSED (3/3)
83+
84+
=== Overall Statistics ===
85+
Total Tasks: 6
86+
Tasks Passed: 6/6
87+
Assertions Passed: 18/18
88+
89+
=== Statistics by Difficulty ===
90+
91+
easy:
92+
Tasks: 3/3
93+
Assertions: 9/9
94+
95+
medium:
96+
Tasks: 3/3
97+
Assertions: 9/9
98+
99+
SUCCESS: All tests passed
100+
Duration: 1m 53s (113s total)
101+
Generating view output from JSON...
102+
View output generation successful
103+
Moved output file to: /home/lyarwood/redhat/devel/src/k8s/kubernetes-mcp-server/pkg/toolsets/kubevirt/tests/results/gevals-with-toolset-claude-code-20251107-124320-out.json
104+
Moved view output to: /home/lyarwood/redhat/devel/src/k8s/kubernetes-mcp-server/pkg/toolsets/kubevirt/tests/results/gevals-with-toolset-claude-code-20251107-124320-out.log

‎pkg/toolsets/kubevirt/tests/results/gevals-gemini-20251107-124320.log‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
Starting evaluation at Fri 7 Nov 12:43:20 GMT 2025...
2+
3+
4+
=== Starting Evaluation ===
5+
6+
Task: create-basic-vm
7+
Difficulty: easy
8+
→ Running agent...
9+
→ Verifying results...
10+
✓ Task passed
11+
12+
Task: create-ubuntu-vm
13+
Difficulty: easy
14+
→ Running agent...
15+
→ Verifying results...
16+
✓ Task passed
17+
18+
Task: create-vm-with-instancetype
19+
Difficulty: medium
20+
→ Running agent...
21+
→ Verifying results...
22+
✓ Task passed
23+
24+
Task: create-vm-with-performance
25+
Difficulty: medium
26+
→ Running agent...
27+
→ Verifying results...
28+
~ Task passed but assertions failed
29+
30+
Task: create-vm-with-size
31+
Difficulty: medium
32+
→ Running agent...
33+
→ Verifying results...
34+
✓ Task passed
35+
36+
Task: troubleshoot-vm
37+
Difficulty: easy
38+
→ Running agent...
39+
→ Verifying results...
40+
~ Task passed but assertions failed
41+
42+
=== Evaluation Complete ===
43+
44+
📄 Results saved to: gevals-gemini-cli-kubernetes-basic-operations-out.json
45+
46+
=== Results Summary ===
47+
48+
Task: create-basic-vm
49+
Path: /home/lyarwood/redhat/devel/src/k8s/kubernetes-mcp-server/pkg/toolsets/kubevirt/tests/tasks/create-vm-basic/create-vm-basic.yaml
50+
Difficulty: easy
51+
Task Status: PASSED
52+
Assertions: PASSED (3/3)
53+
54+
Task: create-ubuntu-vm
55+
Path: /home/lyarwood/redhat/devel/src/k8s/kubernetes-mcp-server/pkg/toolsets/kubevirt/tests/tasks/create-vm-ubuntu/create-vm-ubuntu.yaml
56+
Difficulty: easy
57+
Task Status: PASSED
58+
Assertions: PASSED (3/3)
59+
60+
Task: create-vm-with-instancetype
61+
Path: /home/lyarwood/redhat/devel/src/k8s/kubernetes-mcp-server/pkg/toolsets/kubevirt/tests/tasks/create-vm-with-instancetype/create-vm-with-instancetype.yaml
62+
Difficulty: medium
63+
Task Status: PASSED
64+
Assertions: PASSED (3/3)
65+
66+
Task: create-vm-with-performance
67+
Path: /home/lyarwood/redhat/devel/src/k8s/kubernetes-mcp-server/pkg/toolsets/kubevirt/tests/tasks/create-vm-with-performance/create-vm-with-performance.yaml
68+
Difficulty: medium
69+
Task Status: PASSED
70+
Assertions: FAILED (2/3)
71+
- MaxToolCalls: Too many tool calls: expected <= 20, got 26
72+
73+
Task: create-vm-with-size
74+
Path: /home/lyarwood/redhat/devel/src/k8s/kubernetes-mcp-server/pkg/toolsets/kubevirt/tests/tasks/create-vm-with-size/create-vm-with-size.yaml
75+
Difficulty: medium
76+
Task Status: PASSED
77+
Assertions: PASSED (3/3)
78+
79+
Task: troubleshoot-vm
80+
Path: /home/lyarwood/redhat/devel/src/k8s/kubernetes-mcp-server/pkg/toolsets/kubevirt/tests/tasks/troubleshoot-vm/troubleshoot-vm.yaml
81+
Difficulty: easy
82+
Task Status: PASSED
83+
Assertions: FAILED (2/3)
84+
- MaxToolCalls: Too many tool calls: expected <= 20, got 38
85+
86+
=== Overall Statistics ===
87+
Total Tasks: 6
88+
Tasks Passed: 6/6
89+
Assertions Passed: 16/18
90+
91+
=== Statistics by Difficulty ===
92+
93+
easy:
94+
Tasks: 3/3
95+
Assertions: 8/9
96+
97+
medium:
98+
Tasks: 3/3
99+
Assertions: 8/9
100+
101+
SUCCESS: All tests passed
102+
Duration: 18m 55s (1135s total)
103+
Generating view output from JSON...
104+
View output generation successful
105+
Moved output file to: /home/lyarwood/redhat/devel/src/k8s/kubernetes-mcp-server/pkg/toolsets/kubevirt/tests/results/gevals-with-toolset-gemini-20251107-124320-out.json
106+
Moved view output to: /home/lyarwood/redhat/devel/src/k8s/kubernetes-mcp-server/pkg/toolsets/kubevirt/tests/results/gevals-with-toolset-gemini-20251107-124320-out.log

‎pkg/toolsets/kubevirt/tests/results/gevals-openai-agent-Granite-3.3-8B-Instruct-20251107-124320.log‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
Starting evaluation at Fri 7 Nov 12:43:20 GMT 2025...
2+
3+
4+
=== Starting Evaluation ===
5+
6+
Task: create-basic-vm
7+
Difficulty: easy
8+
→ Running agent...
9+
→ Verifying results...
10+
✓ Task passed
11+
12+
Task: create-ubuntu-vm
13+
Difficulty: easy
14+
→ Running agent...
15+
→ Verifying results...
16+
✓ Task passed
17+
18+
Task: create-vm-with-instancetype
19+
Difficulty: medium
20+
→ Running agent...
21+
→ Verifying results...
22+
✓ Task passed
23+
24+
Task: create-vm-with-performance
25+
Difficulty: medium
26+
→ Running agent...
27+
→ Verifying results...
28+
✓ Task passed
29+
30+
Task: create-vm-with-size
31+
Difficulty: medium
32+
→ Running agent...
33+
→ Verifying results...
34+
✓ Task passed
35+
36+
Task: troubleshoot-vm
37+
Difficulty: easy
38+
→ Running agent...
39+
→ Verifying results...
40+
✓ Task passed
41+
42+
=== Evaluation Complete ===
43+
44+
📄 Results saved to: gevals-openai-kubevirt-vm-operations-out.json
45+
46+
=== Results Summary ===
47+
48+
Task: create-basic-vm
49+
Path: /home/lyarwood/redhat/devel/src/k8s/kubernetes-mcp-server/pkg/toolsets/kubevirt/tests/tasks/create-vm-basic/create-vm-basic.yaml
50+
Difficulty: easy
51+
Task Status: PASSED
52+
Assertions: PASSED (3/3)
53+
54+
Task: create-ubuntu-vm
55+
Path: /home/lyarwood/redhat/devel/src/k8s/kubernetes-mcp-server/pkg/toolsets/kubevirt/tests/tasks/create-vm-ubuntu/create-vm-ubuntu.yaml
56+
Difficulty: easy
57+
Task Status: PASSED
58+
Assertions: PASSED (3/3)
59+
60+
Task: create-vm-with-instancetype
61+
Path: /home/lyarwood/redhat/devel/src/k8s/kubernetes-mcp-server/pkg/toolsets/kubevirt/tests/tasks/create-vm-with-instancetype/create-vm-with-instancetype.yaml
62+
Difficulty: medium
63+
Task Status: PASSED
64+
Assertions: PASSED (3/3)
65+
66+
Task: create-vm-with-performance
67+
Path: /home/lyarwood/redhat/devel/src/k8s/kubernetes-mcp-server/pkg/toolsets/kubevirt/tests/tasks/create-vm-with-performance/create-vm-with-performance.yaml
68+
Difficulty: medium
69+
Task Status: PASSED
70+
Assertions: PASSED (3/3)
71+
72+
Task: create-vm-with-size
73+
Path: /home/lyarwood/redhat/devel/src/k8s/kubernetes-mcp-server/pkg/toolsets/kubevirt/tests/tasks/create-vm-with-size/create-vm-with-size.yaml
74+
Difficulty: medium
75+
Task Status: PASSED
76+
Assertions: PASSED (3/3)
77+
78+
Task: troubleshoot-vm
79+
Path: /home/lyarwood/redhat/devel/src/k8s/kubernetes-mcp-server/pkg/toolsets/kubevirt/tests/tasks/troubleshoot-vm/troubleshoot-vm.yaml
80+
Difficulty: easy
81+
Task Status: PASSED
82+
Assertions: PASSED (3/3)
83+
84+
=== Overall Statistics ===
85+
Total Tasks: 6
86+
Tasks Passed: 6/6
87+
Assertions Passed: 18/18
88+
89+
=== Statistics by Difficulty ===
90+
91+
easy:
92+
Tasks: 3/3
93+
Assertions: 9/9
94+
95+
medium:
96+
Tasks: 3/3
97+
Assertions: 9/9
98+
99+
SUCCESS: All tests passed
100+
Duration: 3m 14s (194s total)
101+
Generating view output from JSON...
102+
View output generation successful
103+
Moved output file to: /home/lyarwood/redhat/devel/src/k8s/kubernetes-mcp-server/pkg/toolsets/kubevirt/tests/results/gevals-with-toolset-openai-agent-Granite-3.3-8B-Instruct-20251107-124320-out.json
104+
Moved view output to: /home/lyarwood/redhat/devel/src/k8s/kubernetes-mcp-server/pkg/toolsets/kubevirt/tests/results/gevals-with-toolset-openai-agent-Granite-3.3-8B-Instruct-20251107-124320-out.log

0 commit comments

Comments
 (0)