Skip to content

Commit 1821a6b

Browse files
NiloCKNiloCK
committed
add working docs...
partly out of date atm, but worth keeping
1 parent ee2c302 commit 1821a6b

File tree

2 files changed

+327
-0
lines changed

2 files changed

+327
-0
lines changed

‎agent/a.4.orchestration-and-prompts.md‎

Lines changed: 189 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,189 @@
1+
# Assessment: Orchestration & Prompts for MCP Content Mining
2+
3+
## Context & Requirements Synthesis
4+
5+
### User Priorities Identified
6+
1. **Two Primary Mining Modes**:
7+
- **Greenfield content mining** - Starting from scratch with new source material
8+
- **Diff content mining** - Analyzing changes/updates to existing content
9+
2. **Rough ELO estimation** - Platform handles real-time convergence, agents provide sane initial sorting
10+
3. **Leverage existing patterns** from `@packages/edit-ui/src/components/BulkImportView.vue`
11+
4. **Atomic tool composition** - Agents orchestrate using create_card, update_card, etc.
12+
5. **Flexible guidance** - Prompts suggest strategies without rigid constraints
13+
14+
### Existing Bulk Import Knowledge
15+
From `/agent/mcp/prompt-bulk.md`, the platform already has:
16+
- **Detailed fillIn formatting rules** - `{{answer}}`, `{{multiple|options}}`, `{{correct||distractor1|distractor2}}`
17+
- **ELO calibration guidelines** - 100-500 beginner, 500-1000 early, 1000-1500 intermediate, etc.
18+
- **Tag formatting conventions** - no spaces, dots/hyphens allowed, comma-separated
19+
- **Card separation syntax** - double `---` lines
20+
- **Markdown support** - code blocks, formatting within questions and answers
21+
- **Content best practices** - concise answers, clear questions, code examples
22+
23+
## MCP Prompts vs Tools Analysis
24+
25+
### Why Prompts Are Superior for Content Mining
26+
27+
**MCP Prompts provide:**
28+
- ✅ **Expert guidance** without execution constraints
29+
- ✅ **Parameter customization** for different contexts
30+
- ✅ **Agent autonomy** - can adapt, combine, or ignore advice
31+
- ✅ **Composability** - work with agent's own intelligence
32+
- ✅ **Maintainability** - easy to update expertise without code changes
33+
34+
**Compared to orchestrator tools which would:**
35+
- ❌ **Lock in rigid workflows** that may not fit all scenarios
36+
- ❌ **Hide decision-making** from agent reasoning
37+
- ❌ **Create debugging complexity** with nested tool calls
38+
- ❌ **Limit creativity** with predefined patterns
39+
40+
### Agent Decision Process for Prompts
41+
42+
**Agent prompt selection workflow:**
43+
1. **User request analysis** - "Generate quizzes from this Go codebase"
44+
2. **Available prompts discovery** - Lists MCP prompts via protocol
45+
3. **Context matching** - Identifies this as greenfield mining scenario
46+
4. **Prompt invocation** - Calls `greenfield_content_mining` with repo context
47+
5. **Guidance integration** - Synthesizes prompt advice with own reasoning
48+
6. **Tool orchestration** - Uses create_card, tag_card atomically following strategy
49+
50+
## Proposed MCP Prompt Architecture
51+
52+
### Core Prompt Templates (4 Essential)
53+
54+
#### 1. `greenfield_content_mining`
55+
**Purpose:** Guide systematic analysis of new source material for comprehensive courseware creation
56+
57+
**Parameters:**
58+
- `sourceType` - "codebase", "documentation", "tutorials", "papers"
59+
- `domain` - "programming", "math", "science", etc.
60+
- `targetAudience` - "beginner", "intermediate", "advanced"
61+
- `scopeConstraint` - "single-file", "module", "full-project"
62+
63+
**Template provides:**
64+
- **Repository structure analysis** - How to identify key concepts vs implementation details
65+
- **Content prioritization** - Core concepts first, edge cases later
66+
- **Knowledge dependency mapping** - Prerequisites and logical sequencing
67+
- **Coverage strategies** - Breadth vs depth decisions
68+
- **ELO estimation guidance** - Initial difficulty assessment based on concept complexity
69+
70+
#### 2. `diff_content_mining`
71+
**Purpose:** Guide analysis of changes/updates to identify new quiz opportunities
72+
73+
**Parameters:**
74+
- `changeType` - "feature-addition", "refactor", "bug-fix", "docs-update"
75+
- `diffScope` - "lines-changed", "files-affected", "concepts-modified"
76+
- `existingCoverage` - "sparse", "moderate", "comprehensive"
77+
78+
**Template provides:**
79+
- **Change impact analysis** - What concepts are newly introduced vs modified
80+
- **Gap identification** - What existing cards need updates vs new creation
81+
- **Incremental strategies** - How to build on existing content
82+
- **Deprecation handling** - Managing outdated concepts
83+
- **Version tracking** - Linking content to specific code versions
84+
85+
#### 3. `fillIn_question_crafting`
86+
**Purpose:** Guide creation of effective fill-in-the-blank questions using Vue-Skuilder syntax
87+
88+
**Parameters:**
89+
- `contentType` - "code", "theory", "api-usage", "troubleshooting"
90+
- `cognitiveLevel` - "recall", "application", "analysis", "synthesis"
91+
- `answerComplexity` - "single-word", "short-phrase", "code-snippet"
92+
93+
**Template provides:**
94+
- **Format patterns** from bulk import experience:
95+
- `{{simple_answer}}` for direct recall
96+
- `{{option1|option2}}` for multiple acceptable answers
97+
- `{{correct||distractor1|distractor2}}` for multiple choice
98+
- **Code integration best practices** - Balancing context with focused questions
99+
- **Answer entropy management** - Keeping answers matchable while meaningful
100+
- **Distractor selection** - Common mistakes and plausible alternatives
101+
102+
#### 4. `elo_rough_calibration`
103+
**Purpose:** Guide initial ELO estimation knowing real-world performance will refine
104+
105+
**Parameters:**
106+
- `conceptComplexity` - "fundamental", "applied", "nuanced", "edge-case"
107+
- `prerequisiteDepth` - "none", "basic", "substantial", "advanced"
108+
- `cognitiveLoad` - "recall", "comprehension", "application", "analysis"
109+
110+
**Template provides:**
111+
- **Initial ELO bands** refined from bulk import guidelines:
112+
- 100-500: Basic vocabulary, simple facts
113+
- 500-1000: Applied knowledge, straightforward procedures
114+
- 1000-1500: Integration concepts, moderate problem-solving
115+
- 1500-2000: Complex applications, multi-step reasoning
116+
- 2000+: Expert insights, edge cases, advanced synthesis
117+
- **Comparative benchmarking** - How to assess relative to existing content
118+
- **Uncertainty handling** - When to err conservative vs aggressive
119+
- **Refinement expectations** - Understanding platform will auto-adjust
120+
121+
## Prompt Implementation Strategy
122+
123+
### MCP Technical Integration
124+
125+
**Prompt registration pattern:**
126+
```typescript
127+
this.mcpServer.registerPrompt(
128+
'greenfield_content_mining',
129+
{
130+
title: 'Greenfield Content Mining Strategy',
131+
description: 'Systematic approach for mining new source material',
132+
arguments: [
133+
{ name: 'sourceType', description: 'Type of source material being analyzed' },
134+
{ name: 'domain', description: 'Subject domain of the content' },
135+
{ name: 'targetAudience', description: 'Intended learning audience level' },
136+
{ name: 'scopeConstraint', description: 'Scope of analysis (file, module, project)' }
137+
]
138+
},
139+
async (args) => ({
140+
messages: [
141+
{
142+
role: 'user',
143+
content: generateGreenfield ContentMiningPrompt(args)
144+
}
145+
]
146+
})
147+
);
148+
```
149+
150+
### Content Strategy
151+
152+
**Prompts should contain:**
153+
- ✅ **Structured methodologies** - Step-by-step approaches
154+
- ✅ **Decision frameworks** - When to choose different strategies
155+
- ✅ **Quality criteria** - What makes good vs poor questions
156+
- ✅ **Practical examples** - Concrete patterns for different content types
157+
- ✅ **Vue-Skuilder specific** - Leverage platform features and conventions
158+
159+
**Prompts should NOT contain:**
160+
- ❌ **Rigid scripts** - Agents need flexibility
161+
- ❌ **Implementation details** - Keep focused on strategy
162+
- ❌ **Tool invocation** - Agents decide when to use create_card, etc.
163+
- ❌ **Fixed sequences** - Allow for creative orchestration
164+
165+
## Organizational Benefits
166+
167+
### Separation of Concerns
168+
- **Prompts**: Domain expertise and strategy guidance
169+
- **Agent**: Intelligence, creativity, and orchestration
170+
- **Tools**: Atomic operations (create, update, tag, delete)
171+
- **Platform**: Real-time ELO convergence and performance tracking
172+
173+
### Maintainability
174+
- **Prompt updates** don't require code changes
175+
- **Strategy evolution** can be managed independently
176+
- **Domain expertise** centralized and reusable
177+
- **Testing simplified** - prompts testable in isolation
178+
179+
### Scalability
180+
- **New domains** just need new prompt templates
181+
- **Different agent types** can use same prompt guidance
182+
- **Cross-platform portability** via MCP standard
183+
- **Expertise sharing** across agent implementations
184+
185+
## Recommendation
186+
187+
Implement the 4 core prompt templates as Phase 2.2, replacing the complex orchestrator tool approach. This provides the perfect balance of expert guidance with agent autonomy, leveraging existing Vue-Skuilder patterns while maintaining full flexibility for creative content mining strategies.
188+
189+
The agent can then choose appropriate prompts based on context, synthesize the guidance with their own reasoning, and execute via the atomic tools we've already implemented.

‎agent/mcp/prompt-bulk.md‎

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
## Guide for Authoring Bulk Flashcard Content (`fillIn` Type)
2+
3+
This guide explains how to format plain text for bulk import of flashcards. The system processes this text to create individual flashcards with questions, answers, and tags. Each card will be of the "fill-in-the-blank" or "multiple choice" style.
4+
5+
**General Format Rules:**
6+
7+
1. **Card Separation**:
8+
* Each card's content must be separated from the next by **two consecutive lines, each containing exactly three hyphens (`---`)**.
9+
* Example:
10+
```text
11+
Content for Card 1...
12+
tags: example
13+
---
14+
---
15+
Content for Card 2...
16+
```
17+
18+
2. **Tags and ELO Metadata**:
19+
* Tags are optional.
20+
* Tags do not contain spaces, but can use dots or hyphens.
21+
* If included, they **must** be on one of the **last lines** of a card's content block (before the `---` separator).
22+
* The line must start with `tags:` (case-insensitive) followed by a comma-separated list of tags.
23+
* Example: `tags: javascript, programming, web-development`
24+
* If this line is omitted, the card will have no tags.
25+
* You can also include an ELO rating by adding a line with `elo: [integer]` (case-insensitive).
26+
* The ELO rating indicates difficulty level, using a chess-like rating scale:
27+
* 100-500: Beginner level, basic knowledge
28+
* 500-1000: Some knowledge, early understanding
29+
* 1000-1500: Intermediate knowledge
30+
* 1500-2000: Advanced understanding
31+
* 2000-2400: Expert level
32+
* 2400+: Elite/master level
33+
* Example: `elo: 1250` or `tags: go, functions` followed by `elo: 850`
34+
35+
3. **Plain Text Only**:
36+
* The bulk input is for plain text markdown content only. Do not attempt to include or reference image/audio uploads directly in this bulk format.
37+
38+
**Content Formatting for `fillIn` Questions:**
39+
40+
The main content of your card is markdown. Special interactive elements (blanks or multiple-choice options) are defined using double curly braces: `{{ ... }}`. The content inside the curly braces can also contain markdown formatting, including code blocks.
41+
42+
1. **Basic Fill-in-the-Blank (Single Answer)**:
43+
* Use `{{answer}}` where the blank should appear in your question.
44+
* The text inside the `{{ }}` is the exact expected answer.
45+
* **Example Card**:
46+
```text
47+
The capital of France is {{Paris}}.
48+
tags: geography, europe
49+
```
50+
51+
2. **Fill-in-the-Blank (Multiple Accepted Answers)**:
52+
* If multiple text inputs are acceptable for a single blank, separate them with a pipe `|` character inside the `{{ }}`.
53+
* The user only needs to type one of these to be correct.
54+
* **Example Card**:
55+
```text
56+
A common way to declare a variable in JavaScript that can be reassigned is using the {{let|var}} keyword.
57+
tags: javascript, programming
58+
```
59+
*(Here, both "let" and "var" would be accepted as correct for the blank).*
60+
61+
3. **Multiple Choice Questions (Radio Buttons)**:
62+
* This format also uses the `{{ ... }}` syntax but with a specific internal structure to define correct answers and distractors.
63+
* The structure is: `{{correct_answer1|correct_answer2||distractor1|distractor2}}`
64+
* **Correct Answers**: List one or more correct answers first, separated by `|`.
65+
* **Separator**: Use a double pipe `||` to separate the list of correct answers from the list of distractors.
66+
* **Distractors**: List one or more incorrect answers (distractors) after the `||`, separated by `|`.
67+
* The system will automatically create a multiple-choice question. It will pick *one* of your specified correct answers and display it along with *all* the specified distractors, shuffled, as radio button options.
68+
* **Example Card**:
69+
```text
70+
Which of the following is a primary color?
71+
{{Red|Blue|Yellow||Green|Orange|Purple}}
72+
tags: art, colors, multiple choice
73+
```
74+
*(In this example, "Red", "Blue", and "Yellow" are all correct. The system might generate a question with choices like "Blue", "Green", "Orange", "Purple" (shuffled). If the user selects "Blue", they are correct.)*
75+
76+
* **Example Card (Single Correct Answer for Multiple Choice)**:
77+
```text
78+
What is the main gas found in the air we breathe?
79+
{{Nitrogen||Oxygen|Carbon Dioxide|Hydrogen}}
80+
tags: science, chemistry, atmosphere
81+
```
82+
*(Here, only "Nitrogen" is correct. The choices presented would be "Nitrogen", "Oxygen", "Carbon Dioxide", "Hydrogen" (shuffled).)*
83+
84+
**Summary of `{{ ... }}` Syntax:**
85+
86+
* `{{answer}}`: Simple blank, one correct string (can include markdown).
87+
* `{{ans1|ans2}}`: Simple blank, multiple accepted strings (user types one).
88+
* `{{correctAns1|correctAns2||distractor1|distractor2}}`: Multiple choice. One of the `correctAns` will be shown with all `distractors`.
89+
* All answers can contain markdown, including code blocks, lists, and other formatting.
90+
91+
**Complete Example of a Small Bulk Input:**
92+
93+
```text
94+
What is the chemical symbol for water?
95+
{{H2O|HOH}}
96+
tags: chemistry, science
97+
elo: 500
98+
99+
---
100+
---
101+
102+
Which planet is known as the Red Planet?
103+
{{Mars||Jupiter|Saturn|Venus}}
104+
tags: astronomy, solar system, multiple choice
105+
elo: 750
106+
107+
---
108+
---
109+
110+
The `typeof` operator in JavaScript returns a ____ indicating the type of the unevaluated operand.
111+
{{string}}
112+
elo: 1250
113+
tags: javascript, programming, operators
114+
```
115+
116+
**Tips for Authors:**
117+
118+
* **Clarity is Key**: Ensure your questions are clear and the blanks/choices are unambiguous.
119+
* **Markdown**: Standard markdown can be used for formatting the question text (e.g., bold, italics, lists), but keep it simple.
120+
* **Code Blocks in Questions**: Do not use the `{{ ... }}` syntax inside code blocks (```). If you need to indicate a blank in code, place the `{{ ... }}` outside the code block and make it clear what should be filled in.
121+
* **Code Examples in Questions**: It can be very useful to include code blocks in the main body of your question, then ask about that code. For example:
122+
```go
123+
func main() {
124+
x := 10
125+
fmt.Println(x)
126+
}
127+
```
128+
The above program will {{compile and run correctly||fail at compilation|fail at runtime}}
129+
* **Code Blocks in Answers**: Code blocks ARE allowed inside the `{{ ... }}` syntax as part of the answer. For example:
130+
131+
{{```go
132+
func main() {
133+
fmt.Println("Hello")
134+
}
135+
```}}
136+
* **Keep Answers Concise**: Avoid very long answers in fill-in-the-blank questions. Long answers have high entropy, making it difficult for users to match the expected answer even if they understand the concept. Prefer short, specific answers (1-3 words is ideal) or use multiple-choice for complex answers.
137+
* **Review `{{ ... }}` Carefully**: Typos within the `{{ }}` syntax, especially with the `|` and `||` separators, can lead to incorrect parsing.
138+
* **Test Small Batches**: If you're unsure, try importing a small batch of 2-3 cards first to verify they are parsed as expected.

0 commit comments

Comments
 (0)