How to use
- Copy the installation command and review the source package before installing it in your own agent environment.
- Check the source document for required tools, credentials and supported hosts.
- Describe your input and desired deliverable; review the output against the source material.
npx skills add https://github.com/majesticlabs-dev/majestic-marketplace --skill skill-evalBefore you start
- Installation does not include a model subscription, API allowance or permission to access your files.
- Repository stars and installation counts describe activity, not an independent quality or security test.
Original skill document
Rendered from the source repository. Headings, examples, code, tables, links, and referenced images are preserved.
Skill Eval
Test a skill with parallel eval runs and iterate based on results.
Input
SKILL_PATH = $ARGUMENTSIf empty: AskUserQuestion("Which skill do you want to test? Provide the path.")
Workflow
Step 1: Validate Skill Exists
If not exists(SKILL_PATH/SKILL.md):
ERROR "No SKILL.md found at SKILL_PATH"
SKILL_CONTENT = Read(SKILL_PATH/SKILL.md)
SKILL_NAME = extract name from frontmatter
SKILL_DESCRIPTION = extract description from frontmatterStep 2: Load or Create Evals
EVALS_PATH = SKILL_PATH/evals/evals.json
If exists(EVALS_PATH):
EVALS = Read(EVALS_PATH)
Present evals to user, ask: "Use these evals, modify, or create new ones?"
Else:
Generate 2-3 realistic test prompts based on SKILL_CONTENT
Each prompt should be what a real user would actually say
Include enough detail: file paths, context, specific requests
Present to user: "Here are test cases I'd like to try. Look right, or want changes?"Wait for user confirmation before proceeding.
Eval format:
json
{
"skill_name": "SKILL_NAME",
"evals": [
{
"id": 1,
"prompt": "User's realistic task prompt",
"expected_output": "Description of expected result",
"expectations": [
"The output includes X",
"The skill used approach Y"
]
}
]
}Save confirmed evals to EVALS_PATH.
Step 3: Set Up Workspace
WORKSPACE = SKILL_PATH-workspace
ITERATION = find highest iteration-N in WORKSPACE + 1, or 1 if none
ITER_DIR = WORKSPACE/iteration-ITERATION
mkdir -p ITER_DIRStep 4: Spawn All Runs
For each EVAL in EVALS, spawn TWO subagents in the SAME turn:
With-skill run:
Agent(prompt: """
Execute this task:
- Read the skill at: SKILL_PATH/SKILL.md
- Follow the skill's instructions to complete this task: EVAL.prompt
- Save all outputs to: ITER_DIR/eval-EVAL.id/with_skill/outputs/
""")Baseline run (no skill):
Agent(prompt: """
Execute this task (no special instructions):
- Task: EVAL.prompt
- Save all outputs to: ITER_DIR/eval-EVAL.id/without_skill/outputs/
""")Launch ALL runs (with-skill + baseline for every eval) in parallel.
Step 5: Draft Assertions While Runs Execute
While runs are in progress, draft quantitative assertions for each eval:
- Assertions must be objectively verifiable
- Give each a descriptive name
- Skip assertions for subjective outcomes (writing style, design quality)
Write eval_metadata.json for each eval directory:
json
{
"eval_id": 1,
"eval_name": "descriptive-name",
"prompt": "The user's task prompt",
"assertions": ["Output includes X", "File is valid JSON"]
}Explain assertions to user while waiting.
Step 6: Grade Results
Once all runs complete:
For each run (withskill and withoutskill):
Agent(prompt: """
Read agents/skill-grader.md and follow its instructions.
- expectations: EVAL.expectations
- transcript_path: ITER_DIR/eval-EVAL.id/CONFIG/transcript.md
- outputs_dir: ITER_DIR/eval-EVAL.id/CONFIG/outputs/
- eval_prompt: EVAL.prompt
""", subagent_type: "majestic-tools:agents:skill-grader")Step 7: Present Results
For each eval, show:
- Prompt
- With-skill pass rate vs baseline pass rate
- Key differences in outputs
- Any eval feedback from grader
Summary table:
| Eval | With Skill | Baseline | Delta |
|------|-----------|----------|-------|
| 1 | 85% | 35% | +50% |Ask user: "How do these look? Any specific feedback on the outputs?"
Step 8: Iterate (if needed)
If user has feedback:
1. Analyze feedback — generalize, don't overfit to specific cases
2. Improve the skill (Edit SKILL.md)
3. Explain changes made and why
4. Ask: "Want to rerun the evals with the updated skill?"
5. If yes: goto Step 3 (new iteration)
If user is satisfied:
"Skill looks good!"Improvement Guidelines
When rewriting skills based on feedback:
- Generalize — don't overfit to the 2-3 test cases; the skill will be used many times
- Keep it lean — remove instructions that aren't pulling their weight
- Explain the why — tell the model why things matter instead of rigid MUSTs
- Look for repeated work — if all runs wrote similar helper scripts, bundle the script in the skill
- Read transcripts — check if the skill wastes time on unproductive steps
Error Handling
| Condition | Action |
|---|---|
| Subagent run times out | Note timeout, grade available outputs |
| No outputs produced | FAIL all expectations for that run |
| Skill has syntax errors | Fix before running evals |
| User wants to stop iterating | Accept and summarize current state |
