사용 방법
- 설치 명령을 복사하고 자신의 에이전트 환경에 설치하기 전에 소스 패키지를 확인하세요.
- 필요한 도구, 인증 정보, 지원 환경은 원본 문서에서 확인하세요.
- 입력과 원하는 산출물을 설명하고 출처와 결과를 대조하세요.
npx skills add https://github.com/cheaper-inference/omniroute --skill cli-eval시작 전 확인
- 설치에는 모델 구독, API 한도, 파일 접근 권한이 포함되지 않습니다.
- 별점 수와 설치 수는 활동 지표이며 독립적인 품질·보안 검증이 아닙니다.
원본 저장소의 제목, 예시, 코드, 표, 링크, 이미지를 유지해 표시합니다.
<!-- generated by src/lib/agentSkills/generator.ts; manual edits will be overwritten -->
Overview
Create and run evaluation suites, watch live benchmark progress, view scorecards, compare model performance, and integrate eval runs with CI workflows from the CLI.
Quick install
npm install -g omniroute # or: npx omniroute
omniroute --versionSubcommands
eval
Example:
omniroute evaleval suites
Example:
omniroute eval suiteseval list
Example:
omniroute eval listeval get <suiteId>
Example:
omniroute eval get <suiteId>eval create
Flags:
--file <path>
Example:
omniroute eval createeval run <suiteId>
Flags:
-m, --model <id>--combo <name>--concurrency <n>--tag <tag>--watch
Example:
omniroute eval run <suiteId>eval list
Flags:
--suite <id>--status <s>--since <ts>--limit <n>
Example:
omniroute eval listeval get <runId>
Example:
omniroute eval get <runId>eval results <runId>
Flags:
--failed
Example:
omniroute eval results <runId>eval cancel <runId>
Flags:
--yes
Example:
omniroute eval cancel <runId>eval scorecard <runId>
Example:
omniroute eval scorecard <runId>simulate [prompt]
Flags:
--file <path>-m, --model <id>--combo <name>--reasoning-effort <level>--thinking-budget <n>--explain
Example:
omniroute simulate [prompt]<!-- skill:custom-start --> <!-- Migrated from skills/omniroute-cli-eval/SKILL.md (preserved curated content) -->
OmniRoute — CLI Evals
Requires the omniroute CLI. See CLI entry-point skill for install + global flags.
What are evals?
Evals are automated test suites that score LLM outputs against expected answers or rubrics. OmniRoute stores suites and run results in its local database.
Eval suites
omniroute eval suites list # List all eval suites
omniroute eval suites list --json # JSON output
omniroute eval suites get <suiteId> # Full suite definitionCreate a suite
omniroute eval suites create \
--name "code-quality" \
--rubric "exact-match" \
--samples-file ./samples.jsonl # JSONL: {input, expected_output}Rubric options: exact-match, contains, llm-judge, regex.
--samples-file format (one JSON object per line):
{"input": "What is 2+2?", "expected_output": "4"}
{"input": "Translate 'hello' to Spanish", "expected_output": "hola"}Run an eval
omniroute eval suites run <suiteId> \
--model claude-sonnet-4-6 # Run suite against a specific model
omniroute eval suites run <suiteId> \
--model gpt-4o \
--watch # Live TUI progress (EvalWatch)The run is asynchronous. Use --watch for a live terminal dashboard or poll manually:
RUN_ID=$(omniroute eval suites run <suiteId> --model claude-sonnet-4-6 --output json | jq -r '.id')
omniroute eval get $RUN_IDManage runs
omniroute eval list # List all eval runs
omniroute eval list --json
omniroute eval get <runId> # Run details (status, model, score)
omniroute eval results <runId> # Per-sample results
omniroute eval scorecard <runId> # Full scorecard with pass/fail per sample
omniroute eval cancel <runId> # Cancel a running evalScorecard output
omniroute eval scorecard <runId> --output jsonResponse fields per sample:
{
"id": "sample-1",
"score": 0.95,
"passed": true,
"input": "What is 2+2?",
"output": "4",
"expected": "4"
}Comparing models
Run the same suite against multiple models and compare:
for MODEL in claude-sonnet-4-6 gpt-4o gemini-2.0-flash; do
omniroute eval suites run $SUITE_ID --model $MODEL --output json | jq '{model: .model, score: .score}'
doneCI integration
# Run and fail CI if score drops below threshold
SCORE=$(omniroute eval suites run $SUITE_ID --model claude-sonnet-4-6 --output json | jq -r '.score')
python3 -c "import sys; score=float('$SCORE'); sys.exit(0 if score >= 0.90 else 1)"Errors
suites createfails withinvalid rubric→ use one of:exact-match,contains,llm-judge,regexsuites runreturnsmodel not found→ verify model ID withomniroute models --search <name>eval getshowsstatus: failed→ checkomniroute logs --search evalfor error detailsscorecardreturns empty results → the run may still berunning; pollomniroute eval get <runId>untilstatusiscompleted
<!-- skill:custom-end -->
