ai-evals-course/evals-skills

write-code-eval

Write code evaluators for known failure modes with objective rules.

소스 보기

사용 방법

  1. 설치 명령을 복사하고 자신의 에이전트 환경에 설치하기 전에 소스 패키지를 확인하세요.
  2. 필요한 도구, 인증 정보, 지원 환경은 원본 문서에서 확인하세요.
  3. 입력과 원하는 산출물을 설명하고 출처와 결과를 대조하세요.
npx skills add https://github.com/ai-evals-course/evals-skills --skill write-code-eval

시작 전 확인

  • 설치에는 모델 구독, API 한도, 파일 접근 권한이 포함되지 않습니다.
  • 별점 수와 설치 수는 활동 지표이며 독립적인 품질·보안 검증이 아닙니다.
원본 Skill 문서

원본 저장소의 제목, 예시, 코드, 표, 링크, 이미지를 유지해 표시합니다.

Write a code evaluator

Start with a failure mode found through error analysis. Write one check for that failure mode, much like a unit test that asserts what should hold for each trace.

  1. State the rule and identify the trace fields or reference data the check needs. If the rule requires interpretation, use write-judge-prompt.
  2. Implement the check in the project's language and eval framework. Return a result and a reason in the format that framework expects.
  3. Test known passes and failures, including borderline cases. Run the check on available traces and inspect mistakes. If the rule uses a proxy for human judgment, compare its results with human labels.

Examples

Failure modePossible check
Invalid output structureParse the output and check required fields
Missing or forbidden textMatch a string or pattern
Citation not in retrieved documentsCompare cited IDs with retrieved IDs
Bad tool callCheck arguments against the tool schema or run the call in a safe test environment
Wrong valueCompare the output with a reference value

Choose the check from the failure rule. For a failure with both objective and interpretive parts, check the objective part with code and use a judge for the rest.

관련 작업

같은 저장소의 Skills

더 많은 Skills

모든 Skills
ai-evals-course
커뮤니티

error-discovery

Run error analysis on a dataset. Build a review UI, select diverse samples, monitor annotations, and organize failure modes.

1시간 설치 수
1
GitHub Stars
1.7천
업데이트
10월 24일
ai-evals-course
커뮤니티

evaluate-rag

Guides evaluation of RAG pipeline retrieval and generation quality. Use when evaluating a retrieval-augmented generation system, measuring retrieval quality, assessing generation faithfulness or relevance, generating synthetic QA pairs for retrieval testing, or optimizing chunking strategies.

1시간 설치 수
1
GitHub Stars
1.7천
업데이트
10월 24일
ai-evals-course
커뮤니티

build-review-interface

Build a custom browser-based annotation interface tailored to your data for reviewing LLM traces and collecting structured feedback. Use when you need to build an annotation tool, review traces, or collect human labels.

1시간 설치 수
2
GitHub Stars
1.7천
업데이트
10월 24일
ai-evals-course
커뮤니티

eval-audit

Audit an LLM eval pipeline and surface problems: missing error analysis, unvalidated judges, vanity metrics, etc. Use when inheriting an eval system, when unsure whether evals are trustworthy, or as a starting point when no eval infrastructure exists. Do NOT use when the goal is to build a new evaluator from scratch (use error-discovery, write-judge-prompt, or validate-evaluator instead).

1시간 설치 수
2
GitHub Stars
1.7천
업데이트
10월 24일