사용 방법
- 설치 명령을 복사하고 자신의 에이전트 환경에 설치하기 전에 소스 패키지를 확인하세요.
- 필요한 도구, 인증 정보, 지원 환경은 원본 문서에서 확인하세요.
- 입력과 원하는 산출물을 설명하고 출처와 결과를 대조하세요.
npx skills add https://github.com/getnao/nao --skill create-context-tests시작 전 확인
- 설치에는 모델 구독, API 한도, 파일 접근 권한이 포함되지 않습니다.
- 별점 수와 설치 수는 활동 지표이며 독립적인 품질·보안 검증이 아닙니다.
원본 저장소의 제목, 예시, 코드, 표, 링크, 이미지를 유지해 표시합니다.
create-context-tests
nao test runs each natural-language prompt through the agent, executes both the agent's SQL and the test's expected SQL against the warehouse, and diffs the result data row-by-row. A test passes only if the actual data matches — same rows, same values. The suite is the reliability benchmark; every change to RULES.md is measured against it. Reference: docs.getnao.io/nao-agent/context-engineering/evaluation.
How many tests
One test per key metric in `## Key Metrics Reference` is the floor. Then add tests for: time scoping (especially "last 8 weeks" / "last 30 days"), CTE / multi-step queries, edge cases (NULLs, empty windows), and ambiguous wording ("our users", "active") to validate naming-convention rules.
Two authoring rules — apply to every test
Rule 1 — Prompts read like real chat. Vague, short, no table/column/method hints. The test verifies the agent reaches the right answer from a real-user input.
| Bad | Good |
|---|---|
"What was the churn rate from fct_subscriptions in Q1?" | "How's churn looking this quarter?" |
"Compute MRR as SUM(mrr_amount) where status='active'" | "What's our MRR?" |
Rule 2 — Output column names encode format / unit, not source. A column name communicates how to interpret the value.
| Bad | Good |
|---|---|
churn_rate_from_fct_subscriptions | churn_rate_float_0_1 |
mrr_amount_fct_stripe_mrr | mrr_usd_dollars |
signup_at_dim_users | signup_date_yyyy_mm_dd |
Naming patterns: <metric>_float_0_1 or <metric>_percentage_0_100 for rates; <metric>_<currency>_<unit> for money; <thing>_count; <thing>_date_yyyy_mm_dd. See templates/test.yaml.
Steps
- Ask once: does the user have trusted source-of-truth queries (Looker, dashboards, prior benchmarks)? If yes, transform each into a test (rewrite SELECT to apply Rule 2; reverse-engineer a Rule 1 prompt). For metrics without a trusted query, draft new tests one per metric.
- Save flat under `tests/` (no subfolders), one YAML file per test. Use
templates/test.yaml.
- Have the user validate — confirm prompts match their team's phrasing and SQL matches their definition of truth.
- Run `nao test`. Prerequisites:
cdinto the project directory (wherenao_config.yamllives).- Start
nao chat &in the background (the test runner reuses the chat server). - LLM configured in
nao_config.yaml. - First run prompts for login credentials — let the user type them; don't script around it.
- If you see
AI_APICallError: Not Foundathttps://api.anthropic.com/messages(no/v1/), rununset ANTHROPIC_BASE_URL ANTHROPIC_API_KEYfirst (parent agent CLI is leaking env vars). Seesetup-contextfor the full note.
nao test -m <model_id> -t 10 # -t = parallelism- Recap results: pass rate, token cost, wall-clock time. Cite this as the baseline.
- Diagnose failures (optional): read
tests/outputs/for each failure, identify the rule gap, propose the smallest fix, then route towrite-context-rules(oraudit-contextfor systemic issues). Re-run between fixes so impact is attributable.
Guardrails
- Tests' SQL must execute as-is — no
<placeholder>inFROM. Use real table / column names. - Never leak the answer in
promptor output column names (Rules 1 + 2). - One test per metric is the floor; coverage tests come after.
- Apply one context fix at a time between runs.
- If a test contradicts
RULES.md, stop and ask which is correct — it's a bug in one or the other.
Templates
templates/test.yaml— single-test format.
