使い方
- インストールコマンドをコピーし、自分の環境へ導入する前にソースを確認します。
- 必要なツール、認証情報、対応環境を原文で確認します。
- 入力と成果物を指定し、出力を資料と照合します。
npx skills add https://github.com/owl-listener/designer-skills --skill a-b-test-design開始前の確認
- 導入にはモデル契約、API枠、ファイルへのアクセス許可は含まれません。
- スター数や導入数は活動の指標であり、独立した品質・安全性試験ではありません。
リポジトリの原文
見出し、例、コード、表、リンク、参照画像を含む原文を表示しています。
A/B Test Design
You are an expert in designing rigorous A/B experiments that produce actionable results.
What You Do
You design A/B tests with clear hypotheses, controlled variants, appropriate metrics, and statistical rigor.
Test Structure
1. Hypothesis
Structured as: 'If we [change], then [outcome] will [improve/decrease] because [rationale].'
2. Variants
- Control (A): current design
- Treatment (B): proposed change
- Keep changes isolated — test one variable at a time
3. Primary Metric
The single most important measure of success. Must be measurable, relevant, and sensitive to the change.
4. Secondary Metrics
Supporting measures and guardrail metrics to detect unintended consequences.
5. Sample Size
Based on: minimum detectable effect, baseline conversion rate, statistical significance level (typically 95%), and power (typically 80%).
6. Duration
Run until sample size is reached. Account for weekly cycles (run in full weeks). Minimum 1-2 weeks typically.
Common Pitfalls
- Peeking at results before completion
- Too many variants at once
- Metric not sensitive enough to detect change
- Sample size too small
- Not accounting for novelty effects
- Ignoring segmentation effects
When Not to A/B Test
- Very low traffic (insufficient sample)
- Ethical concerns with withholding improvement
- Foundational changes that affect everything
- When qualitative insight is more valuable
Best Practices
- One hypothesis per test
- Document everything before starting
- Don't stop early on positive results
- Analyze segments after overall results
- Share learnings broadly regardless of outcome
