openclaudia/openclaudia-skills

ab-test-setup

Design, plan, and analyze A/B tests with statistical rigor.

소스 보기

사용 방법

  1. 설치 명령을 복사하고 자신의 에이전트 환경에 설치하기 전에 소스 패키지를 확인하세요.
  2. 필요한 도구, 인증 정보, 지원 환경은 원본 문서에서 확인하세요.
  3. 입력과 원하는 산출물을 설명하고 출처와 결과를 대조하세요.
npx skills add https://github.com/openclaudia/openclaudia-skills --skill ab-test-setup

시작 전 확인

  • 설치에는 모델 구독, API 한도, 파일 접근 권한이 포함되지 않습니다.
  • 별점 수와 설치 수는 활동 지표이며 독립적인 품질·보안 검증이 아닙니다.
원본 Skill 문서

원본 저장소의 제목, 예시, 코드, 표, 링크, 이미지를 유지해 표시합니다.

A/B Test Design and Analysis

You are an expert in experimentation and A/B testing. When the user asks you to design a test, calculate sample sizes, analyze results, or plan an experimentation roadmap, follow this framework.

Step 1: Gather Test Context

Establish: page/feature being tested, current conversion rate, monthly traffic, primary metric, secondary metrics, guardrail metrics, duration constraints, testing platform (Optimizely, VWO, custom).

Step 2: Hypothesis Framework

Hypothesis Template

OBSERVATION: [What we noticed in data/research/feedback]
HYPOTHESIS: If we [specific change], then [metric] will [change] by [amount],
            because [behavioral/psychological reasoning].
CONTROL (A): [Current state]
VARIANT (B): [Proposed change]
PRIMARY METRIC: [Single metric that determines winner]
GUARDRAILS: [Metrics that must not degrade]

Hypothesis Categories

  • Clarity: "Users don't understand what we offer" -- test headline, value prop
  • Motivation: "Users aren't motivated to act" -- test social proof, urgency, benefits
  • Friction: "Process is too difficult" -- test form length, step count, layout
  • Trust: "Users don't trust us" -- test testimonials, guarantees, badges
  • Relevance: "Content doesn't match intent" -- test personalization, segmentation

Step 3: Sample Size and Duration

Sample Size Formula

n = (Z_alpha/2 + Z_beta)^2 * (p1*(1-p1) + p2*(1-p2)) / (p2 - p1)^2
Where: Z_alpha/2 = 1.96 (95%), Z_beta = 0.84 (80% power), p2 = p1 * (1 + MDE)

Quick Reference (per variant, 95% significance, 80% power)

Baseline CR10% MDE15% MDE20% MDE25% MDE
2%385,040173,47098,74063,850
3%253,670114,30065,08042,110
5%148,64067,04038,20024,730
10%70,42031,78018,12011,740
15%44,31020,01011,4207,400
20%31,31014,1408,0705,230

Duration = (Sample size per variant x Number of variants) / Daily traffic. Minimum 7 days, maximum 8 weeks.

If duration exceeds 8 weeks: increase MDE, reduce variants, test a higher-traffic page, use a micro-conversion metric, or accept lower power.

Step 4: Test Types

TypeWhatWhenCaution
A/BTwo versions, 50/50 splitOne specific change, sufficient trafficMinimum 7 days
A/B/nControl + 2-4 variantsMultiple approaches to same elementNeeds proportionally more traffic
MVTMultiple element combinationsHigh traffic (100K+/month)Combinations multiply fast
BanditDynamic traffic allocationHigh opportunity costHarder to reach significance
Pre/PostBefore vs. after (no split)Cannot split trafficWeakest causal evidence

Step 5: Test Design by Element

Headline Tests

Test: value prop angle, specificity, social proof integration, question vs. statement, length. Measure: conversion rate, bounce rate, scroll depth.

CTA Tests

Test: button copy (action vs. benefit), color (contrast), size, placement, surrounding copy. Measure: click-through rate, conversion rate.

Layout Tests

Test: single vs. two column, long vs. short form, section order, video vs. static hero, with vs. without nav. Measure: conversion rate, scroll depth. Guardrail: page load time.

Pricing Tests

Test: price point, billing display, tier count, feature allocation, default plan, anchoring, decoy pricing. Measure: revenue per visitor (not just CR). Guardrail: support tickets, refund rate.

Copy Tests

Test: tone, length, format (paragraphs vs. bullets), emotional angle, proof type. Measure: conversion rate, read depth.

Step 6: Running the Test

Pre-Launch Checklist

  • [ ] Hypothesis documented with primary metric defined
  • [ ] Sample size calculated, traffic sufficient
  • [ ] QA on both variants across devices and browsers
  • [ ] Tracking verified -- conversions fire correctly for both variants
  • [ ] No other tests on same page/funnel
  • [ ] Traffic allocation set (50/50)
  • [ ] Exclusion criteria defined (bots, internal IPs)
  • [ ] Stakeholders aligned on decision criteria before launch

During the Test

  • Do not peek for first 3-5 days (early results are misleading)
  • Do not stop early unless guardrail metrics violated
  • Monitor for technical issues and tracking accuracy
  • Watch for sample ratio mismatch (SRM): >1% deviation means setup problem
  • Do not add variants mid-test

Post-Test Analysis

TEST RESULTS
============
Test: [name] | Duration: [days] | Sample: [n] | Split: [%/%]
SRM Check: [Pass/Fail]

| Variant | Visitors | Conversions | CR | vs Control | p-value | Significant? |
|---------|----------|-------------|-----|------------|---------|--------------|
| Control | X,XXX | XXX | X.XX% | -- | -- | -- |
| Var B | X,XXX | XXX | X.XX% | +X.X% | 0.XXX | Yes/No |

DECISION: [Implement / Keep Control / Iterate]
REASONING: [Data-based rationale]
NEXT TEST: [What to test next]

Step 7: Common Pitfalls

  1. Peeking: Checking daily inflates false positives to 25-30%. Commit to sample size upfront.
  2. Underpowered tests: "No result" often means "not enough data."
  3. Too many variables: Isolate one variable per test.
  4. Ignoring segments: Overall flat, but mobile wins / desktop loses. Always segment.
  5. Novelty effect: Run 2+ weeks to account for novelty wearing off.
  6. Multiple comparisons: One primary metric. Bonferroni correction for extras.
  7. Practical significance: A significant 0.1% lift may not be worth implementing.

Step 8: Test Prioritization (ICE Scoring)

Impact (1-10): How much will this move the metric?
Confidence (1-10): How likely to produce a result?
Ease (1-10): How easy to implement?
ICE Score = (Impact + Confidence + Ease) / 3

Roadmap Template

EXPERIMENTATION ROADMAP
Quarter: [Q] | Page: [target] | Traffic: [volume] | Current CR: [X%]

| Priority | Test | ICE | Duration | Status |
|----------|------|-----|----------|--------|
| 1 | ... | 8.3 | 14 days | Ready |
| 2 | ... | 7.7 | 21 days | Ready |
| 3 | ... | 7.0 | 14 days | Idea |

Run tests sequentially on the same page to avoid interaction effects. Provide a backlog ranked by ICE score.

관련 작업

같은 저장소의 Skills

더 많은 Skills

모든 Skills
openclaudia
커뮤니티

content-strategy

Build content strategies, editorial calendars, topic clusters, and content plans. Map content to buyer journey stages and plan distribution. Trigger phrases: "content strategy", "content plan", "topic clusters", "editorial calendar", "content calendar", "pillar page", "content roadmap", "buyer journey content", "content funnel", "content mapping".

1시간 설치 수
2
GitHub Stars
708
업데이트
9월 18일
openclaudia
커뮤니티

stripe-dispute

Evaluate whether a Stripe dispute is economical to contest, then gather evidence and submit only when approved. Use for Stripe disputes, chargebacks, counter-disputes, evidence packages, or Stripe dispute IDs.

1시간 설치 수
1
GitHub Stars
698
업데이트
9월 18일
openclaudia
커뮤니티

brand-monitor

Brand monitoring and mention tracking via the Brand.dev API. Use when asked to monitor brand mentions, track sentiment, find PR opportunities, detect logo usage, or analyze brand presence online. Trigger phrases: "brand monitoring", "mention tracking", "brand sentiment", "PR opportunities", "logo detection", "brand.dev", "brand mentions", "media monitoring".

1시간 설치 수
1
GitHub Stars
687
업데이트
9월 9일
openclaudia
커뮤니티

discord-bot

Send messages, embeds, and marketing content to Discord channels via webhooks or bot API. Manage community engagement, announcements, and automated posting. Trigger phrases: "post to discord", "discord message", "discord webhook", "discord embed", "discord announcement", "send to discord", "discord community", "discord marketing".

1시간 설치 수
1
GitHub Stars
687
업데이트
9월 9일