lilmgenius/paperthin

mandela

Audit any eval, metric, experiment, or benchmark for leakage: does external ground-truth enter independently, or are the model, scorer, and designer just confirming a result no outside truth ever produced?

소스 보기

사용 방법

  1. 설치 명령을 복사하고 자신의 에이전트 환경에 설치하기 전에 소스 패키지를 확인하세요.
  2. 필요한 도구, 인증 정보, 지원 환경은 원본 문서에서 확인하세요.
  3. 입력과 원하는 산출물을 설명하고 출처와 결과를 대조하세요.
npx skills add https://github.com/lilmgenius/paperthin --skill mandela

시작 전 확인

  • 설치에는 모델 구독, API 한도, 파일 접근 권한이 포함되지 않습니다.
  • 별점 수와 설치 수는 활동 지표이며 독립적인 품질·보안 검증이 아닙니다.
원본 Skill 문서

원본 저장소의 제목, 예시, 코드, 표, 링크, 이미지를 유지해 표시합니다.

Audit whether a validation receives independent outside ground-truth.

Goal

The Mandela Effect names a population confidently remembering something that never independently happened. A leaky validation has the same shape: designer, model, and scorer confirm each other. mandela uses eight leakage patterns to check whether a validation is independent.

Workflow

  1. Identify the validation: eval, metric, experiment, holdout, or "how we'll know". Name its components: model, scorer, designer, and dataset.
  2. Ask the core question: does external ground-truth enter independently?
  3. Test all 8 patterns below. Some apply only to particular components, such as a human subject or a scorer. Report only patterns that fire, each by name.
  4. Recall, not reason: a memorized answer recited instead of one actually derived; the system already knows the result it is supposedly computing.
  5. Wrong null hypothesis: an ablation that removes a surface label but not the underlying signal the system actually exploits, so the "control" still leaks.
  6. Shared hallucination: two components verifying each other; circularity reported as a number.
  7. Tautology: a scorer grading buckets it drew itself.
  8. Verifier = designer: a private, unreproducible recipe in a holdout's clothes.
  9. Shared-pool bias: train and holdout drawn from one labeler pool, so one bias enters both sides.
  10. Frame injection: a question that hands the subject the hypothesis.
  11. Demand characteristics: measured subjects who know they're being measured.
  12. Give the independent-ground-truth fix for each hit.

Rules

  • Blinding the output value can still leak the collection recipe.
  • Read-only: name the leak and the independence fix; don't rewrite the experiment.
  • For a high-stakes validation, you may add one independent auditor with fresh context (N=1). Give it only the validation design, without this session's reasoning, to rerun the eight patterns blind. The default remains same-session and read-only.

Verification

Turn mandela on this audit:

  1. Run patterns #3 through #5 on yourself: are you a scorer grading buckets you drew (Tautology, #4)? Is the verifier the designer (#5)? Is your verdict a shared hallucination with the design's own claims (#3)?
  2. Could a reader who didn't run the audit reach your verdict from the cited evidence alone, using independent ground-truth?
  3. The report names the root, not a laundry list.
같은 저장소의 Skills

더 많은 Skills

모든 Skills
lilmgenius
커뮤니티

re0-order

Move items only, reword nothing; realign a listing whose order has gone arbitrary (a list, table, catalog, sections, an enum) into a logical sequence under one stated principle. Nothing is added or removed. Use when the user names a listing, or a scope holding one, whose order no longer helps a reader follow the set, or when a pipeline the user invoked passes that scope on. Acts only inside that scope.

1시간 설치 수
5
GitHub Stars
1.5천
업데이트
11월 15일
lilmgenius
커뮤니티

readback

Verify the model's understanding of a user's instruction before spending non-trivial work. Use when a request is long, bundled, high-stakes, hard to undo, or has ambiguous scope or referents such as this, that, it, the other one, whatever is cleaner, or whichever order makes sense. Restate internally, cross-check against available context, proceed silently when resolved, and surface only a genuine surviving fork.

1시간 설치 수
5
GitHub Stars
1.5천
업데이트
11월 15일
lilmgenius
커뮤니티

whychk

Right after you pick an option, have a fresh outsider find the sharpest gap in the choice, then explain that gap in your own words, one gap at a time.

1시간 설치 수
5
GitHub Stars
1.5천
업데이트
11월 15일
lilmgenius
커뮤니티

aim

When data lands with a thin ask (files, a repo, a paste, a chat scrollback), read it and propose what the user is trying to do with it for them to confirm. Use the moment material arrives without a fully-formed ask, or when the ask reads thinner than the data suggests.

1시간 설치 수
4
GitHub Stars
1.5천
업데이트
11월 15일