lilmgenius/paperthin

mandela

Audit any eval, metric, experiment, or benchmark for leakage: does external ground-truth enter independently, or are the model, scorer, and designer just confirming a result no outside truth ever produced?

Voir la source

Mode d’emploi

  1. Copiez la commande et examinez le paquet avant de l’installer dans votre environnement agent.
  2. Consultez le document original pour les outils, identifiants et environnements requis.
  3. Décrivez les entrées et le livrable souhaité ; comparez la sortie aux sources.
npx skills add https://github.com/lilmgenius/paperthin --skill mandela

Avant de commencer

  • L’installation n’inclut ni abonnement au modèle, ni quota API, ni accès aux fichiers.
  • Étoiles et installations reflètent l’activité, pas un test indépendant de qualité ou de sécurité.
Document Skill original

Rendu depuis le dépôt source en conservant titres, exemples, code, tableaux, liens et images.

Audit whether a validation receives independent outside ground-truth.

Goal

The Mandela Effect names a population confidently remembering something that never independently happened. A leaky validation has the same shape: designer, model, and scorer confirm each other. mandela uses eight leakage patterns to check whether a validation is independent.

Workflow

  1. Identify the validation: eval, metric, experiment, holdout, or "how we'll know". Name its components: model, scorer, designer, and dataset.
  2. Ask the core question: does external ground-truth enter independently?
  3. Test all 8 patterns below. Some apply only to particular components, such as a human subject or a scorer. Report only patterns that fire, each by name.
  4. Recall, not reason: a memorized answer recited instead of one actually derived; the system already knows the result it is supposedly computing.
  5. Wrong null hypothesis: an ablation that removes a surface label but not the underlying signal the system actually exploits, so the "control" still leaks.
  6. Shared hallucination: two components verifying each other; circularity reported as a number.
  7. Tautology: a scorer grading buckets it drew itself.
  8. Verifier = designer: a private, unreproducible recipe in a holdout's clothes.
  9. Shared-pool bias: train and holdout drawn from one labeler pool, so one bias enters both sides.
  10. Frame injection: a question that hands the subject the hypothesis.
  11. Demand characteristics: measured subjects who know they're being measured.
  12. Give the independent-ground-truth fix for each hit.

Rules

  • Blinding the output value can still leak the collection recipe.
  • Read-only: name the leak and the independence fix; don't rewrite the experiment.
  • For a high-stakes validation, you may add one independent auditor with fresh context (N=1). Give it only the validation design, without this session's reasoning, to rerun the eight patterns blind. The default remains same-session and read-only.

Verification

Turn mandela on this audit:

  1. Run patterns #3 through #5 on yourself: are you a scorer grading buckets you drew (Tautology, #4)? Is the verifier the designer (#5)? Is your verdict a shared hallucination with the design's own claims (#3)?
  2. Could a reader who didn't run the audit reach your verdict from the cited evidence alone, using independent ground-truth?
  3. The report names the root, not a laundry list.
du même dépôt

Autres Skills

Tous les Skills
lilmgenius
Communauté

re0-order

Move items only, reword nothing; realign a listing whose order has gone arbitrary (a list, table, catalog, sections, an enum) into a logical sequence under one stated principle. Nothing is added or removed. Use when the user names a listing, or a scope holding one, whose order no longer helps a reader follow the set, or when a pipeline the user invoked passes that scope on. Acts only inside that scope.

Installations / 1h
5
GitHub Stars
1,5 k
Mis à jour
15 nov.
lilmgenius
Communauté

readback

Verify the model's understanding of a user's instruction before spending non-trivial work. Use when a request is long, bundled, high-stakes, hard to undo, or has ambiguous scope or referents such as this, that, it, the other one, whatever is cleaner, or whichever order makes sense. Restate internally, cross-check against available context, proceed silently when resolved, and surface only a genuine surviving fork.

Installations / 1h
5
GitHub Stars
1,5 k
Mis à jour
15 nov.
lilmgenius
Communauté

whychk

Right after you pick an option, have a fresh outsider find the sharpest gap in the choice, then explain that gap in your own words, one gap at a time.

Installations / 1h
5
GitHub Stars
1,5 k
Mis à jour
15 nov.
lilmgenius
Communauté

aim

When data lands with a thin ask (files, a repo, a paste, a chat scrollback), read it and propose what the user is trying to do with it for them to confirm. Use the moment material arrives without a fully-formed ask, or when the ask reads thinner than the data suggests.

Installations / 1h
4
GitHub Stars
1,5 k
Mis à jour
15 nov.