lilmgenius/paperthin

mandela

Audit any eval, metric, experiment, or benchmark for leakage: does external ground-truth enter independently, or are the model, scorer, and designer just confirming a result no outside truth ever produced?

View source

How to use

  1. Copy the installation command and review the source package before installing it in your own agent environment.
  2. Check the source document for required tools, credentials and supported hosts.
  3. Describe your input and desired deliverable; review the output against the source material.
npx skills add https://github.com/lilmgenius/paperthin --skill mandela

Before you start

  • Installation does not include a model subscription, API allowance or permission to access your files.
  • Repository stars and installation counts describe activity, not an independent quality or security test.
Original skill document

Rendered from the source repository. Headings, examples, code, tables, links, and referenced images are preserved.

Audit whether a validation receives independent outside ground-truth.

Goal

The Mandela Effect names a population confidently remembering something that never independently happened. A leaky validation has the same shape: designer, model, and scorer confirm each other. mandela uses eight leakage patterns to check whether a validation is independent.

Workflow

  1. Identify the validation: eval, metric, experiment, holdout, or "how we'll know". Name its components: model, scorer, designer, and dataset.
  2. Ask the core question: does external ground-truth enter independently?
  3. Test all 8 patterns below. Some apply only to particular components, such as a human subject or a scorer. Report only patterns that fire, each by name.
  4. Recall, not reason: a memorized answer recited instead of one actually derived; the system already knows the result it is supposedly computing.
  5. Wrong null hypothesis: an ablation that removes a surface label but not the underlying signal the system actually exploits, so the "control" still leaks.
  6. Shared hallucination: two components verifying each other; circularity reported as a number.
  7. Tautology: a scorer grading buckets it drew itself.
  8. Verifier = designer: a private, unreproducible recipe in a holdout's clothes.
  9. Shared-pool bias: train and holdout drawn from one labeler pool, so one bias enters both sides.
  10. Frame injection: a question that hands the subject the hypothesis.
  11. Demand characteristics: measured subjects who know they're being measured.
  12. Give the independent-ground-truth fix for each hit.

Rules

  • Blinding the output value can still leak the collection recipe.
  • Read-only: name the leak and the independence fix; don't rewrite the experiment.
  • For a high-stakes validation, you may add one independent auditor with fresh context (N=1). Give it only the validation design, without this session's reasoning, to rerun the eight patterns blind. The default remains same-session and read-only.

Verification

Turn mandela on this audit:

  1. Run patterns #3 through #5 on yourself: are you a scorer grading buckets you drew (Tautology, #4)? Is the verifier the designer (#5)? Is your verdict a shared hallucination with the design's own claims (#3)?
  2. Could a reader who didn't run the audit reach your verdict from the cited evidence alone, using independent ground-truth?
  3. The report names the root, not a laundry list.
from this repository

More skills

All skills
lilmgenius
Community

re0-order

Move items only, reword nothing; realign a listing whose order has gone arbitrary (a list, table, catalog, sections, an enum) into a logical sequence under one stated principle. Nothing is added or removed. Use when the user names a listing, or a scope holding one, whose order no longer helps a reader follow the set, or when a pipeline the user invoked passes that scope on. Acts only inside that scope.

1h installs
5
GitHub stars
1.5K
Updated
Nov 15
lilmgenius
Community

readback

Verify the model's understanding of a user's instruction before spending non-trivial work. Use when a request is long, bundled, high-stakes, hard to undo, or has ambiguous scope or referents such as this, that, it, the other one, whatever is cleaner, or whichever order makes sense. Restate internally, cross-check against available context, proceed silently when resolved, and surface only a genuine surviving fork.

1h installs
5
GitHub stars
1.5K
Updated
Nov 15
lilmgenius
Community

whychk

Right after you pick an option, have a fresh outsider find the sharpest gap in the choice, then explain that gap in your own words, one gap at a time.

1h installs
5
GitHub stars
1.5K
Updated
Nov 15
lilmgenius
Community

aim

When data lands with a thin ask (files, a repo, a paste, a chat scrollback), read it and propose what the user is trying to do with it for them to confirm. Use the moment material arrives without a fully-formed ask, or when the ask reads thinner than the data suggests.

1h installs
4
GitHub stars
1.5K
Updated
Nov 15