lilmgenius/paperthin

mandela

Audit any eval, metric, experiment, or benchmark for leakage: does external ground-truth enter independently, or are the model, scorer, and designer just confirming a result no outside truth ever produced?

Ver código-fonte

Como usar

  1. Copie o comando e verifique o pacote antes de instalar no seu ambiente de agente.
  2. Consulte no original as ferramentas, credenciais e ambientes necessários.
  3. Descreva os dados de entrada e a entrega esperada; confira a saída com as fontes.
npx skills add https://github.com/lilmgenius/paperthin --skill mandela

Antes de começar

  • A instalação não inclui assinatura de modelo, franquia API ou acesso aos seus arquivos.
  • Estrelas e instalações indicam atividade, não um teste independente de qualidade ou segurança.
Documento original do Skill

Renderizado do repositório de origem, preservando títulos, exemplos, código, tabelas, links e imagens.

Audit whether a validation receives independent outside ground-truth.

Goal

The Mandela Effect names a population confidently remembering something that never independently happened. A leaky validation has the same shape: designer, model, and scorer confirm each other. mandela uses eight leakage patterns to check whether a validation is independent.

Workflow

  1. Identify the validation: eval, metric, experiment, holdout, or "how we'll know". Name its components: model, scorer, designer, and dataset.
  2. Ask the core question: does external ground-truth enter independently?
  3. Test all 8 patterns below. Some apply only to particular components, such as a human subject or a scorer. Report only patterns that fire, each by name.
  4. Recall, not reason: a memorized answer recited instead of one actually derived; the system already knows the result it is supposedly computing.
  5. Wrong null hypothesis: an ablation that removes a surface label but not the underlying signal the system actually exploits, so the "control" still leaks.
  6. Shared hallucination: two components verifying each other; circularity reported as a number.
  7. Tautology: a scorer grading buckets it drew itself.
  8. Verifier = designer: a private, unreproducible recipe in a holdout's clothes.
  9. Shared-pool bias: train and holdout drawn from one labeler pool, so one bias enters both sides.
  10. Frame injection: a question that hands the subject the hypothesis.
  11. Demand characteristics: measured subjects who know they're being measured.
  12. Give the independent-ground-truth fix for each hit.

Rules

  • Blinding the output value can still leak the collection recipe.
  • Read-only: name the leak and the independence fix; don't rewrite the experiment.
  • For a high-stakes validation, you may add one independent auditor with fresh context (N=1). Give it only the validation design, without this session's reasoning, to rerun the eight patterns blind. The default remains same-session and read-only.

Verification

Turn mandela on this audit:

  1. Run patterns #3 through #5 on yourself: are you a scorer grading buckets you drew (Tautology, #4)? Is the verifier the designer (#5)? Is your verdict a shared hallucination with the design's own claims (#3)?
  2. Could a reader who didn't run the audit reach your verdict from the cited evidence alone, using independent ground-truth?
  3. The report names the root, not a laundry list.
do mesmo repositório

Mais Skills

Todos os Skills
lilmgenius
Comunidade

re0-order

Move items only, reword nothing; realign a listing whose order has gone arbitrary (a list, table, catalog, sections, an enum) into a logical sequence under one stated principle. Nothing is added or removed. Use when the user names a listing, or a scope holding one, whose order no longer helps a reader follow the set, or when a pipeline the user invoked passes that scope on. Acts only inside that scope.

Instalações / 1h
5
GitHub Stars
1,5 mil
Atualizado
15 de nov.
lilmgenius
Comunidade

readback

Verify the model's understanding of a user's instruction before spending non-trivial work. Use when a request is long, bundled, high-stakes, hard to undo, or has ambiguous scope or referents such as this, that, it, the other one, whatever is cleaner, or whichever order makes sense. Restate internally, cross-check against available context, proceed silently when resolved, and surface only a genuine surviving fork.

Instalações / 1h
5
GitHub Stars
1,5 mil
Atualizado
15 de nov.
lilmgenius
Comunidade

whychk

Right after you pick an option, have a fresh outsider find the sharpest gap in the choice, then explain that gap in your own words, one gap at a time.

Instalações / 1h
5
GitHub Stars
1,5 mil
Atualizado
15 de nov.
lilmgenius
Comunidade

aim

When data lands with a thin ask (files, a repo, a paste, a chat scrollback), read it and propose what the user is trying to do with it for them to confirm. Use the moment material arrives without a fully-formed ask, or when the ask reads thinner than the data suggests.

Instalações / 1h
4
GitHub Stars
1,5 mil
Atualizado
15 de nov.