terrylica/cc-skills

crucible-research-foundations

Validate findings, design shuffled nulls, check label leakage, review causal features.

Quelltext ansehen

Verwendung

  1. Installationsbefehl kopieren und das Quellpaket vor der Installation in der eigenen Agentenumgebung prüfen.
  2. Benötigte Werkzeuge, Zugangsdaten und Hosts im Original prüfen.
  3. Eingabe und gewünschtes Ergebnis beschreiben und die Ausgabe mit den Quellen abgleichen.
npx skills add https://github.com/terrylica/cc-skills --skill crucible-research-foundations

Vor dem Start

  • Die Installation umfasst weder Modellabo noch API-Kontingent oder Dateizugriffsrechte.
  • Sterne und Installationen zeigen Aktivität, keine unabhängige Qualitäts- oder Sicherheitsprüfung.
Originales Skill-Dokument

Aus dem Quell-Repository gerendert; Überschriften, Beispiele, Code, Tabellen, Links und Bilder bleiben erhalten.

Research Foundations — 6 epistemic disciplines

Self-Evolving Skill: This skill improves through use. If a discipline's guidance fails in practice or a new trap emerges, update the relevant section AND append to references/evolution-log.md. Don't defer.

Read these in order. The first three (causal, labels, nulls) are the hardest prerequisites — violating any of them silently invalidates every downstream result.


1. Causal-feature invariant (bars[:i])

Every feature f[i] used at trigger/decision bar i must be computable using only bars[0:i] — never bars[i], never bars[i+1:]. Violation produces look-ahead bias; findings silently become worthless.

Canonical pattern:

python
for i in range(n):
    lo = max(0, i - window)
    wind = values[lo:i]  # EXCLUSIVE upper bound — no peeking
    f[i] = compute(wind)

Note lo:i (exclusive), not lo:i+1. This discipline "feels off by one" but is correct.

Verification test (add to every new feature function):

python
def test_causality(fn, n=1000):
    bars = generate_test_bars(n)
    f_orig = fn(bars)
    bars_mod = bars.copy()
    bars_mod[500:] *= 2  # perturb the FUTURE
    f_mod = fn(bars_mod)
    assert np.array_equal(f_orig[:500], f_mod[:500]), "look-ahead detected"

Silent-bug signature: impossibly clean results (tw > 10 bps on FX, win rate > 70%, OOS matches IS perfectly).

Full reference: findings/methodology/10-causal-feature-invariant.md.


2. Label-leakage (bar-local scaling kills window leakage)

Forward labels must be scaled to the triggering bar's own range, NEVER to a window-wide scale. Window-relative labels are tautological.

Trap: If you label fwd+H = UP when close[i+H] - close[i] > window.span/20, then when close[i] is near window.min (loc=B), fwd=UP is near-automatic. Agents will report spurious "signals".

Fix: use bar-local triple-barrier labels:

python
r = high[i] - low[i]   # THIS bar's range, not window's
tp_level = close[i] + tp_mult * r
sl_level = close[i] - sl_mult * r
# walk forward, exit at first tp/sl/expiry

Symptom that you fell into the trap: apparent signal strengthens monotonically with loc quintile; collapses when you test adjacent cells.

Full reference: findings/methodology/02-label-leakage-bar-local-scaling.md.


3. Shuffled-null design (3 null types — get the right one)

Shuffled-null tests are mandatory before trust, but the choice of what to shuffle is a design decision.

Hypothesis classShuffle WHATSession example
"Feature X predicts outcomes"Shuffle the feature valuesPhase F-B (used wrong null, "falsified" a real signal)
"Trigger pattern fires at informative times"Shuffle the trigger mask (preserve fire-rate, move locations)Phase C (validated ngramtriplefast_up at z=+5.74)
"Filter improves selection"Shuffle which trades pass the filterPhase L-C (evaluated filters against N-size random draws)

Rule: ask "what is the alternative hypothesis, in one sentence?" If you can't state it, you don't know what you're testing.

Common mistakes:

  • Using feature-shuffle when testing a trigger pattern → destroys temporal structure the pattern depends on → real signal looks worse than shuffled noise
  • Under-tight null (null std huge relative to observed effect) → no statistical power
  • Over-tight null (too few permutations) → unreliable z-estimates; use ≥100 for z<3, ≥1000 for z<2

Full reference: findings/methodology/03-shuffled-null-design.md.


4. Agent significance corrections (z-scores are overstated 2-3×)

LLM agents systematically overstate z-scores. Treat agent-reported p-values as upper bounds.

Two overstatement patterns:

  1. Selection reported as a single test: agent tries 25 variants (or picks the top combo from 17,280) and reports the winner's in-sample z as if it were the only test. The winner is a hypothesis, not a finding. Do NOT repair this with a trial-count correction (Bonferroni, a sqrt(2 ln K) null-max, the Deflated Sharpe Ratio, the minimum backtest length (MinBTL), FDR/FWER, e-LOND): this repository does not compute, report, rank by or cite them (issue #35). Instead register the winner (rule, threshold, window) and re-test it on data no variant was selected on; report that z.
  2. Confused sample-mean z with binomial-proportion z: 53.5% vs 50% on N=840 gives z≈2.0 not 4.2.

Always verify:

  • How many variants did the agent try before reporting one? If more than one, was the reported rule confirmed on unused data?
  • Re-derive z yourself: (real - null.mean) / null.std
  • Was the hypothesis (rule, threshold, window) fixed before the run? Judge parsimony on the model itself, not by the number of trials.

Trust thresholds:

  • z > 5, N > 500: likely real, test further
  • z in [3, 5]: promising, mandatory gate validation
  • z in [2, 3]: suspect, require adjacent-cell gradient + null test
  • z < 2: treat as null

Full reference: findings/methodology/09-agent-significance-corrections.md.


5. Record-keeping discipline (append-only ledger + audit folders)

Every investigation — positive or null — must produce a permanent, discoverable record.

3-layer architecture:

findings/
├── evolution/
│   ├── evolution.jsonl              # append-only ledger
│   └── audits/
│       └── YYYY-MM-DD-slug/
│           ├── CLAUDE.md            # navigator
│           ├── verdict.md           # plain-English conclusion
│           ├── CHRONICLE.md         # narrative (for major findings)
│           ├── <reproducer>.py      # script that regenerates headline numbers
│           └── <artifact>.json      # raw telemetry
└── methodology/                     # universal principles

Ledger entry fields: id, date, status, supersedes, superseded_by, headline, key_numbers, evidence (file paths), sha256_results.

The supersedes pattern: when a later finding replaces an earlier one, ADD a new entry with supersedes: "OLD-ID"; UPDATE the old entry with superseded_by: "NEW-ID". Do NOT delete the older audit folder.

Full reference: findings/methodology/07-record-keeping-discipline.md.


6. Post-mortem-before-abandon

Before declaring a signal dead, enrich every trade with causal pre-entry features and hunt filters on individual losses. A "sometimes works" signal is often a filterable signal in disguise.

Pipeline:

  1. Run the signal across full history; collect N trade outcomes
  2. Compute ~20-30 causal features at each trigger bar
  3. Emit per-trade parquet + CSV (one row per trade)
  4. Ship to multi-lens agents (see Skill B)
  5. Each agent hunts filters that separate winners from losers
  6. Evaluate filters against shuffled-null (see §3)

Kill-selectivity metric: losers_killed / max(1, winners_killed). < 1.0 = harmful; 1.0-1.2 = marginal; 1.2-1.5 = useful; > 1.5 = strong.

Session example: +0.178 bps baseline → +0.514 bps after Phase-L filter. 2.9× lift from enrichment-driven filter hunt.

Full reference: findings/methodology/06-per-trade-enrichment-postmortem.md.


Confirmation counts (provisional, as of session ca9d7ffa)

PrincipleConfirmedNotes
1. causal-feature-invariant18+ (every phase)Fundamental; drop only with proof
2. label-leakage2Directly caught spurious "lower-rejection-at-bottom"
3. shuffled-null-design4Phase F-B wrong-null, Phase C right-null, Phase L filter-null, Phase M mgmt-null
4. agent-sig-corrections5+Combinatorialist, transition-asymmetry, trade-mgmt agents all overstated
5. record-keeping5 ledger entriesFull chain for NGRAM3FU-STRADDLE
6. post-mortem1Phase L delivered the filter; needs re-confirmation on other campaigns

Higher confirmed = more trustworthy. Principle 6 has only one confirmation and should be treated as provisional.


Post-Execution Reflection

After invoking this skill:

  1. Did applying a principle catch a bug or false positive? Increment its confirmed count in the table above; note the session where it fired in references/evolution-log.md.
  2. Did a principle fail (bad guidance)? Demote it in the table; add a superseded_by pointer in references/archive/ with resurrect_if: conditions.
  3. New trap that isn't covered? Draft a new section here and append to the evolution log.
  4. Never silently move on. This skill's value compounds only if reality-corrections flow back.
aus demselben Repository

Weitere Skills

Alle Skills
terrylica
Community

academic-pdf-to-gfm

Convert academic PDF papers to GitHub-renderable GFM markdown with math equations. TRIGGERS - PDF, GitHub markdown, math

Installationen / 1h
1
GitHub Stars
2
Aktualisiert
5. Nov.
terrylica
Community

adr-code-traceability

Add ADR references to code for traceability. TRIGGERS - ADR traceability, code reference, document decision in code.

Installationen / 1h
1
GitHub Stars
2
Aktualisiert
5. Nov.
terrylica
Community

arxiv-source-first-paper-ingest

Read an arXiv paper faithfully by fetching the authors' own LaTeX source instead of OCR-ing the PDF. Extracts every display equation with its label, environment and enclosing proposition, renders each to MathML, and proves usability by compiling each formula with a real TeX engine. Use when you need a paper's formulas exactly — to replicate a method, implement a loss function or custom objective, verify a quoted number, or check a claim against what the authors actually wrote. TRIGGERS - read th

Installationen / 1h
1
GitHub Stars
2
Aktualisiert
5. Nov.
terrylica
Community

azure-provision

Bootstrap Azure automation when interactive az login is blocked by a Conditional Access policy (error 530035, "Device state: Unregistered"). Mints a service principal through the Azure Portal session ONCE — using Microsoft Graph over a captured token, not DOM clicking — then provisions resource groups, Cognitive Services / AI Language / Speech resources and their keys entirely via az/REST, writing key AND endpoint together into the SCS vault. Use when az login fails but the portal works, when mo

Installationen / 1h
1
GitHub Stars
2
Aktualisiert
5. Nov.