affaan-m/ecc

benchmark-optimization-loop

Turn a user's improvement goal into a bounded benchmark-guided loop: define success and hard constraints, establish a baseline, test one-hypothesis variants, and promote the best measured admissible result with reproducible commands.

Vedi sorgente

Come usare

  1. Copia il comando e controlla il pacchetto prima di installarlo nel tuo ambiente agente.
  2. Controlla strumenti, credenziali e ambienti richiesti nel documento originale.
  3. Descrivi input e risultato desiderato; confronta l’output con le fonti.
npx skills add https://github.com/affaan-m/ecc --skill benchmark-optimization-loop

Prima di iniziare

  • L’installazione non include abbonamenti, quote API o accesso ai file.
  • Stelle e installazioni indicano attività, non un test indipendente di qualità o sicurezza.
Documento Skill originale

Contenuto dal repository con titoli, esempi, codice, tabelle, link e immagini preservati.

Benchmark Optimization Loop

Start with what the user needs to accomplish, then choose the benchmark. A request to improve correctness or quality need not become a speed optimization. Requests such as "make it 20x faster" or "try 50 recursive optimizations" still need a bounded, measurable loop.

When to Activate

Use for goal-driven implementation changes that need repeated measured comparisons, or to establish the benchmark and plan for such a task. Reuse an existing adequate regression check for a one-off fix; do not add a search loop without a reason.

Tool Requirements

Use the host's already-authorized Read, Write, Edit, Bash, Grep, and Glob tools or their equivalents. Loading this workflow does not grant tools or expand existing permissions.

Required Baseline

Do not optimize until these exist:

  • the user task, operation being optimized, and desired outcome;
  • the success threshold and correctness, quality, safety, or behavior constraints

that must stay green;

  • the primary metric, its direction and definition: for example discrepancy

amount, task completion rate, rubric score, wall time, p95 latency, rows/sec, cost/resolved task, memory, or error rate;

  • representative inputs and independent expected results or a calibrated rubric;
  • the current baseline with implementation, input, evaluator, and environment

versions sufficient to rerun it;

  • the search budget: max variants, max time, max spend, max data impact.

Reuse confirmed requirements and authorization. If a material success threshold or tradeoff is missing, propose a concrete choice and obtain the user's decision. If the user asks for an unrealistic target, keep the ambition but make the loop bounded and measurable. A plan-only request stays a plan; paid calls, production load, and deployment retain their existing authorization boundaries.

Freeze the metric, sample set, expected results, and evaluator for the comparison. Expected results must not be copied from the candidate implementation. Evaluate hard constraints separately from aggregate scores. For subjective quality, calibrate the rubric against accepted examples and record reviewer disagreement; the optimizing agent's self-score alone is not acceptance evidence.

Record why the metric represents the user outcome. A proxy may help diagnosis, but improving it does not by itself prove that the user's task improved.

Loop

  1. Measure the baseline, reusing reliable prior evidence only when comparable.
  2. Identify the gap from evidence relevant to the user goal.
  3. Generate variants that test one hypothesis each.
  4. Run variants with the same input shape and evaluation rules under comparable

conditions; record failures, timeouts, retries, and their costs.

  1. Reject variants that fail correctness, other hard constraints, safety, or

reproducibility.

  1. Promote the best measured admissible variant against the agreed objective.
  2. Codify the winning path in a script, command, test, config, or doc.
  3. Rerun the baseline and winner to confirm the delta, then perform the agreed

independent acceptance check before declaring the goal met.

Keep a ledger for every run: hypothesis, exact command, implementation and benchmark versions, case counts, primary result, hard-constraint results, failures/retries, evidence location, and the keep/reject decision. Failures remain in task-level denominators. If a statistic covers successes only, report failure rates and timeouts beside it; distinguish single attempts from complete tasks.

Optimize the implementation rather than quietly lowering the standard. If a sample or evaluator is wrong, document the evidence. A revision that changes the meaning of success needs user confirmation; version it and rerun both baseline and candidate under the revised rules, preserving the original evidence.

Variant Tables

The following values are illustrative, not measured evidence or default targets. For a speed goal, track variants like this:

text
Variant | Hypothesis | Command | Time | Correct? | Notes
baseline | current path | npm run job | 120s | yes | stable
batch-500 | fewer round trips | npm run job -- --batch 500 | 42s | yes | winner
parallel-8 | more workers | npm run job -- --workers 8 | 31s | no | rate limited

For a report-correctness goal, independently supplied reconciliation totals could define the primary measure; faster runtime would not excuse wrong totals:

text
Variant | Hypothesis | Critical mismatches | All periods processed? | Decision
baseline | current mapping | 4 | yes | baseline
mapping-fix | handle missing category | 0 | yes | candidate for acceptance
skip-errors | omit failing periods | 0 | no | reject: incomplete task

Recursive Search

For recursive or hyperparameter work:

  • persist every run to a ledger;
  • compare against the prior accepted winner, not only the previous run;
  • reserve independent acceptance cases where feasible, controlling answer

visibility; a known replay check is regression evidence, not an independent generalization check;

  • once acceptance feedback informs further tuning, treat those cases as

development evidence and obtain fresh independent cases for final acceptance;

  • stop when improvement is within noise, correctness or another hard constraint

fails, cost exceeds the budget, or the search starts changing more variables than it can explain; retain the prior accepted winner and report the gap;

  • stop optional search when the requested goal is achieved.

Use phrases like "best measured admissible variant" instead of "global optimum" unless the search space was actually exhaustive. If independent cases are unavailable, state that limitation instead of claiming independent acceptance.

Promotion Gate

A variant cannot become the new default until:

  • correctness tests and all other hard constraints pass;
  • the objective's measured delta is repeated or explained;
  • the agreed acceptance check supports the user goal, with untested paths and

proxy-only results explicitly identified;

  • rollback is obvious;
  • the change is encoded in source control or a durable runbook;
  • the final summary includes exact commands, versions, measurements, and limits.

Report benchmark success, independent acceptance, and deployed user outcomes separately. If deployment is authorized, follow the project's release process and verify the deployed version against the same user-outcome definition; otherwise report the candidate's pre-deployment status. This skill does not install a scheduler or grant release authority.

Related Skills and Sources

Reuse related skills when available in the host's installed profile; optional references below link to ECC's canonical sources.

  • benchmark collects browser/API/build performance

baselines; reuse it when those measurements serve the chosen goal.

evals and grader methods; reuse an appropriate existing evaluator.

loops and runaway prevention; this skill conducts the measured search.

for competitive analysis, rather than implementation variants.

dallo stesso repository

Altri Skills

Tutti gli Skills
affaan-m
Community

rust-testing

Rust testing patterns including unit tests, integration tests, async testing, property-based testing, mocking, and coverage. Follows TDD methodology. Use when writing Rust tests — unit, integration, async, property-based, or coverage.

Installazioni / 24h
489
GitHub Stars
308,2K
Aggiornato
20 nov
affaan-m
Community

search-first

Research-before-coding workflow: search npm/PyPI, MCP servers, skills, and GitHub for existing tools before writing custom code, then adopt, extend, or build. Launches the researcher agent for non-trivial needs. Use when starting a feature, adding a dependency or integration, or about to write a utility that may already exist.

Installazioni / 1h
5
GitHub Stars
308,2K
Aggiornato
20 nov
affaan-m
Community

browser-qa

Run automated post-deploy UI verification with a browser automation MCP (claude-in-chrome, Playwright, or Puppeteer): console-error and Core Web Vitals smoke checks, form and auth-flow interaction tests, screenshot visual regression across three breakpoints, and axe-core accessibility audits ending in a SHIP / DO-NOT-SHIP verdict. Use when testing a deployed feature on staging or preview, before shipping frontend changes, reviewing a frontend PR, or checking responsive layout and accessibility.

Installazioni / 1h
2
GitHub Stars
307,9K
Aggiornato
20 nov
affaan-m
Community

docker-patterns

Docker and Docker Compose patterns for local development, hardened CLI installer harnesses, container security, networking, volumes, and multi-service orchestration. Use when creating or reviewing Dockerfiles and Compose services, testing installers across Linux distributions, or planning accurate native macOS and Windows validation.

Installazioni / 1h
2
GitHub Stars
307,9K
Aggiornato
20 nov