parallel-web/parallel-agent-skills

parallel-search-evaluation

Use for search evaluation, a POC or bake-off, Parallel vs Exa/Tavily comparisons, SimpleQA/DeepSearchQA benchmarks, or diagnosing search evaluation results.

소스 보기

사용 방법

  1. 설치 명령을 복사하고 자신의 에이전트 환경에 설치하기 전에 소스 패키지를 확인하세요.
  2. 필요한 도구, 인증 정보, 지원 환경은 원본 문서에서 확인하세요.
  3. 입력과 원하는 산출물을 설명하고 출처와 결과를 대조하세요.
npx skills add https://github.com/parallel-web/parallel-agent-skills --skill parallel-search-evaluation

시작 전 확인

  • 설치에는 모델 구독, API 한도, 파일 접근 권한이 포함되지 않습니다.
  • 별점 수와 설치 수는 활동 지표이며 독립적인 품질·보안 검증이 아닙니다.
원본 Skill 문서

원본 저장소의 제목, 예시, 코드, 표, 링크, 이미지를 유지해 표시합니다.

Evaluate Parallel Search

Turn the user's workload into a reproducible end-to-end comparison. Deliver a runnable harness when implementation is requested, or an actionable plan when planning is requested. Reuse an existing harness before creating another one. Do not invent a Parallel CLI eval command.

To choose a Parallel API or build the integration, use parallel-api. To replace another provider's integration, use migrate-to-parallel. Use this skill to measure quality, cost and latency.

<!-- markdownlint-disable MD033 MD034 -->

Onboarding

Follow this section when a Parallel setup prompt sends you here, or when the user is evaluating Parallel or comparing it with other providers.

  • Work quietly: give progress updates only for decisions the user must make. Use the quoted messages below word for word, filling in <placeholders>. Anywhere else, write at most two short lines.
  • Ask one question at a time, with your question UI if you have one.
  1. When the comparison needs one of these decisions and the project doesn't answer it, ask:

"What should Parallel be compared with? For example: Exa, Tavily, your current setup, or a public benchmark." "Do you have real questions from your app, with known answers?

  1. Yes: tell me where they are.
  2. No: start from a public benchmark."
  3. API key: if PARALLELAPIKEY is set in the shell or in the project's secret file, use that key (check without printing it). Otherwise, send:

"Live runs need a Parallel API key:

  1. Sign in or sign up at https://platform.parallel.ai and create a key.
  2. Add PARALLELAPIKEY=<your key> to <file>.
  3. Reply done, or reply skip to get the plan and the test harness without live runs."

<file> is the project's existing secret file. If there is none, create .env and add .env to .gitignore first.

  1. Ask: "Want a quick look first?
  2. Spot check: I'll run 5 of your questions through Parallel and <provider>, side by side.
  3. Full evaluation: a fixed question set, repeated runs, graded results.

A spot check doesn't replace the full evaluation."

  1. In its own message, report quality, cost, latency and the main failure types in at most ten lines, with the path to the full report.
  2. If Parallel caused a problem (this skill, the docs, the API, the key or the skill install), ask in a separate message: "One issue came up during the evaluation: <one-line description>. Would you like a note to send to support@parallel.ai?" Skip problems in the user's own project.

<!-- markdownlint-enable MD033 MD034 -->

Start with the workload

Read the public evaluation guide, Search best practices, and Extract best practices before implementing requests. Check current modes and pricing; record the documentation date and configuration rather than embedding price or latency promises.

Establish the application, question types, providers, existing agent/model, verified answers, and quality/cost/latency target. Infer these from supplied code and data; ask only for missing decisions that change the comparison. Prepare offline artifacts while credentials or a live-run budget remain unresolved. Use existing authorization for paid calls; otherwise establish scope and spend limits before execution. Never request keys in chat.

Choose the product that matches the application:

ApplicationEvaluation boundary
User owns the agent loopSearch + Extract, including answer generation
One call should return a cited answerResponses API
Schema-bound enrichment or asynchronous researchTask API

Always set Search mode explicitly for REST requests: omitting it selects advanced, changing latency and price. Unless the user specifies otherwise, start with fast for interactive workloads or advanced for harder research, then test another relevant mode. The evaluation guide still recommends basic; record this discrepancy and the chosen rationale rather than silently inheriting that starting point. Honor workload constraints such as language support and endpoint availability. For Search MCP, consult its current routing and override rules: pin the mode for authenticated calls where supported; anonymous calls reject overrides, so record the actual server-selected mode or require authentication for a controlled mode comparison. Keep the model fixed within a comparison; a different model/mode pairing is a separate experiment.

Build and freeze the comparison

Use references/eval-contract.md when creating dataset records, run manifests, traces, grading output, and reports. Resolve this reference relative to this skill's directory.

  1. Prefer real production questions and independently verified answers. Match domain, wording, freshness, and task shape. Review synthetic labels before running. Pin question IDs and dataset version; separate pilot questions from held-out questions. Public benchmarks supplement this set and need a consistent policy for answer-key contamination.
  2. Hold the model, research prompt, budgets, judge, and extraction policy constant across providers. Expose only the selected provider's search tool. Start with each provider's documented defaults and tool definition, including descriptions; change a setting only with evidence from the workload or pilot, and record the evidence and deviation in the manifest. Avoid arbitrary result caps or custom tool descriptions. If extraction differs as part of a provider stack, disclose that boundary. Keep gold answers and annotations out of agent prompts, query generation, and tool messages.
  3. Have the model write an objective that restates the full information need with every relevant constraint, including entities, scope, dates, and source requirements. Pair it with focused keyword queries of 3–6 words each: one for a simple fact, up to three useful angles for richer questions. Do not duplicate the query text in the objective or copy the whole task into each query. Preserve result titles, URLs, and excerpts in the answering model's context. Use /v1/search and /v1/extract for new integrations; verify current request schemas rather than renaming old endpoints.
  4. Default to multi-hop Search + Extract for applications; use single-hop only for an explicitly one-request benchmark. Fix total calls, deadline, retry policy, and spend bounds before dispatch. Use a fresh session per question, configuration, and repetition; reuse it across that trial's related Search and Extract calls.

Recommend a cost-parity comparison: align result count, content per page, product tier, and reasoning level where supported, and account for total Search/Extract/model cost including input and output tokens. Start with documented defaults as the baseline; any parity overrides need a recorded rationale and a separate configuration. Report requested and actual result counts and content volumes, residual mismatches, and settings that cannot be equated. Do not treat equal API-call counts as equal cost or imply parity when it was not achieved.

SettingExecutor behavior
Single-hopAllow exactly one Search API request, no Extract; disable tools for the final answer. Enforce the limit in code, including concurrent requests. Several keyword queries in one request still count as one call.
Multi-hopExpose Search + Extract and allow evidence-driven follow-ups within the total budget. Dispatch independent calls from the same turn concurrently; reserve budget before dispatch, return results under their original call IDs, and preserve successes when siblings fail.

Use the same research prompt across providers and settings; change tool availability and executor budgets separately. For public benchmarks, use the guide's research prompt and official grader where available. Record output-format overrides and label altered benchmark protocols as adapted evaluations.

Pilot, grade, and interpret

Inspect a small fixed pilot: query quality, warnings, excerpt delivery, tool limits, concurrency, and trace completeness. Fix integration faults and rerun affected configurations; do not tune during a scored run. Freeze settings before evaluating held-out questions. Run each configuration at least three times and report variation.

Flag every failure to the user with affected trial IDs and impact: rate limits, HTTP errors, timeouts, partial results, token truncation, dropped results, and endpoint fallback (including /v1 → /v1beta). Record each occurrence even if a retry succeeds. Never silently fall back or count a Parallel-side failure as a retrieval miss. Surface failures during execution and include their counts and trace references in the final report; preserve partial successes and the declared scoring denominator.

Grade final answers by task shape: factual correctness and citation support; list precision and recall; structured field accuracy; or an explicit research rubric. Require judge reasoning. For custom rubrics, rank honest inability to retrieve above confident misinformation. Hand-check at least 10% of judged trials, covering successes and failures. Keep citation audits separate from official benchmark scores.

Classify failures from traces: no search call, synthesis failure despite sufficient evidence, provider error, or retrieval miss despite a reasonable query. Do not attribute missing excerpts or other harness faults to retrieval quality. Distinguish account rate limits from provider capability.

Include failures and timeouts in primary results under the declared scoring policy. Bound retries and include all their time and cost. Report Search/Extract/model charges, tokens, calls, budget exhaustion, and search and end-to-end latency at p50/p95/p99. Time the whole task by wall clock, including query reformulation and answer generation; do not sum concurrent call durations. Cost per resolved task includes spend on unsuccessful trials; report it as undefined if nothing resolves.

Deliver the reproducible configuration, dataset and trace locations, grading rationale, failure breakdown, and quality versus cost/latency comparisons. Recommend configurations on the measured Pareto frontier that meet the user's targets, with uncertainty and workload slices. If calls were not run, label the deliverable as an unexecuted plan or harness and state what remains needed.

After results, propose 2–3 follow-up configurations tied to observed failures or bottlenecks: retrieval misses → advanced; thin excerpts → enable or increase Extract use; high latency → fast; high model-token cost → a smaller content budget. Change one factor per comparison, record the rationale, and use a new frozen manifest. Present these as proposals, not automatically launched runs; execute only when the user requests them within the authorized scope and spend.

같은 저장소의 Skills

더 많은 Skills

모든 Skills
parallel-web
공식

parallel-api

Build apps and agents on the Parallel API: Search, Extract, Responses, Task, Decisions and Monitor, through the parallel-web Python and TypeScript SDKs or REST. Use when writing, reviewing or debugging code that calls Parallel; when choosing a Parallel API, search mode, reasoning effort or Task processor; when adding web search, page extraction, cited answers, research, enrichment, web-grounded decisions or web monitoring to an app; when swapping another search provider for Parallel; when settin

1시간 설치 수
2
GitHub Stars
95
업데이트
11월 17일
parallel-web
공식

parallel-cli-setup

Set up and maintain the Parallel CLI (install, auth, balance, skills install)

1시간 설치 수
2
GitHub Stars
94
업데이트
11월 16일
parallel-web
공식

parallel-findall

Discover entities (companies, people, products, etc.) matching a natural-language description. Use when the user asks to 'find all X' or 'list every Y that…' — e.g., 'Find AI startups that raised Series A in 2026', 'List roofing companies in Charlotte NC', 'Show me YC W24 dev tools companies'. Different from web-search (which returns webpages) and deep-research (which returns a narrative report). Use this when the user wants a structured list of entities.

1시간 설치 수
2
GitHub Stars
94
업데이트
11월 16일
parallel-web
공식

parallel-deep-research

ONLY use when user explicitly says 'deep research', 'exhaustive', 'comprehensive report', or 'thorough investigation'. Slower and more expensive than parallel-web-search. For normal research/lookup requests, use parallel-web-search instead. Supports follow-ups with a known prior Task interaction ID.

1시간 설치 수
3
GitHub Stars
93
업데이트
11월 11일