実際の例を見る
Promptfoo: compare outputs and assertion failures side by side
Original documentation viewer image with twelve demonstration cases, Model A columns, outputs and assertion explanations. Passing percentages, token/latency figures and the 2025 eval ID belong to the author demo; they are not current-model rankings, real provider benchmarks or results of the separate recipes below.
ライセンス
Full original license retained. Author interface previews, usage recipes and illustrative/expected output remain distinct from independently reproduced results. No source instructions, scripts, installations, models, API requests, accounts, memory writes, schedules, mail or browser tasks were executed. Native-resolution source image converted to lossless WebP with complete decoded RGBA equality; no crop, retouch, upscaling or replacement of source labels.
Copyright (c) Promptfoo 2025
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
Promptfoo: inspect pass rates and score distributions
Original documentation chart illustration for pass rate, score distribution and pairwise prompt scores. These figures explain viewer controls, not a measured benchmark or evidence that one named model is better. The complete source image is retained, with no invented underlying dataset.
ライセンス
Full original license retained. Author interface previews, usage recipes and illustrative/expected output remain distinct from independently reproduced results. No source instructions, scripts, installations, models, API requests, accounts, memory writes, schedules, mail or browser tasks were executed. Native-resolution source image converted to lossless WebP with complete decoded RGBA equality; no crop, retouch, upscaling or replacement of source labels.
Copyright (c) Promptfoo 2025
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
Promptfoo: complete structured-output evaluation config
Complete author YAML for two math cases and alternative JSON response schemas. Keep only providers you can access, configure their keys and replace Azure host/deployment placeholders. Model names, API version and schema options are source configuration requiring current provider support; expected assertions are not passing test results. No evaluation was run.
入力例
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: Multi-provider structured JSON outputs with schema validation
prompts:
# Schema output with `steps` array
- label: 'With step-by-step'
raw: 'You are a helpful math tutor. Solve {{problem}}'
config:
response_format:
type: json_schema
json_schema:
name: math_response
strict: true
schema:
type: object
properties:
steps:
type: array
items:
type: object
properties:
explanation:
type: string
output:
type: string
required:
- explanation
- output
additionalProperties: false
final_answer:
type: string
required:
- steps
- final_answer
additionalProperties: false
# Schema output without `steps` array
- label: 'Without step-by-step'
raw: 'You are a helpful math tutor. Solve {{problem}}'
config:
response_format:
type: json_schema
json_schema:
name: math_response
strict: true
schema:
type: object
properties:
final_answer:
type: string
required:
- final_answer
additionalProperties: false
providers:
# OpenAI with structured outputs (response_format)
- openai:chat:gpt-4o-mini
# Azure OpenAI with structured outputs (response_format)
- id: azure:chat:gpt-4o-2024-08-06
config:
apiHost: 'YOUR_ORG.openai.azure.com'
apiVersion: '2024-08-01-preview'
deploymentName: 'YOUR_DEPLOYMENT_NAME'
# Anthropic with structured outputs (output_format) - with steps
- id: anthropic:messages:claude-sonnet-5
label: 'Anthropic (with steps)'
config:
max_tokens: 2048
output_format:
type: json_schema
schema:
type: object
properties:
steps:
type: array
items:
type: object
properties:
explanation:
type: string
output:
type: string
required:
- explanation
- output
additionalProperties: false
final_answer:
type: string
required:
- steps
- final_answer
additionalProperties: false
# Anthropic with structured outputs (output_format) - without steps
- id: anthropic:messages:claude-sonnet-5
label: 'Anthropic (without steps)'
config:
max_tokens: 2048
output_format:
type: json_schema
schema:
type: object
properties:
final_answer:
type: string
required:
- final_answer
additionalProperties: false
defaultTest:
assert:
- type: is-json
- type: javascript
value: typeof output === 'object'
tests:
- description: Solving for the number of dragons in a cave
vars:
problem: 'If a wizard has 8x dragons and finds 31 more in a cave, but only has room for 2 total, how many dragons must the wizard release? (8x + 31 = 2)'
assert:
- type: contains-any
value:
- -3.625
- -29/8
- '-3.625'
- '-29/8'
- description: Calculating the area of a perfectly square pizza
vars:
problem: 'A perfectly square pizza has a side length of sqrt(x) meters. If the side length is 4.9 meters, what is the area in square meters? (sqrt(x) = 4.9)'
assert:
- type: contains-any
value:
- 24.01
- '24.01'
ライセンス
Full original license retained. Author interface previews, usage recipes and illustrative/expected output remain distinct from independently reproduced results. No source instructions, scripts, installations, models, API requests, accounts, memory writes, schedules, mail or browser tasks were executed. Complete source files/blocks retained. Only outer fences or documented indentation removed; multi-file recipes use editorial path labels and fences, with each separate binding in the review. Source versions/model aliases, template syntax and placeholders are unchanged.
Copyright (c) Promptfoo 2025
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
Promptfoo: evaluate separate multi-turn conversation threads
Complete paired YAML and prompt.json Nunjucks template. Save both files together, configure provider access and run in the matching Promptfoo setup. The .json file intentionally contains template loops, not standalone JSON. Conversation IDs separate the example threads; no model answers, factual accuracy or current provider compatibility were verified here.
入力例
examples/config-multi-turn/promptfooconfig.yaml
```yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: Multi-turn conversation evaluation and testing
prompts:
- id: file://prompt.json
label: multi-turn-conversation
providers:
- openai:gpt-4.1-mini
# Set up the system message for all tests
defaultTest:
vars:
system_message: You are a helpful assistant
# Test out conversations with multiple turns
tests:
# First conversation about Facebook
- vars:
question: Who founded Facebook?
metadata:
conversationId: facebook_thread
- vars:
question: Where does he live?
metadata:
conversationId: facebook_thread
- vars:
question: Which state is that in?
metadata:
conversationId: facebook_thread
# Second conversation about weather (runs independently)
- vars:
question: What's the weather like in San Francisco?
metadata:
conversationId: weather_thread
- vars:
question: Do I need an umbrella today?
metadata:
conversationId: weather_thread
# Third conversation about food (runs independently)
- vars:
question: Whats a good recipe for pasta?
metadata:
conversationId: food_thread
- vars:
question: How long should I cook it?
metadata:
conversationId: food_thread
```
examples/config-multi-turn/prompt.json
```text
[
{
"role": "system",
"content": "{{ system_message }}"
},
{% for completion in _conversation %}
{
"role": "user",
"content": "{{ completion.prompt[completion.prompt.length - 1].content }}"
},
{
"role": "assistant",
"content": "{{ completion.output }}"
},
{% endfor %}
{
"role": "user",
"content": "{{ question }}"
}
]
```ライセンス
Full original license retained. Author interface previews, usage recipes and illustrative/expected output remain distinct from independently reproduced results. No source instructions, scripts, installations, models, API requests, accounts, memory writes, schedules, mail or browser tasks were executed. Complete source files/blocks retained. Only outer fences or documented indentation removed; multi-file recipes use editorial path labels and fences, with each separate binding in the review. Source versions/model aliases, template syntax and placeholders are unchanged.
Copyright (c) Promptfoo 2025
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
Promptfooでできること
- 01Compare prompts and model outputs
- 02Define repeatable evaluation cases
- 03Run red-team checks and CI workflows
他と異なる点
It organizes test cases and checks; meaningful results depend on the chosen datasets, assertions and providers. Evaluations using external APIs can send data to those providers.
出典の原文 · EN検討に向く人
- Developers evaluating AI applications
- Teams maintaining repeatable model checks
確認すること
- 実際の作業を一つ試し、要件に合うか比較する。
- 保存・共有・学習に利用されるデータを確認する。
- 公式サイトで現在の料金、利用制限、解約条件を確認する。
関連するタスク
公式リンクは2026年10月21日に確認しました。機能の主張は独立して検証していません。
