In Aktion ansehen
Promptfoo: compare outputs and assertion failures side by side
Original documentation viewer image with twelve demonstration cases, Model A columns, outputs and assertion explanations. Passing percentages, token/latency figures and the 2025 eval ID belong to the author demo; they are not current-model rankings, real provider benchmarks or results of the separate recipes below.
Lizenz
Full original license retained. Author interface previews, usage recipes and illustrative/expected output remain distinct from independently reproduced results. No source instructions, scripts, installations, models, API requests, accounts, memory writes, schedules, mail or browser tasks were executed. Native-resolution source image converted to lossless WebP with complete decoded RGBA equality; no crop, retouch, upscaling or replacement of source labels.
Copyright (c) Promptfoo 2025
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
Promptfoo: inspect pass rates and score distributions
Original documentation chart illustration for pass rate, score distribution and pairwise prompt scores. These figures explain viewer controls, not a measured benchmark or evidence that one named model is better. The complete source image is retained, with no invented underlying dataset.
Lizenz
Full original license retained. Author interface previews, usage recipes and illustrative/expected output remain distinct from independently reproduced results. No source instructions, scripts, installations, models, API requests, accounts, memory writes, schedules, mail or browser tasks were executed. Native-resolution source image converted to lossless WebP with complete decoded RGBA equality; no crop, retouch, upscaling or replacement of source labels.
Copyright (c) Promptfoo 2025
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
Promptfoo: complete structured-output evaluation config
Complete author YAML for two math cases and alternative JSON response schemas. Keep only providers you can access, configure their keys and replace Azure host/deployment placeholders. Model names, API version and schema options are source configuration requiring current provider support; expected assertions are not passing test results. No evaluation was run.
Beispieleingabe
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: Multi-provider structured JSON outputs with schema validation
prompts:
# Schema output with `steps` array
- label: 'With step-by-step'
raw: 'You are a helpful math tutor. Solve {{problem}}'
config:
response_format:
type: json_schema
json_schema:
name: math_response
strict: true
schema:
type: object
properties:
steps:
type: array
items:
type: object
properties:
explanation:
type: string
output:
type: string
required:
- explanation
- output
additionalProperties: false
final_answer:
type: string
required:
- steps
- final_answer
additionalProperties: false
# Schema output without `steps` array
- label: 'Without step-by-step'
raw: 'You are a helpful math tutor. Solve {{problem}}'
config:
response_format:
type: json_schema
json_schema:
name: math_response
strict: true
schema:
type: object
properties:
final_answer:
type: string
required:
- final_answer
additionalProperties: false
providers:
# OpenAI with structured outputs (response_format)
- openai:chat:gpt-4o-mini
# Azure OpenAI with structured outputs (response_format)
- id: azure:chat:gpt-4o-2024-08-06
config:
apiHost: 'YOUR_ORG.openai.azure.com'
apiVersion: '2024-08-01-preview'
deploymentName: 'YOUR_DEPLOYMENT_NAME'
# Anthropic with structured outputs (output_format) - with steps
- id: anthropic:messages:claude-sonnet-5
label: 'Anthropic (with steps)'
config:
max_tokens: 2048
output_format:
type: json_schema
schema:
type: object
properties:
steps:
type: array
items:
type: object
properties:
explanation:
type: string
output:
type: string
required:
- explanation
- output
additionalProperties: false
final_answer:
type: string
required:
- steps
- final_answer
additionalProperties: false
# Anthropic with structured outputs (output_format) - without steps
- id: anthropic:messages:claude-sonnet-5
label: 'Anthropic (without steps)'
config:
max_tokens: 2048
output_format:
type: json_schema
schema:
type: object
properties:
final_answer:
type: string
required:
- final_answer
additionalProperties: false
defaultTest:
assert:
- type: is-json
- type: javascript
value: typeof output === 'object'
tests:
- description: Solving for the number of dragons in a cave
vars:
problem: 'If a wizard has 8x dragons and finds 31 more in a cave, but only has room for 2 total, how many dragons must the wizard release? (8x + 31 = 2)'
assert:
- type: contains-any
value:
- -3.625
- -29/8
- '-3.625'
- '-29/8'
- description: Calculating the area of a perfectly square pizza
vars:
problem: 'A perfectly square pizza has a side length of sqrt(x) meters. If the side length is 4.9 meters, what is the area in square meters? (sqrt(x) = 4.9)'
assert:
- type: contains-any
value:
- 24.01
- '24.01'
Lizenz
Full original license retained. Author interface previews, usage recipes and illustrative/expected output remain distinct from independently reproduced results. No source instructions, scripts, installations, models, API requests, accounts, memory writes, schedules, mail or browser tasks were executed. Complete source files/blocks retained. Only outer fences or documented indentation removed; multi-file recipes use editorial path labels and fences, with each separate binding in the review. Source versions/model aliases, template syntax and placeholders are unchanged.
Copyright (c) Promptfoo 2025
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
Promptfoo: evaluate separate multi-turn conversation threads
Complete paired YAML and prompt.json Nunjucks template. Save both files together, configure provider access and run in the matching Promptfoo setup. The .json file intentionally contains template loops, not standalone JSON. Conversation IDs separate the example threads; no model answers, factual accuracy or current provider compatibility were verified here.
Beispieleingabe
examples/config-multi-turn/promptfooconfig.yaml
```yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: Multi-turn conversation evaluation and testing
prompts:
- id: file://prompt.json
label: multi-turn-conversation
providers:
- openai:gpt-4.1-mini
# Set up the system message for all tests
defaultTest:
vars:
system_message: You are a helpful assistant
# Test out conversations with multiple turns
tests:
# First conversation about Facebook
- vars:
question: Who founded Facebook?
metadata:
conversationId: facebook_thread
- vars:
question: Where does he live?
metadata:
conversationId: facebook_thread
- vars:
question: Which state is that in?
metadata:
conversationId: facebook_thread
# Second conversation about weather (runs independently)
- vars:
question: What's the weather like in San Francisco?
metadata:
conversationId: weather_thread
- vars:
question: Do I need an umbrella today?
metadata:
conversationId: weather_thread
# Third conversation about food (runs independently)
- vars:
question: Whats a good recipe for pasta?
metadata:
conversationId: food_thread
- vars:
question: How long should I cook it?
metadata:
conversationId: food_thread
```
examples/config-multi-turn/prompt.json
```text
[
{
"role": "system",
"content": "{{ system_message }}"
},
{% for completion in _conversation %}
{
"role": "user",
"content": "{{ completion.prompt[completion.prompt.length - 1].content }}"
},
{
"role": "assistant",
"content": "{{ completion.output }}"
},
{% endfor %}
{
"role": "user",
"content": "{{ question }}"
}
]
```Lizenz
Full original license retained. Author interface previews, usage recipes and illustrative/expected output remain distinct from independently reproduced results. No source instructions, scripts, installations, models, API requests, accounts, memory writes, schedules, mail or browser tasks were executed. Complete source files/blocks retained. Only outer fences or documented indentation removed; multi-file recipes use editorial path labels and fences, with each separate binding in the review. Source versions/model aliases, template syntax and placeholders are unchanged.
Copyright (c) Promptfoo 2025
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
Was Promptfoo bietet
- 01Compare prompts and model outputs
- 02Define repeatable evaluation cases
- 03Run red-team checks and CI workflows
Was es unterscheidet
It organizes test cases and checks; meaningful results depend on the chosen datasets, assertions and providers. Evaluations using external APIs can send data to those providers.
Originaltext der Quelle · ENFür wen es geeignet ist
- Developers evaluating AI applications
- Teams maintaining repeatable model checks
Was zu prüfen ist
- Eine echte Aufgabe testen und das Ergebnis mit den Anforderungen vergleichen.
- Prüfen, welche Daten gespeichert, weitergegeben oder zum Training genutzt werden.
- Aktuelle Preise, Nutzungslimits und Kündigungsbedingungen auf der offiziellen Website prüfen.
Verwandte Aufgaben
Offizieller Link am 21. Okt. 2026 geprüft. Funktionsangaben wurden nicht unabhängig getestet.
