What is a Pipeline?
A pipeline is a sequence of blocks that execute in order. Each block performs one specific task, and data flows through the pipeline via named slots.
HttpRequest → JsonParser → Validate → LLMJudge → Assert
- Readable: Each block has a clear purpose
- Reusable: Mix and match blocks for different scenarios
- Maintainable: Easy to add, remove, or reorder steps
Basic Pipeline Structure
{
"tests": [{
"id": "my-test",
"pipeline": [
{
"id": "step1",
"block": "HttpRequest",
"input": { "url": "https://api.example.com" },
"output": "response"
},
{
"id": "step2",
"block": "JsonParser",
"input": "${response.body}",
"output": "data"
}
],
"assertions": {
"response.status": 200
}
}]
}
- id: Unique identifier for this step
- block: The block type to use
- input: Data to process (from previous blocks)
- output: Where to store the result
- config (optional): Block-specific configuration
Execution Flow
Initialize
Pipeline loads and prepares all blocks
Execute in Sequence
Each block runs one after another:
- Reads inputs from the DataBus
- Processes data
- Writes outputs to the DataBus
Validate Assertions
After all blocks complete, assertions are checked
Report Results
Success or failure with detailed information
Why Pipelines?
Composability
Build complex tests from simple blocks:
{
"pipeline": [
{ "block": "HttpRequest" }, // Fetch data
{ "block": "JsonParser" }, // Parse response
{ "block": "ValidateContent" }, // Check content
{ "block": "LLMJudge" }, // Semantic validation
{ "block": "ValidateTools" } // Check tool calls
]
}
Reusability
Use the same blocks in different combinations:
// Test 1: Simple validation
{
"pipeline": [
{ "block": "HttpRequest" },
{ "block": "ValidateContent" }
]
}
// Test 2: Semantic validation
{
"pipeline": [
{ "block": "HttpRequest" },
{ "block": "LLMJudge" }
]
}
Maintainability
Easy to modify tests:
// Add a new step
{
"pipeline": [
{ "block": "HttpRequest" },
{ "block": "JsonParser" },
{ "block": "ValidateContent" }, // ← New step
{ "block": "LLMJudge" }
]
}
Multiple Tests
You can run multiple tests in sequence:
{
"name": "User API Tests",
"tests": [
{
"id": "create-user",
"pipeline": [ /* ... */ ]
},
{
"id": "get-user",
"pipeline": [ /* ... */ ]
},
{
"id": "update-user",
"pipeline": [ /* ... */ ]
}
]
}
Each test runs in isolation with its own DataBus. Tests don’t share data unless you use setup/teardown.
Setup and Teardown
For shared setup across tests:
{
"setup": [
{
"id": "auth",
"block": "HttpRequest",
"input": {
"url": "https://api.example.com/auth/login",
"method": "POST",
"body": { "username": "test", "password": "test123" }
},
"output": "authToken"
}
],
"tests": [
{
"id": "test1",
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "https://api.example.com/users",
"headers": {
"Authorization": "Bearer ${authToken.body.token}"
}
}
}
]
}
],
"teardown": [
{
"id": "logout",
"block": "HttpRequest",
"input": {
"url": "https://api.example.com/auth/logout",
"method": "POST"
}
}
]
}
Next Steps
