Pipelines

What is a Pipeline?

A pipeline is a sequence of blocks that execute in order. Each block performs one specific task, and data flows through the pipeline via named slots.

HttpRequest → JsonParser → Validate → LLMJudge → Assert

This architecture makes tests:

Readable: Each block has a clear purpose
Reusable: Mix and match blocks for different scenarios
Maintainable: Easy to add, remove, or reorder steps

Basic Pipeline Structure

{
  "tests": [{
    "id": "my-test",
    "pipeline": [
      {
        "id": "step1",
        "block": "HttpRequest",
        "input": { "url": "https://api.example.com" },
        "output": "response"
      },
      {
        "id": "step2",
        "block": "JsonParser",
        "input": "${response.body}",
        "output": "data"
      }
    ],
    "assertions": {
      "response.status": 200
    }
  }]
}

Each block has:

id: Unique identifier for this step
block: The block type to use
input: Data to process (from previous blocks)
output: Where to store the result
config (optional): Block-specific configuration

Execution Flow

Initialize

Pipeline loads and prepares all blocks

Execute in Sequence

Each block runs one after another:

Reads inputs from the DataBus
Processes data
Writes outputs to the DataBus

Validate Assertions

After all blocks complete, assertions are checked

Report Results

Success or failure with detailed information

Why Pipelines?

Composability

Build complex tests from simple blocks:

{
  "pipeline": [
    { "block": "HttpRequest" },      // Fetch data
    { "block": "JsonParser" },       // Parse response
    { "block": "ValidateContent" },  // Check content
    { "block": "LLMJudge" },         // Semantic validation
    { "block": "ValidateTools" }     // Check tool calls
  ]
}

Reusability

Use the same blocks in different combinations:

// Test 1: Simple validation
{
  "pipeline": [
    { "block": "HttpRequest" },
    { "block": "ValidateContent" }
  ]
}

// Test 2: Semantic validation
{
  "pipeline": [
    { "block": "HttpRequest" },
    { "block": "LLMJudge" }
  ]
}

Maintainability

Easy to modify tests:

// Add a new step
{
  "pipeline": [
    { "block": "HttpRequest" },
    { "block": "JsonParser" },
    { "block": "ValidateContent" },  // ← New step
    { "block": "LLMJudge" }
  ]
}

Multiple Tests

You can run multiple tests in sequence:

{
  "name": "User API Tests",
  "tests": [
    {
      "id": "create-user",
      "pipeline": [ /* ... */ ]
    },
    {
      "id": "get-user",
      "pipeline": [ /* ... */ ]
    },
    {
      "id": "update-user",
      "pipeline": [ /* ... */ ]
    }
  ]
}

Each test runs in isolation with its own DataBus. Tests don’t share data unless you use setup/teardown.

Setup and Teardown

For shared setup across tests:

{
  "setup": [
    {
      "id": "auth",
      "block": "HttpRequest",
      "input": {
        "url": "https://api.example.com/auth/login",
        "method": "POST",
        "body": { "username": "test", "password": "test123" }
      },
      "output": "authToken"
    }
  ],
  "tests": [
    {
      "id": "test1",
      "pipeline": [
        {
          "block": "HttpRequest",
          "input": {
            "url": "https://api.example.com/users",
            "headers": {
              "Authorization": "Bearer ${authToken.body.token}"
            }
          }
        }
      ]
    }
  ],
  "teardown": [
    {
      "id": "logout",
      "block": "HttpRequest",
      "input": {
        "url": "https://api.example.com/auth/logout",
        "method": "POST"
      }
    }
  ]
}

Setup runs once before all tests. Teardown runs once after all tests (even if tests fail).

What is a Pipeline?

Basic Pipeline Structure

Execution Flow

Why Pipelines?

Composability

Reusability

Maintainability

Multiple Tests

Setup and Teardown

Next Steps

Data Flow

Browse Blocks

​What is a Pipeline?

​Basic Pipeline Structure

​Execution Flow

​Why Pipelines?

​Composability

​Reusability

​Maintainability

​Multiple Tests

​Setup and Teardown

​Next Steps

Data Flow

Browse Blocks

What is a Pipeline?

Basic Pipeline Structure

Execution Flow

Why Pipelines?

Composability

Reusability

Maintainability

Multiple Tests

Setup and Teardown

Next Steps