Data Flow

The DataBus

The DataBus is a key-value store that holds all data during test execution. Think of it as a shared memory where blocks can read and write data using named slots.

┌─────────────┐
│   DataBus   │
├─────────────┤
│ response    │ ← HttpRequest writes here
│ parsed      │ ← JsonParser writes here
│ validation  │ ← Validate writes here
└─────────────┘

Writing to the DataBus

Blocks write their output to named slots using the output field:

{
  "block": "HttpRequest",
  "input": { "url": "https://api.example.com/users/1" },
  "output": "response"  // ← Writes to 'response' slot
}

After this block executes, the DataBus contains:

{
  response: {
    status: 200,
    body: '{"id": 1, "name": "John"}',
    headers: { /* ... */ }
  }
}

Reading from the DataBus

Use ${} syntax to read from slots:

{
  "block": "JsonParser",
  "input": "${response.body}",  // ← Reads from 'response.body'
  "output": "parsed"
}

You can access nested properties using dot notation:

"input": "${response.headers.content-type}"
"input": "${parsed.data.user.email}"
"input": "${validation.passed}"

Variable Resolution

The DataBus resolves variables when blocks execute:

{
  "pipeline": [
    {
      "id": "fetch",
      "block": "HttpRequest",
      "input": { "url": "${BASE_URL}/users" },  // ← Resolves ${BASE_URL}
      "output": "users"
    },
    {
      "id": "process",
      "block": "JsonParser",
      "input": "${users.body}",  // ← Resolves ${users.body}
      "output": "parsed"
    }
  ]
}

Resolution Order

Variables are resolved from multiple sources in this priority:

Environment - From .env file (accessed via ${env.VARIABLE})
Context - Defined in the test file
DataBus - From previous blocks

{
  "context": {
    "BASE_URL": "${env.API_URL}",  // From .env
    "API_KEY": "hardcoded-key"
  },
  "tests": [{
    "pipeline": [
      {
        "block": "HttpRequest",
        "input": {
          "url": "${BASE_URL}/data",        // From context
          "headers": {
            "Authorization": "${API_KEY}"    // From context
          }
        },
        "output": "response"
      },
      {
        "block": "JsonParser",
        "input": "${response.body}",         // From DataBus
        "output": "data"
      }
    ]
  }]
}

Data Transformation

Data can be transformed as it flows:

{
  "pipeline": [
    {
      "block": "HttpRequest",
      "output": "raw"           // Raw HTTP response
    },
    {
      "block": "JsonParser",
      "input": "${raw.body}",
      "output": "parsed"        // Parsed JSON
    },
    {
      "block": "ValidateContent",
      "input": {
        "from": "parsed.message",
        "as": "text"
      },
      "output": "validation"    // Validation result
    }
  ]
}

DataBus state after each block:

After HttpRequest

{
  raw: {
    status: 200,
    body: '{"message": "Hello"}',
    headers: {}
  }
}

After JsonParser

{
  raw: { /* ... */ },
  parsed: {
    message: "Hello"
  }
}

After ValidateContent

{
  raw: { /* ... */ },
  parsed: { /* ... */ },
  validation: {
    passed: true,
    checks: { /* ... */ }
  }
}

Context Variables

Define reusable variables in context:

{
  "context": {
    "BASE_URL": "https://api.example.com",
    "API_VERSION": "v1",
    "ENDPOINT": "${BASE_URL}/${API_VERSION}/users"
  },
  "tests": [{
    "pipeline": [{
      "block": "HttpRequest",
      "input": { "url": "${ENDPOINT}" }
    }]
  }]
}

Environment Variables

Reference environment variables:

.env

API_URL=https://api.example.com
API_KEY=secret123

{
  "context": {
    "BASE_URL": "${env.API_URL}",
    "API_KEY": "${env.API_KEY}"
  }
}

Debugging Data Flow

Enable debug logging to see data flow:

export LOG_LEVEL=DEBUG
npx semtest test.json

You’ll see:

[DEBUG] DataBus.set('response', {...})
[DEBUG] DataBus.get('response.body') → '{"id": 1}'
[DEBUG] Resolving ${response.body} → {"id": 1}

Best Practices

Use Descriptive Slot Names

// Good
"output": "userProfile"
"output": "authToken"
"output": "validationResult"

// Bad
"output": "data"
"output": "result"
"output": "temp"

Keep Data Flat When Possible

// Simpler to reference
"output": "userId"    // Use ${userId}

// vs
"output": "user"      // Use ${user.data.profile.id}

Use Context for Repeated Values

{
  "context": {
    "AUTH_HEADER": "Bearer ${env.API_KEY}"
  },
  "tests": [{
    "pipeline": [
      {
        "block": "HttpRequest",
        "input": {
          "headers": { "Authorization": "${AUTH_HEADER}" }
        }
      }
    ]
  }]
}

The DataBus

Writing to the DataBus

Reading from the DataBus

Variable Resolution

Resolution Order

Data Transformation

Context Variables

Environment Variables

Debugging Data Flow

Best Practices

Next Steps

Input Formats

Output Formats

​The DataBus

​Writing to the DataBus

​Reading from the DataBus

​Variable Resolution

​Resolution Order

​Data Transformation

​Context Variables

​Environment Variables

​Debugging Data Flow

​Best Practices

​Next Steps

Input Formats

Output Formats

The DataBus

Writing to the DataBus

Reading from the DataBus

Variable Resolution

Resolution Order

Data Transformation

Context Variables

Environment Variables

Debugging Data Flow

Best Practices

Next Steps