# Custom Blocks
Source: https://docs.semantictest.dev/advanced/custom-blocks
Extend SemanticTest with your own custom blocks
**Work in Progress**
This documentation is currently being written. Check back soon for detailed information on creating custom blocks!
## Coming Soon
This guide will cover:
* Creating custom block classes
* Defining inputs and outputs
* Registering custom blocks
* Best practices for block development
* Example custom blocks
In the meantime, check out the [Blocks Overview](/blocks/overview) to see how built-in blocks work.
# Debugging
Source: https://docs.semantictest.dev/advanced/debugging
Debug your tests and troubleshoot issues
**Work in Progress**
This documentation is currently being written. Check back soon for detailed debugging information!
## Coming Soon
This guide will cover:
* Using debug mode
* Understanding error messages
* Inspecting DataBus state
* Debugging block execution
* Common issues and solutions
In the meantime, try running tests with debug mode to see detailed output:
```bash theme={null}
export LOG_LEVEL=DEBUG
npx semtest test.json
```
# Environment Variables
Source: https://docs.semantictest.dev/advanced/environment-variables
Manage configuration with environment variables
**Work in Progress**
This documentation is currently being written. Check back soon for detailed information on environment variables!
## Coming Soon
This guide will cover:
* Using `.env` files
* Accessing environment variables in tests
* Environment-specific configurations
* Secrets management
* Best practices for configuration
In the meantime, check the [Data Flow](/concepts/data-flow) documentation for how `${env.VARIABLE}` works.
# Programmatic Usage
Source: https://docs.semantictest.dev/advanced/programmatic-usage
Use SemanticTest in your JavaScript/TypeScript code
**Work in Progress**
This documentation is currently being written. Check back soon for detailed information on programmatic usage!
## Coming Soon
This guide will cover:
* Using SemanticTest in Node.js applications
* Creating pipelines programmatically
* Custom test runners
* Integrating with CI/CD
* TypeScript support
In the meantime, check the [Quickstart Guide](/quickstart) for a basic programmatic example.
# Test Suites
Source: https://docs.semantictest.dev/advanced/test-suites
Organize and run multiple tests efficiently
**Work in Progress**
This documentation is currently being written. Check back soon for detailed information on test suites!
## Coming Soon
This guide will cover:
* Organizing tests into suites
* Shared setup and teardown
* Running multiple test files
* Parallel test execution
* Suite-level configuration
In the meantime, check out the [Quickstart Guide](/quickstart) for basic test examples.
# Multi-Turn Conversations
Source: https://docs.semantictest.dev/ai-testing/multi-turn-conversations
Test conversational AI flows with context and memory
## What are Multi-Turn Conversations?
**Multi-turn conversations** are interactions where the AI remembers previous messages and maintains context:
```
User: "I need a laptop"
AI: "What's your budget?"
User: "$1000" ← References "laptop" and "budget" from before
AI: "Here are laptops under $1000" ← Remembers user needs laptop for ~$1000
```
The AI must:
* ✅ Remember previous messages
* ✅ Understand references ("it", "that", "the one you mentioned")
* ✅ Maintain conversation state (user's budget, preferences, etc.)
* ✅ Stay contextually relevant
## Why Test Multi-Turn Conversations?
Single-turn testing misses critical issues:
### Context Loss
```
Turn 1: "Find flights to Paris"
Turn 2: "What about hotels?"
❌ Without context: AI asks "Where?"
✅ With context: AI knows user means Paris
```
### Reference Resolution
```
Turn 1: "Show me the MacBook Pro and Dell XPS"
Turn 2: "What's the battery life on the second one?"
❌ Without context: AI confused
✅ With context: AI knows "second one" = Dell XPS
```
### State Management
```
Turn 1: "I want to book a flight"
Turn 2: "From Boston to Seattle"
Turn 3: "Next Friday"
AI must accumulate: flight booking, BOS→SEA, date=next Friday
```
## Testing Approaches
### Approach 1: Stateless (Recommended)
Test each conversation turn **independently** with full history:
```json theme={null}
{
"tests": [
{
"id": "turn-1",
"pipeline": [
{
"block": "HttpRequest",
"input": {
"body": {
"messages": [
{ "role": "user", "content": "I need a laptop" }
]
}
}
}
]
},
{
"id": "turn-2",
"pipeline": [
{
"block": "HttpRequest",
"input": {
"body": {
"messages": [
{ "role": "user", "content": "I need a laptop" },
{ "role": "assistant", "content": "What's your budget?" },
{ "role": "user", "content": "$1000" }
]
}
}
}
]
}
]
}
```
**Pros:**
* ✅ Each test is independent
* ✅ Can run tests in parallel
* ✅ Easy to debug individual turns
* ✅ No shared state between tests
**Cons:**
* ❌ More verbose (repeat full history)
* ❌ Doesn't test actual conversation flow
### Approach 2: Stateful
Test conversation as **continuous flow**:
```json theme={null}
{
"tests": [{
"id": "full-conversation",
"pipeline": [
{
"id": "turn-1",
"block": "HttpRequest",
"input": {
"body": {
"messages": [{ "role": "user", "content": "I need a laptop" }]
}
},
"output": "response1"
},
{
"block": "JsonParser",
"input": "${response1.body}",
"output": { "parsed": "data1" }
},
{
"id": "turn-2",
"block": "HttpRequest",
"input": {
"body": {
"messages": [
{ "role": "user", "content": "I need a laptop" },
{ "role": "assistant", "content": "${data1.message}" },
{ "role": "user", "content": "$1000" }
]
}
},
"output": "response2"
}
]
}]
}
```
**Pros:**
* ✅ Tests actual conversation flow
* ✅ Validates state management
* ✅ Closer to real usage
**Cons:**
* ❌ Tests are sequential (slower)
* ❌ Hard to isolate failures
* ❌ One failure breaks entire test
## Using History in LLMJudge
LLMJudge accepts conversation `history` to evaluate responses in context:
```json theme={null}
{
"block": "LLMJudge",
"input": {
"text": "${aiResponse}",
"history": [
{ "role": "user", "content": "I'm looking for a laptop" },
{ "role": "assistant", "content": "What's your budget?" },
{ "role": "user", "content": "Around $1000" }
],
"expected": {
"expectedBehavior": "Recommends laptops in the $800-1200 range, remembering user's $1000 budget"
}
},
"output": "judgement"
}
```
Without history, LLMJudge wouldn't know the "\$1000 budget" context.
## Real-World Examples
### 1. Customer Service Bot
```json theme={null}
{
"name": "Customer Service Conversation",
"context": {
"CHAT_URL": "${env.CHAT_API_URL}"
},
"tests": [
{
"id": "turn-1-greeting",
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "${CHAT_URL}/chat",
"method": "POST",
"body": {
"messages": [
{ "role": "user", "content": "Hello" }
]
}
},
"output": "response"
},
{
"block": "JsonParser",
"input": "${response.body}",
"output": { "parsed": "data" }
},
{
"block": "LLMJudge",
"input": {
"text": "${data.message}",
"expected": {
"expectedBehavior": "Friendly greeting and asks how they can help"
}
},
"output": "judgement"
}
],
"assertions": {
"judgement.score": { "gte": 0.8 }
}
},
{
"id": "turn-2-state-problem",
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "${CHAT_URL}/chat",
"method": "POST",
"body": {
"messages": [
{ "role": "user", "content": "Hello" },
{ "role": "assistant", "content": "Hi! How can I help you today?" },
{ "role": "user", "content": "My order hasn't arrived" }
]
}
},
"output": "response"
},
{
"block": "JsonParser",
"input": "${response.body}",
"output": { "parsed": "data" }
},
{
"block": "LLMJudge",
"input": {
"text": "${data.message}",
"history": [
{ "role": "user", "content": "Hello" },
{ "role": "assistant", "content": "Hi! How can I help you today?" },
{ "role": "user", "content": "My order hasn't arrived" }
],
"expected": {
"expectedBehavior": "Shows empathy and asks for order number or details. Must be empathetic, not blame the customer, and ask for order information."
}
},
"output": "judgement"
}
],
"assertions": {
"judgement.score": { "gte": 0.85 }
}
},
{
"id": "turn-3-provide-order",
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "${CHAT_URL}/chat",
"method": "POST",
"body": {
"messages": [
{ "role": "user", "content": "Hello" },
{ "role": "assistant", "content": "Hi! How can I help you today?" },
{ "role": "user", "content": "My order hasn't arrived" },
{ "role": "assistant", "content": "I'm sorry to hear that. Could you provide your order number?" },
{ "role": "user", "content": "It's #12345" }
]
}
},
"output": "response"
},
{
"block": "StreamParser",
"input": "${response.body}",
"output": {
"text": "aiMessage",
"toolCalls": "tools"
}
},
{
"block": "ValidateTools",
"input": {
"from": "tools",
"as": "toolCalls"
},
"config": {
"expected": ["lookup_order"],
"validateArgs": {
"lookup_order": {
"order_id": "12345"
}
}
},
"output": "toolValidation"
},
{
"block": "LLMJudge",
"input": {
"text": "${aiMessage}",
"toolCalls": "${tools}",
"history": [
{ "role": "user", "content": "Hello" },
{ "role": "assistant", "content": "Hi! How can I help you today?" },
{ "role": "user", "content": "My order hasn't arrived" },
{ "role": "assistant", "content": "I'm sorry to hear that. Could you provide your order number?" },
{ "role": "user", "content": "It's #12345" }
],
"expected": {
"expectedBehavior": "Looks up order #12345 and provides status update"
}
},
"output": "judgement"
}
],
"assertions": {
"toolValidation.passed": true,
"judgement.score": { "gte": 0.8 }
}
}
]
}
```
### 2. Shopping Assistant
```json theme={null}
{
"name": "Shopping Assistant Conversation",
"tests": [{
"id": "laptop-shopping-flow",
"pipeline": [
{
"id": "turn-1",
"block": "MockData",
"config": {
"data": {
"messages": [
{ "role": "user", "content": "I need a laptop for programming" }
]
}
},
"output": "context1"
},
{
"block": "HttpRequest",
"input": {
"url": "${SHOP_API}/chat",
"method": "POST",
"body": "${context1.messages}"
},
"output": "response1"
},
{
"block": "JsonParser",
"input": "${response1.body}",
"output": { "parsed": "aiReply1" }
},
{
"block": "LLMJudge",
"input": {
"text": "${aiReply1.message}",
"expected": {
"expectedBehavior": "Asks clarifying questions about budget, preferred OS, or performance needs"
}
},
"output": "check1"
},
{
"id": "turn-2",
"block": "HttpRequest",
"input": {
"url": "${SHOP_API}/chat",
"method": "POST",
"body": {
"messages": [
{ "role": "user", "content": "I need a laptop for programming" },
{ "role": "assistant", "content": "${aiReply1.message}" },
{ "role": "user", "content": "Budget is $1500, prefer MacOS" }
]
}
},
"output": "response2"
},
{
"block": "JsonParser",
"input": "${response2.body}",
"output": { "parsed": "aiReply2" }
},
{
"block": "LLMJudge",
"input": {
"text": "${aiReply2.message}",
"history": [
{ "role": "user", "content": "I need a laptop for programming" },
{ "role": "assistant", "content": "${aiReply1.message}" },
{ "role": "user", "content": "Budget is $1500, prefer MacOS" }
],
"expected": {
"expectedBehavior": "Recommends MacBook models within $1500 budget, suitable for programming"
}
},
"output": "check2"
}
],
"assertions": {
"check1.score": { "gte": 0.8 },
"check2.score": { "gte": 0.85 }
}
}]
}
```
### 3. Tutoring Bot (Long Context)
```json theme={null}
{
"tests": [{
"id": "math-tutoring-session",
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "${TUTOR_API}/chat",
"method": "POST",
"body": {
"messages": [
{ "role": "user", "content": "I don't understand how to solve quadratic equations" },
{ "role": "assistant", "content": "Let me help! A quadratic equation has the form ax² + bx + c = 0. Have you learned the quadratic formula?" },
{ "role": "user", "content": "No, what is it?" },
{ "role": "assistant", "content": "The quadratic formula is x = (-b ± √(b²-4ac)) / 2a. Let's try an example: x² + 5x + 6 = 0" },
{ "role": "user", "content": "How do I start?" }
]
}
},
"output": "response"
},
{
"block": "JsonParser",
"input": "${response.body}",
"output": { "parsed": "data" }
},
{
"block": "LLMJudge",
"input": {
"text": "${data.message}",
"history": [
{ "role": "user", "content": "I don't understand how to solve quadratic equations" },
{ "role": "assistant", "content": "Let me help! A quadratic equation has the form ax² + bx + c = 0. Have you learned the quadratic formula?" },
{ "role": "user", "content": "No, what is it?" },
{ "role": "assistant", "content": "The quadratic formula is x = (-b ± √(b²-4ac)) / 2a. Let's try an example: x² + 5x + 6 = 0" },
{ "role": "user", "content": "How do I start?" }
],
"expected": {
"expectedBehavior": "Guides student through identifying a=1, b=5, c=6 in the example x² + 5x + 6 = 0, remembering the quadratic formula already explained. Must reference the specific example, not solve it for them but guide them, and be encouraging and patient."
}
},
"output": "judgement"
}
],
"assertions": {
"judgement.score": { "gte": 0.9 }
}
}]
}
```
## Testing Context Retention
### Test Reference Resolution
```json theme={null}
{
"tests": [{
"id": "test-pronoun-resolution",
"pipeline": [
{
"block": "HttpRequest",
"input": {
"body": {
"messages": [
{ "role": "user", "content": "Tell me about the MacBook Pro and the Dell XPS" },
{ "role": "assistant", "content": "The MacBook Pro has M3 chip... The Dell XPS has Intel processor..." },
{ "role": "user", "content": "Which one has better battery life?" }
]
}
},
"output": "response"
},
{
"block": "JsonParser",
"input": "${response.body}",
"output": { "parsed": "data" }
},
{
"block": "LLMJudge",
"input": {
"text": "${data.message}",
"history": [
{ "role": "user", "content": "Tell me about the MacBook Pro and the Dell XPS" },
{ "role": "assistant", "content": "The MacBook Pro has M3 chip... The Dell XPS has Intel processor..." },
{ "role": "user", "content": "Which one has better battery life?" }
],
"expected": {
"expectedBehavior": "Compares battery life between MacBook Pro and Dell XPS, understanding 'which one' refers to these two laptops"
}
},
"output": "judgement"
}
],
"assertions": {
"judgement.score": { "gte": 0.8 }
}
}]
}
```
### Test Information Accumulation
```json theme={null}
{
"tests": [{
"id": "test-state-accumulation",
"pipeline": [
{
"block": "HttpRequest",
"input": {
"body": {
"messages": [
{ "role": "user", "content": "I want to book a flight" },
{ "role": "assistant", "content": "Great! Where are you flying from?" },
{ "role": "user", "content": "Boston" },
{ "role": "assistant", "content": "And where to?" },
{ "role": "user", "content": "Seattle" },
{ "role": "assistant", "content": "When would you like to travel?" },
{ "role": "user", "content": "Next Friday" }
]
}
},
"output": "response"
},
{
"block": "StreamParser",
"input": "${response.body}",
"output": {
"text": "aiMessage",
"toolCalls": "tools"
}
},
{
"block": "ValidateTools",
"input": { "from": "tools", "as": "toolCalls" },
"config": {
"expected": ["search_flights"],
"validateArgs": {
"search_flights": {
"origin": "Boston",
"destination": "Seattle"
}
}
},
"output": "toolCheck"
},
{
"block": "LLMJudge",
"input": {
"text": "${aiMessage}",
"toolCalls": "${tools}",
"history": [
{ "role": "user", "content": "I want to book a flight" },
{ "role": "assistant", "content": "Great! Where are you flying from?" },
{ "role": "user", "content": "Boston" },
{ "role": "assistant", "content": "And where to?" },
{ "role": "user", "content": "Seattle" },
{ "role": "assistant", "content": "When would you like to travel?" },
{ "role": "user", "content": "Next Friday" }
],
"expected": {
"expectedBehavior": "Searches for flights from Boston to Seattle for next Friday, having accumulated all the information"
}
},
"output": "judgement"
}
],
"assertions": {
"toolCheck.passed": true,
"judgement.score": { "gte": 0.8 }
}
}]
}
```
## Best Practices
```json theme={null}
{
"block": "LLMJudge",
"input": {
"text": "${aiResponse}",
"history": [...all previous messages...],
"expected": {...}
}
}
```
Without history, LLMJudge can't judge context-dependent responses.
```json theme={null}
{
"tests": [
{ "id": "turn-1", "pipeline": [...] },
{ "id": "turn-2", "pipeline": [...] }, // Independent
{ "id": "turn-3", "pipeline": [...] }
]
}
```
Easier to debug and run in parallel.
* Reference resolution ("it", "that one", "the previous")
* Information accumulation (collecting details over turns)
* Topic switching (changing subject mid-conversation)
* Clarification requests ("What did you mean by...")
* Error recovery ("Sorry, I meant...")
AI models have context limits (e.g., 8K tokens). Test long conversations:
```json theme={null}
{
"tests": [{
"id": "test-long-conversation",
"pipeline": [
{
"block": "HttpRequest",
"input": {
"body": {
"messages": [
// 50+ messages here
]
}
}
}
]
}]
}
```
Ensure AI still responds appropriately.
Test that information from early turns is remembered:
```json theme={null}
{
"expected": {
"expectedBehavior": "Remembers user's name (Alice) from turn 1 and budget ($1000) from turn 3"
}
}
```
## Advanced Patterns
### Conversation Repair
Test how AI handles corrections:
```json theme={null}
{
"tests": [{
"id": "test-correction-handling",
"pipeline": [
{
"block": "HttpRequest",
"input": {
"body": {
"messages": [
{ "role": "user", "content": "Book a flight to Paris" },
{ "role": "assistant", "content": "Great! When would you like to fly to Paris?" },
{ "role": "user", "content": "Sorry, I meant London, not Paris" }
]
}
},
"output": "response"
},
{
"block": "JsonParser",
"input": "${response.body}",
"output": { "parsed": "data" }
},
{
"block": "LLMJudge",
"input": {
"text": "${data.message}",
"history": [
{ "role": "user", "content": "Book a flight to Paris" },
{ "role": "assistant", "content": "Great! When would you like to fly to Paris?" },
{ "role": "user", "content": "Sorry, I meant London, not Paris" }
],
"expected": {
"expectedBehavior": "Acknowledges correction and asks about London flight instead"
}
},
"output": "judgement"
}
],
"assertions": {
"judgement.score": { "gte": 0.8 }
}
}]
}
```
### Ambiguity Resolution
Test how AI handles unclear references:
```json theme={null}
{
"tests": [{
"id": "test-ambiguous-reference",
"pipeline": [
{
"block": "HttpRequest",
"input": {
"body": {
"messages": [
{ "role": "user", "content": "Tell me about the iPhone 15 and Samsung S24" },
{ "role": "assistant", "content": "The iPhone 15 has A17 chip... Samsung S24 has Snapdragon..." },
{ "role": "user", "content": "What about its camera?" }
]
}
},
"output": "response"
},
{
"block": "JsonParser",
"input": "${response.body}",
"output": { "parsed": "data" }
},
{
"block": "LLMJudge",
"input": {
"text": "${data.message}",
"history": [
{ "role": "user", "content": "Tell me about the iPhone 15 and Samsung S24" },
{ "role": "assistant", "content": "The iPhone 15 has A17 chip... Samsung S24 has Snapdragon..." },
{ "role": "user", "content": "What about its camera?" }
],
"expected": {
"expectedBehavior": "Either asks for clarification (which phone?) or discusses both cameras"
}
},
"output": "judgement"
}
]
}]
}
```
### Multi-Topic Conversations
Test topic switching:
```json theme={null}
{
"tests": [{
"id": "test-topic-switch",
"pipeline": [
{
"block": "HttpRequest",
"input": {
"body": {
"messages": [
{ "role": "user", "content": "What's the weather in Boston?" },
{ "role": "assistant", "content": "It's 72°F and sunny in Boston" },
{ "role": "user", "content": "Actually, can you help me book a hotel instead?" }
]
}
},
"output": "response"
},
{
"block": "JsonParser",
"input": "${response.body}",
"output": { "parsed": "data" }
},
{
"block": "LLMJudge",
"input": {
"text": "${data.message}",
"history": [
{ "role": "user", "content": "What's the weather in Boston?" },
{ "role": "assistant", "content": "It's 72°F and sunny in Boston" },
{ "role": "user", "content": "Actually, can you help me book a hotel instead?" }
],
"expected": {
"expectedBehavior": "Switches topic gracefully to hotel booking, possibly inferring Boston as location"
}
},
"output": "judgement"
}
]
}]
}
```
## Troubleshooting
### Problem: AI doesn't remember context
**Check:**
1. Is history passed to the API?
2. Is conversation too long (exceeding token limit)?
3. Is API endpoint stateful or stateless?
### Problem: LLMJudge gives low scores despite good response
**Solution:** Include full history:
```json theme={null}
{
"input": {
"history": [...all messages...],
"expected": {
"expectedBehavior": "Response should make sense in context of previous conversation"
}
}
}
```
### Problem: Tests are slow
**Solution:** Use stateless testing:
```json theme={null}
{
"tests": [
{ "id": "turn-1" }, // Independent
{ "id": "turn-2" }, // Run in parallel
{ "id": "turn-3" }
]
}
```
## Next Steps
Deep dive into LLMJudge for validation
Validate AI tool/function calls
Complete LLMJudge documentation
Full conversational agent example
# Testing AI Systems
Source: https://docs.semantictest.dev/ai-testing/overview
Learn how to test non-deterministic AI systems effectively
## The Challenge of Testing AI
Traditional API testing uses **exact matching**:
```json theme={null}
{
"assertions": {
"response.message": "User created successfully"
}
}
```
This breaks immediately with AI responses. Consider asking an AI to schedule a meeting:
**Expected:** "I've scheduled your meeting for 2 PM on Tuesday."
**Actual responses:**
* ✅ "Your meeting is set for Tuesday at 2:00 PM"
* ✅ "Done! Meeting scheduled for 14:00 on Tuesday"
* ✅ "I've booked your Tuesday 2pm slot"
* ❌ All fail with exact matching!
## Why AI Testing is Different
### 1. Non-Deterministic Responses
Same input → different outputs every time:
```bash theme={null}
Input: "Summarize this article"
Output 1: "This article discusses..."
Output 2: "The piece explores..."
Output 3: "Key points include..."
```
### 2. Semantic Equivalence
These mean the same thing:
* "Cannot find user"
* "User not found"
* "That user doesn't exist"
* "I couldn't locate that user"
### 3. Tool/Function Calling
AI agents call tools unpredictably:
```javascript theme={null}
// Both valid for "book a meeting"
[{ name: "search_calendar" }, { name: "create_event" }]
[{ name: "check_availability" }, { name: "create_event" }, { name: "send_confirmation" }]
```
### 4. Streaming Responses
Responses arrive as chunks over Server-Sent Events (SSE):
```
data: {"content":"Hello"}
data: {"content":" there"}
data: {"content":"!"}
```
## SemanticTest's Approach
SemanticTest provides **4 specialized approaches** for AI testing:
### 1. Semantic Validation (LLMJudge)
Use AI to judge AI responses:
```json theme={null}
{
"block": "LLMJudge",
"input": {
"text": "${aiResponse.text}",
"expected": {
"expectedBehavior": "Should confirm meeting time and location"
}
},
"output": "judgement"
}
```
**Result:**
```javascript theme={null}
{
score: 0.95,
passed: true,
reasoning: "Response confirms meeting time (2 PM) and location (Conference Room A)"
}
```
### 2. Tool Call Validation
Validate what tools AI calls and with what arguments:
```json theme={null}
{
"block": "ValidateTools",
"input": {
"from": "aiResponse.toolCalls",
"as": "toolCalls"
},
"config": {
"expected": ["search_calendar", "create_event"],
"order": ["search_calendar", "create_event"]
}
}
```
### 3. Streaming Response Parsing
Parse SSE streams from AI APIs:
```json theme={null}
{
"block": "StreamParser",
"input": "${response.body}",
"config": {
"format": "sse-openai"
},
"output": {
"text": "aiMessage",
"toolCalls": "aiTools"
}
}
```
### 4. Multi-Turn Conversation Testing
Test conversational flows with context:
```json theme={null}
{
"block": "LLMJudge",
"input": {
"text": "${aiResponse.text}",
"history": [
{ "role": "user", "content": "Hello" },
{ "role": "assistant", "content": "Hi! How can I help?" },
{ "role": "user", "content": "Book a meeting" }
],
"expected": {
"expectedBehavior": "Should remember context and ask for meeting details"
}
}
}
```
## Traditional vs Semantic Testing
```json theme={null}
{
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "${AI_API}/chat",
"method": "POST",
"body": {
"message": "Schedule meeting for tomorrow at 2pm"
}
},
"output": "response"
}
],
"assertions": {
"response.body": "I've scheduled your meeting for 2:00 PM tomorrow."
}
}
```
**Problem:** Fails if AI says "Done! Meeting set for 2pm tomorrow" or any variation.
```json theme={null}
{
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "${AI_API}/chat",
"method": "POST",
"body": {
"message": "Schedule meeting for tomorrow at 2pm"
}
},
"output": "response"
},
{
"block": "StreamParser",
"input": "${response.body}",
"config": { "format": "sse-openai" },
"output": { "text": "aiMessage", "toolCalls": "tools" }
},
{
"block": "ValidateTools",
"input": { "from": "tools", "as": "toolCalls" },
"config": {
"expected": ["create_event"]
},
"output": "toolValidation"
},
{
"block": "LLMJudge",
"input": {
"text": "${aiMessage}",
"expected": {
"expectedBehavior": "Confirms meeting scheduled for 2pm tomorrow"
}
},
"output": "judgement"
}
],
"assertions": {
"toolValidation.passed": true,
"judgement.score": { "gt": 0.8 }
}
}
```
**Result:** Passes for any semantically correct response.
## What You'll Learn
Use LLMJudge to validate AI responses semantically instead of exact matches
Test AI agents that call tools/functions with ValidateTools
Parse and test Server-Sent Events (SSE) streams from AI APIs
Test conversational AI with context and memory
## Quick Start
### 1. Install SemanticTest
```bash theme={null}
npm install @blade47/semantic-test
```
### 2. Set OpenAI API Key (for LLMJudge)
```bash theme={null}
export OPENAI_API_KEY="sk-..."
```
**Optional:** LLMJudge requires OpenAI API key. All other blocks work without it!
### 3. Create Your First AI Test
```json test.json theme={null}
{
"name": "AI Chat Test",
"context": {
"AI_URL": "https://api.openai.com/v1/chat/completions",
"API_KEY": "${env.OPENAI_API_KEY}"
},
"tests": [{
"id": "test-greeting",
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "${AI_URL}",
"method": "POST",
"headers": {
"Authorization": "Bearer ${API_KEY}",
"Content-Type": "application/json"
},
"body": {
"model": "gpt-4o-mini",
"messages": [
{ "role": "user", "content": "Hello!" }
]
}
},
"output": "response"
},
{
"block": "JsonParser",
"input": "${response.body}",
"output": { "parsed": "data" }
},
{
"block": "LLMJudge",
"input": {
"text": "${data.choices[0].message.content}",
"expected": {
"expectedBehavior": "Friendly greeting response"
}
},
"output": "judgement"
}
],
"assertions": {
"response.status": 200,
"judgement.score": { "gt": 0.8 }
}
}]
}
```
### 4. Run Tests
```bash theme={null}
npx semtest test.json
```
## When to Use Each Approach
| Use Case | Approach | Block |
| --------------------------------- | ------------------- | ------------------ |
| Validate response quality/meaning | Semantic Validation | LLMJudge |
| Check exact tools called | Tool Validation | ValidateTools |
| Parse streaming AI responses | Stream Parsing | StreamParser |
| Test conversation flows | Multi-Turn Testing | LLMJudge + history |
| Verify specific keywords | Traditional | ValidateContent |
| Check response structure | Traditional | Assertions |
## Best Practices
Use structural validation (ValidateTools) + semantic validation (LLMJudge):
```json theme={null}
{
"pipeline": [
{ "block": "ValidateTools", "config": { "expected": ["search"] } },
{ "block": "LLMJudge", "input": { "expected": { "expectedBehavior": "Polite response" } } }
]
}
```
Exact assertions are faster and cheaper:
```json theme={null}
{
"assertions": {
"response.status": 200,
"data.id": { "gt": 0 }
}
}
```
Reserve LLMJudge for truly non-deterministic content.
Be specific in LLMJudge expectations:
```json theme={null}
// ❌ Vague
"expectedBehavior": "Good response"
// ✅ Specific
"expectedBehavior": "Confirms order #12345 is shipped and provides tracking number"
```
AI systems need edge case testing:
* Ambiguous user input
* Missing context
* Conflicting instructions
* Tool call failures
* Stream interruptions
## Real-World Examples
Complete example testing OpenAI chat API
Test an AI agent that manages calendar events
Traditional API testing patterns
Handle errors and retries
## Next Steps
Start with semantic validation - the core of AI testing
# Semantic Validation
Source: https://docs.semantictest.dev/ai-testing/semantic-validation
Use AI to evaluate AI responses with LLMJudge
## What is Semantic Validation?
**Semantic validation** means checking if a response is *correct in meaning*, not just in exact wording.
### The Problem
```json theme={null}
// User asks: "Schedule a meeting for tomorrow at 2pm"
// All these responses are semantically correct:
"✅ I've scheduled your meeting for 2:00 PM tomorrow"
"✅ Done! Your 2pm meeting is set for tomorrow"
"✅ Meeting booked: Tomorrow at 14:00"
"✅ Your meeting tomorrow at 2 is confirmed"
// Traditional exact matching fails on all but one ❌
```
### The Solution: LLMJudge
**LLMJudge** uses OpenAI models to evaluate whether responses meet your semantic requirements:
```json theme={null}
{
"block": "LLMJudge",
"input": {
"text": "${aiResponse.text}",
"expected": {
"expectedBehavior": "Confirms meeting scheduled for 2pm tomorrow"
}
},
"output": "judgement"
}
```
**Output:**
```javascript theme={null}
{
score: 0.95,
passed: true,
reasoning: "Response clearly confirms meeting time (2pm) and date (tomorrow)"
}
```
## When to Use LLMJudge
**Non-deterministic text content:**
* AI chat responses
* Generated summaries
* Creative content
* Explanations
* Natural language descriptions
**Example:**
```json theme={null}
{
"block": "LLMJudge",
"input": {
"text": "${aiResponse}",
"expected": {
"expectedBehavior": "Explains the concept clearly in simple terms"
}
}
}
```
**Deterministic or structured data:**
* Status codes
* IDs, timestamps
* Exact numbers
* Boolean flags
* JSON structure
**Use traditional assertions instead:**
```json theme={null}
{
"assertions": {
"response.status": 200,
"user.id": { "gt": 0 },
"order.total": 99.99
}
}
```
## Setup
### 1. Get OpenAI API Key
Sign up at [platform.openai.com](https://platform.openai.com) and create an API key.
### 2. Set Environment Variable
```bash theme={null}
export OPENAI_API_KEY="sk-..."
```
Or use a `.env` file:
```bash theme={null}
# .env
OPENAI_API_KEY=sk-...
```
**Cost:** LLMJudge uses GPT-4o-mini by default (\~$0.0001 per evaluation). A typical test with 10 AI responses costs ~$0.001.
## Basic Usage
### Simple Validation
```json theme={null}
{
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "${AI_API}/chat",
"method": "POST",
"body": {
"message": "What's the capital of France?"
}
},
"output": "response"
},
{
"block": "JsonParser",
"input": "${response.body}",
"output": { "parsed": "data" }
},
{
"block": "LLMJudge",
"input": {
"text": "${data.message}",
"expected": {
"expectedBehavior": "States that Paris is the capital of France"
}
},
"output": "judgement"
}
],
"assertions": {
"judgement.score": { "gt": 0.8 }
}
}
```
## Expected Behavior Format
### Be Specific
```json theme={null}
{
"expectedBehavior": "Good response"
}
```
Too vague - what makes a response "good"?
```json theme={null}
{
"expectedBehavior": "Confirms order #12345 is shipped, provides FedEx tracking number, and estimates delivery within 3-5 business days"
}
```
Clear, measurable criteria.
### Include Key Details
```json theme={null}
{
"expectedBehavior": "Explains the difference between REST and GraphQL APIs, mentioning at least: request/response structure, over-fetching, and type safety"
}
```
### Use Action Verbs
Good verbs: confirms, explains, lists, mentions, asks, provides, includes, summarizes
```json theme={null}
{
"expectedBehavior": "Lists 3-5 main advantages of TypeScript over JavaScript"
}
```
## Custom Scoring Criteria
Define your own scoring dimensions:
```json theme={null}
{
"block": "LLMJudge",
"input": {
"text": "${aiResponse}",
"expected": {
"expectedBehavior": "Helpful customer service response"
}
},
"config": {
"criteria": {
"accuracy": 0.5,
"completeness": 0.3,
"relevance": 0.2
}
},
"output": "judgement"
}
```
**Output:**
```javascript theme={null}
{
score: 0.87,
reasoning: "Response is accurate and complete...",
details: {
accuracy: 0.9,
completeness: 0.85,
relevance: 0.87
}
}
```
**Weights must sum to 1.0**
### Available Criteria
LLMJudge supports three scoring criteria. You can adjust their weights:
```json theme={null}
{
"config": {
"criteria": {
"accuracy": 0.4, // How correct is the response? (default: 0.4)
"completeness": 0.3, // Does it cover all required points? (default: 0.3)
"relevance": 0.3 // Is it relevant to the question? (default: 0.3)
}
}
}
```
**Weights must sum to 1.0.** The three criteria are hardcoded and cannot be customized, but you can adjust their relative importance by changing the weights.
### Example Weight Adjustments
```json theme={null}
{
"criteria": {
"accuracy": 0.6,
"completeness": 0.2,
"relevance": 0.2
}
}
```
Use when correctness is most important (e.g., factual questions).
```json theme={null}
{
"criteria": {
"accuracy": 0.4,
"completeness": 0.3,
"relevance": 0.3
}
}
```
Default balanced approach for general use.
```json theme={null}
{
"criteria": {
"accuracy": 0.3,
"completeness": 0.5,
"relevance": 0.2
}
}
```
Use when covering all points is critical (e.g., instructions, summaries).
## Model Selection
Choose the right model for your use case:
```json theme={null}
{
"config": {
"model": "gpt-4o-mini" // or "gpt-4o", "gpt-4-turbo"
}
}
```
### Model Comparison
| Model | Speed | Cost | Accuracy | Use Case |
| --------------- | --------- | ------------- | ----------- | --------------------------------------- |
| **gpt-4o-mini** | ⚡ Fast | 💰 \$0.0001 | ⭐⭐⭐ Good | **Recommended** - Fast, cheap, accurate |
| **gpt-4o** | ⚡ Fast | 💰💰 \$0.0025 | ⭐⭐⭐⭐ Better | Complex evaluations |
| **gpt-4-turbo** | 🐌 Slower | 💰💰💰 \$0.01 | ⭐⭐⭐⭐⭐ Best | Critical validations |
**Start with gpt-4o-mini.** It's 25x cheaper than gpt-4-turbo and handles 95% of use cases well.
## Real-World Examples
### 1. Testing ChatGPT-Style Interface
```json theme={null}
{
"name": "Chatbot Quality Test",
"context": {
"CHAT_URL": "${env.CHAT_API_URL}"
},
"tests": [{
"id": "test-helpful-response",
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "${CHAT_URL}/chat",
"method": "POST",
"body": {
"message": "How do I reset my password?"
}
},
"output": "response"
},
{
"block": "JsonParser",
"input": "${response.body}",
"output": { "parsed": "data" }
},
{
"block": "LLMJudge",
"input": {
"text": "${data.reply}",
"expected": {
"expectedBehavior": "Provides clear step-by-step instructions to reset password, mentioning the settings or account page, being polite and helpful, and not asking for current password in chat"
}
},
"config": {
"criteria": {
"accuracy": 0.4,
"completeness": 0.4,
"relevance": 0.2
}
},
"output": "judgement"
}
],
"assertions": {
"response.status": 200,
"judgement.score": { "gte": 0.8 },
"judgement.details.completeness": { "gte": 0.7 }
}
}]
}
```
### 2. Testing Content Summarization
```json theme={null}
{
"tests": [{
"id": "test-article-summary",
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "${AI_API}/summarize",
"method": "POST",
"body": {
"text": "Long article text here...",
"maxLength": 100
}
},
"output": "response"
},
{
"block": "JsonParser",
"input": "${response.body}",
"output": { "parsed": "data" }
},
{
"block": "ValidateContent",
"input": {
"from": "data.summary",
"as": "text"
},
"config": {
"maxLength": 120
},
"output": "lengthCheck"
},
{
"block": "LLMJudge",
"input": {
"text": "${data.summary}",
"expected": {
"expectedBehavior": "Captures the main points: company revenue growth, new product launch, and market expansion plans"
}
},
"output": "judgement"
}
],
"assertions": {
"lengthCheck.passed": true,
"judgement.score": { "gt": 0.85 }
}
}]
}
```
### 3. Testing Code Generation
```json theme={null}
{
"tests": [{
"id": "test-code-explanation",
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "${AI_API}/explain-code",
"method": "POST",
"body": {
"code": "const sum = arr => arr.reduce((a,b) => a+b, 0)",
"language": "javascript"
}
},
"output": "response"
},
{
"block": "JsonParser",
"input": "${response.body}",
"output": { "parsed": "data" }
},
{
"block": "LLMJudge",
"input": {
"text": "${data.explanation}",
"expected": {
"expectedBehavior": "Explains that the code sums array elements using reduce, mentioning the accumulator pattern in a beginner-friendly way, explaining the reduce function and the initial value (0)"
}
},
"config": {
"criteria": {
"accuracy": 0.5,
"completeness": 0.3,
"relevance": 0.2
}
},
"output": "judgement"
}
],
"assertions": {
"judgement.details.accuracy": { "gte": 0.9 }
}
}]
}
```
## Cost Optimization
LLMJudge makes API calls to OpenAI. Here's how to minimize costs:
### 1. Use gpt-4o-mini
```json theme={null}
{
"config": {
"model": "gpt-4o-mini" // 25x cheaper than gpt-4-turbo
}
}
```
**Savings:** \~$0.99 per 100 evaluations vs $25 with gpt-4-turbo
### 2. Combine with Traditional Validation
Don't use LLMJudge for everything:
```json theme={null}
{
"pipeline": [
// Fast, free structural checks first
{
"block": "ValidateContent",
"config": {
"minLength": 10,
"contains": ["order", "confirmed"]
}
},
// Then semantic validation
{
"block": "LLMJudge",
"input": {
"expected": {
"expectedBehavior": "Professional and reassuring tone"
}
}
}
]
}
```
### 3. Cache Common Validations
If testing same AI responses repeatedly during development:
```bash theme={null}
# Run once, save results
npx semtest tests.json --html --output results.html
# Review results.html instead of re-running
```
### 4. Test Samples in CI/CD
```json theme={null}
{
"tests": [
{
"id": "quick-smoke-test",
"pipeline": [
// Test 1 representative example
]
}
]
}
```
Run full suite less frequently (nightly builds).
## Best Practices
```json theme={null}
// ❌ Too vague
"expectedBehavior": "Good answer"
// ✅ Specific and measurable
"expectedBehavior": "Lists 3-5 benefits of exercise, mentions both physical and mental health"
```
```json theme={null}
{
"expected": {
"expectedBehavior": "Provides medical information including a disclaimer to consult a doctor. Must not diagnose conditions or prescribe medications, only provide general health information."
}
}
```
```json theme={null}
{
"assertions": {
// Strict for critical features
"criticalCheck.score": { "gte": 0.95 },
// Lenient for nice-to-haves
"toneCheck.score": { "gte": 0.7 }
}
}
```
```json theme={null}
{
"pipeline": [
// Structure check (free, fast)
{ "block": "ValidateContent", "config": { "contains": "order" } },
// Tool validation (free, fast)
{ "block": "ValidateTools", "config": { "expected": ["send_email"] } },
// Semantic validation (costs $, slower)
{ "block": "LLMJudge", "input": { "expected": {...} } }
]
}
```
For conversations, pass history:
```json theme={null}
{
"block": "LLMJudge",
"input": {
"text": "${aiResponse}",
"history": [
{ "role": "user", "content": "I'm looking for a laptop" },
{ "role": "assistant", "content": "What's your budget?" },
{ "role": "user", "content": "Around $1000" }
],
"expected": {
"expectedBehavior": "Recommends laptops in $800-1200 range"
}
}
}
```
## Troubleshooting
### Low Scores Despite Good Responses
**Problem:** judgement.score is 0.6 but response looks good
**Solutions:**
1. Make expectedBehavior more specific
2. Lower score threshold: `{ "gte": 0.7 }` instead of `{ "gte": 0.9 }`
3. Check if response actually meets expectations
### High Costs
**Problem:** Test suite costs \$5+ per run
**Solutions:**
1. Switch to gpt-4o-mini
2. Use traditional validation where possible
3. Test fewer examples in CI
4. Cache results during development
### Inconsistent Scores
**Problem:** Same response gets different scores (0.85, then 0.78, then 0.92)
**Solution:** This is normal with AI. Set threshold ranges:
```json theme={null}
{
"assertions": {
"judgement.score": { "gte": 0.75 } // Allow some variance
}
}
```
### Missing Context
**Problem:** LLMJudge doesn't understand multi-turn conversation
**Solution:** Pass conversation history:
```json theme={null}
{
"input": {
"text": "${aiResponse}",
"history": [...previous messages...]
}
}
```
## Next Steps
Learn to validate AI tool/function calls
Parse and test SSE streams from AI APIs
Complete LLMJudge documentation
Full example testing OpenAI chat API
# Testing Streaming Responses
Source: https://docs.semantictest.dev/ai-testing/streaming-responses
Parse and test Server-Sent Events (SSE) streams from AI APIs
## Why AI APIs Use Streaming
Modern AI APIs stream responses using **Server-Sent Events (SSE)** instead of waiting for the complete response:
### Without Streaming
```
User: "Explain quantum computing"
[5 second wait...]
AI: "Quantum computing is a type of computation that..."
```
### With Streaming
```
User: "Explain quantum computing"
AI: "Quantum"
AI: " computing"
AI: " is"
AI: " a"
AI: " type"
...
```
**Benefits:**
* ⚡ Faster perceived response time
* 📱 Better user experience (see response forming)
* 🔄 Can cancel long responses early
* 🛠️ Get tool calls before full response completes
## Server-Sent Events (SSE) Format
AI APIs send responses as SSE streams:
```
data: {"id":"1","choices":[{"delta":{"content":"Hello"}}]}
data: {"id":"1","choices":[{"delta":{"content":" there"}}]}
data: {"id":"1","choices":[{"delta":{"content":"!"}}]}
data: [DONE]
```
Each `data:` line is a chunk. Combined: **"Hello there!"**
## StreamParser Block
**StreamParser** extracts text and tool calls from SSE streams:
```json theme={null}
{
"block": "StreamParser",
"input": "${response.body}",
"config": {
"format": "sse-openai"
},
"output": {
"text": "aiMessage",
"toolCalls": "aiTools"
}
}
```
**Output:**
```javascript theme={null}
{
aiMessage: "I'll search for that information for you.",
aiTools: [
{
name: "search_database",
args: { query: "user data", limit: 10 }
}
]
}
```
## Supported Formats
StreamParser supports multiple streaming formats:
| Format | Provider | Description |
| ------------ | ------------- | ------------------------- |
| `sse-openai` | OpenAI | ChatGPT API, Azure OpenAI |
| `sse-vercel` | Vercel AI SDK | Next.js AI applications |
| `sse` | Generic | Standard SSE format |
| `text` | Any | Plain text (no parsing) |
## Basic Usage
### Parse OpenAI Stream
```json theme={null}
{
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "https://api.openai.com/v1/chat/completions",
"method": "POST",
"headers": {
"Authorization": "Bearer ${env.OPENAI_API_KEY}",
"Content-Type": "application/json"
},
"body": {
"model": "gpt-4o-mini",
"messages": [{
"role": "user",
"content": "Hello!"
}],
"stream": true
}
},
"output": "response"
},
{
"block": "StreamParser",
"input": "${response.body}",
"config": {
"format": "sse-openai"
},
"output": {
"text": "aiMessage"
}
}
],
"assertions": {
"response.status": 200,
"aiMessage": { "minLength": 1 }
}
}
```
### Parse Vercel AI SDK Stream
```json theme={null}
{
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "${YOUR_API}/chat",
"method": "POST",
"body": {
"message": "Hello!"
}
},
"output": "response"
},
{
"block": "StreamParser",
"input": "${response.body}",
"config": {
"format": "sse-vercel"
},
"output": {
"text": "aiMessage",
"toolCalls": "tools"
}
}
]
}
```
## Extracting Content
### Text Only
```json theme={null}
{
"output": {
"text": "aiMessage"
}
}
```
Combines all text chunks into single string.
### Tool Calls Only
```json theme={null}
{
"output": {
"toolCalls": "tools"
}
}
```
Extracts all function/tool calls from stream.
### Both Text and Tools
```json theme={null}
{
"output": {
"text": "aiMessage",
"toolCalls": "tools"
}
}
```
### Include Metadata
```json theme={null}
{
"output": {
"text": "aiMessage",
"toolCalls": "tools",
"metadata": "streamMeta"
}
}
```
**Metadata includes:**
```javascript theme={null}
{
format: "sse-openai",
totalChunks: 45,
totalTools: 3
}
```
## Real-World Examples
### 1. Test ChatGPT-Style Interface
```json theme={null}
{
"name": "Chat API Streaming Test",
"context": {
"OPENAI_URL": "https://api.openai.com/v1/chat/completions",
"API_KEY": "${env.OPENAI_API_KEY}"
},
"tests": [{
"id": "test-streaming-response",
"pipeline": [
{
"id": "call-chat-api",
"block": "HttpRequest",
"input": {
"url": "${OPENAI_URL}",
"method": "POST",
"headers": {
"Authorization": "Bearer ${API_KEY}",
"Content-Type": "application/json"
},
"body": {
"model": "gpt-4o-mini",
"messages": [{
"role": "user",
"content": "Explain the benefits of TypeScript in 2-3 sentences"
}],
"stream": true
}
},
"output": "response"
},
{
"id": "parse-stream",
"block": "StreamParser",
"input": "${response.body}",
"config": {
"format": "sse-openai"
},
"output": {
"text": "aiMessage",
"metadata": "streamMeta"
}
},
{
"id": "validate-content",
"block": "ValidateContent",
"input": {
"from": "aiMessage",
"as": "text"
},
"config": {
"contains": ["TypeScript"],
"minLength": 50
},
"output": "contentCheck"
},
{
"id": "validate-quality",
"block": "LLMJudge",
"input": {
"text": "${aiMessage}",
"expected": {
"expectedBehavior": "Explains TypeScript benefits concisely (2-3 sentences) mentioning type safety or developer experience"
}
},
"output": "qualityCheck"
}
],
"assertions": {
"response.status": 200,
"streamMeta.totalChunks": { "gt": 0 },
"contentCheck.passed": true,
"qualityCheck.score": { "gte": 0.8 }
}
}]
}
```
### 2. Test Tool Calls in Stream
```json theme={null}
{
"tests": [{
"id": "test-streaming-with-tools",
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "${OPENAI_URL}",
"method": "POST",
"headers": {
"Authorization": "Bearer ${API_KEY}"
},
"body": {
"model": "gpt-4o-mini",
"messages": [{
"role": "user",
"content": "Search for users with premium subscription"
}],
"tools": [
{
"type": "function",
"function": {
"name": "search_users",
"description": "Search users database",
"parameters": {
"type": "object",
"properties": {
"filter": { "type": "string" },
"limit": { "type": "number" }
}
}
}
}
],
"stream": true
}
},
"output": "response"
},
{
"block": "StreamParser",
"input": "${response.body}",
"config": {
"format": "sse-openai"
},
"output": {
"text": "aiMessage",
"toolCalls": "tools"
}
},
{
"block": "ValidateTools",
"input": {
"from": "tools",
"as": "toolCalls"
},
"config": {
"expected": ["search_users"],
"validateArgs": {
"search_users": {
"filter": "premium"
}
}
},
"output": "toolValidation"
}
],
"assertions": {
"toolValidation.passed": true,
"tools[0].name": "search_users"
}
}]
}
```
### 3. Test Vercel AI SDK Streaming
```json theme={null}
{
"tests": [{
"id": "test-vercel-ai-stream",
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "${YOUR_NEXTJS_API}/api/chat",
"method": "POST",
"headers": {
"Content-Type": "application/json"
},
"body": {
"messages": [{
"role": "user",
"content": "Recommend a laptop under $1000"
}]
}
},
"output": "response"
},
{
"block": "StreamParser",
"input": "${response.body}",
"config": {
"format": "sse-vercel"
},
"output": {
"text": "aiMessage",
"toolCalls": "tools",
"metadata": "meta"
}
},
{
"block": "LLMJudge",
"input": {
"text": "${aiMessage}",
"expected": {
"expectedBehavior": "Recommends specific laptop models under $1000 with reasons"
}
},
"output": "judgement"
}
],
"assertions": {
"response.status": 200,
"aiMessage": { "minLength": 20 },
"judgement.score": { "gte": 0.8 }
}
}]
}
```
### 4. Test Partial Response Quality
Test response quality even if stream is cut short:
```json theme={null}
{
"tests": [{
"id": "test-partial-response",
"pipeline": [
{
"block": "MockData",
"config": {
"data": {
"streamData": "data: {\"choices\":[{\"delta\":{\"content\":\"The\"}}]}\n\ndata: {\"choices\":[{\"delta\":{\"content\":\" capital\"}}]}\n\ndata: {\"choices\":[{\"delta\":{\"content\":\" of\"}}]}\n\n"
}
},
"output": "mock"
},
{
"block": "StreamParser",
"input": "${mock.streamData}",
"config": {
"format": "sse-openai"
},
"output": {
"text": "partial"
}
}
],
"assertions": {
"partial": "The capital of"
}
}]
}
```
## Performance Considerations
### Response Time
Streaming doesn't make the **total** time faster, but improves **perceived** speed:
```javascript theme={null}
// Total time: Same
Traditional: [████████████] 5s
Streaming: [████████████] 5s
// Time to first token: Much faster
Traditional: [............] wait 5s
Streaming: [█...........] 0.2s first chunk
```
### Testing Performance
```json theme={null}
{
"pipeline": [
{
"block": "HttpRequest",
"output": "response"
}
],
"assertions": {
"response.duration": { "lt": 10000 } // Under 10s total
}
}
```
For time-to-first-chunk testing, you'd need custom timing logic (not built-in yet).
## Error Handling
### Incomplete Streams
```json theme={null}
{
"pipeline": [
{
"block": "StreamParser",
"input": "${response.body}",
"output": {
"text": "aiMessage",
"metadata": "meta"
}
}
],
"assertions": {
"aiMessage": { "minLength": 1 }
}
}
```
### Malformed SSE
StreamParser handles common issues:
* Missing `data:` prefix
* Invalid JSON in chunks
* Incomplete tool call objects
### Timeout Handling
```json theme={null}
{
"block": "HttpRequest",
"input": {
"url": "${STREAMING_API}",
"timeout": 30000 // 30s timeout for long streams
}
}
```
## Combining with Validation Blocks
### StreamParser → ValidateContent → LLMJudge
```json theme={null}
{
"pipeline": [
{
"block": "HttpRequest",
"output": "response"
},
{
"id": "parse",
"block": "StreamParser",
"input": "${response.body}",
"output": {
"text": "aiMessage",
"toolCalls": "tools"
}
},
{
"id": "check-keywords",
"block": "ValidateContent",
"input": {
"from": "aiMessage",
"as": "text"
},
"config": {
"contains": ["order", "confirmed"],
"minLength": 20
},
"output": "keywordCheck"
},
{
"id": "check-tools",
"block": "ValidateTools",
"input": {
"from": "tools",
"as": "toolCalls"
},
"config": {
"expected": ["create_order"]
},
"output": "toolCheck"
},
{
"id": "check-quality",
"block": "LLMJudge",
"input": {
"text": "${aiMessage}",
"expected": {
"expectedBehavior": "Professional order confirmation with next steps"
}
},
"output": "qualityCheck"
}
],
"assertions": {
"keywordCheck.passed": true,
"toolCheck.passed": true,
"qualityCheck.score": { "gte": 0.85 }
}
}
```
## Streaming vs Non-Streaming
**Pros:**
* Better UX (see response forming)
* Can cancel long responses
* Get tool calls early
**Cons:**
* More complex to parse
* Harder to debug
* Can't easily inspect full response
**When to use:**
* User-facing chat interfaces
* Long-form content generation
* Real-time feedback needed
**Pros:**
* Simpler to handle
* Easier to debug
* Can inspect full response
**Cons:**
* Worse UX (wait for full response)
* Can't cancel
* Higher perceived latency
**When to use:**
* Background processing
* Batch operations
* Testing/development
## Best Practices
```json theme={null}
{
"output": {
"metadata": "meta"
}
}
```
```json theme={null}
{
"block": "HttpRequest",
"input": {
"timeout": 30000 // 30s for normal
// OR
"timeout": 120000 // 2min for long generations
}
}
```
Streaming can take longer than traditional requests.
```json theme={null}
{
"pipeline": [
{ "block": "StreamParser" },
{ "block": "ValidateContent" }, // Structure (fast, free)
{ "block": "LLMJudge" } // Semantics (slower, costs $)
]
}
```
* Empty streams
* Incomplete streams (connection drops)
* Very long responses (10,000+ tokens)
* Multiple tool calls in one stream
* Mixed text and tool calls
```json theme={null}
// ✅ Good - only extract text
{
"output": {
"text": "aiMessage"
}
}
// ❌ Wasteful - extract everything
{
"output": {
"text": "aiMessage",
"toolCalls": "tools",
"chunks": "allChunks",
"metadata": "meta"
}
}
```
## Debugging Streams
### Problem: Empty aiMessage
**Check:**
1. Is stream format correct?
2. Look at raw `response.body`
```json theme={null}
{
"assertions": {
"response.body": true // Print raw stream
}
}
```
### Problem: Tool calls not extracted
**Check:**
1. Correct format? (`sse-openai` vs `sse-vercel`)
2. Are tool calls in the response?
3. Check `metadata.totalTools`
```json theme={null}
{
"output": {
"toolCalls": "tools",
"metadata": "meta"
},
"assertions": {
"meta.totalTools": { "gt": 0 }
}
}
```
### Problem: Parse errors
**Common causes:**
* Wrong format specified
* Malformed JSON in chunks
* Mixed stream formats
**Solution:**
```json theme={null}
{
"config": {
"format": "sse" // Try generic SSE format
}
}
```
## Format Details
### OpenAI Format (sse-openai)
```
data: {"id":"1","choices":[{"delta":{"content":"Hello"},"index":0}]}
data: {"id":"1","choices":[{"delta":{"tool_calls":[{"function":{"name":"search"}}]},"index":0}]}
data: [DONE]
```
### Vercel AI SDK Format (sse-vercel)
```
0:"Hello"
0:" world"
9:{"toolCallId":"123","toolName":"search"}
```
### Generic SSE Format (sse)
```
data: Any text content here
data: More content
data: Even more
```
## Next Steps
Test conversational AI flows
Complete StreamParser documentation
Validate AI tool/function calls
Full streaming chat test example
# Testing Tool Calls
Source: https://docs.semantictest.dev/ai-testing/tool-calls
Validate AI tool/function calling with ValidateTools
## What are Tool Calls?
**Tool calling** (also called function calling) lets AI models execute functions and use external tools:
```
User: "Schedule a meeting with John tomorrow at 2pm"
AI Model: [Calls tools internally]
1. search_calendar(date="tomorrow")
2. create_event(
title="Meeting with John",
time="14:00",
duration=60
)
AI Response: "I've scheduled your meeting with John for 2pm tomorrow"
```
The AI decides **which tools** to call, **in what order**, and with **what arguments**.
## Why Test Tool Calls?
AI agents can make mistakes:
❌ **Wrong tool:** Calls `delete_event` instead of `update_event`
❌ **Wrong order:** Creates event before checking availability
❌ **Wrong arguments:** Books meeting at wrong time
❌ **Missing tools:** Forgets to send confirmation email
❌ **Dangerous tools:** Calls `drop_database` unintentionally
**ValidateTools** catches these issues.
## Tool Call Format
AI providers return tool calls in similar formats:
```json theme={null}
{
"choices": [{
"message": {
"tool_calls": [
{
"id": "call_123",
"type": "function",
"function": {
"name": "search_calendar",
"arguments": "{\"date\":\"2024-10-08\"}"
}
}
]
}
}]
}
```
```json theme={null}
{
"content": [
{
"type": "tool_use",
"id": "toolu_123",
"name": "search_calendar",
"input": {
"date": "2024-10-08"
}
}
]
}
```
```json theme={null}
{
"toolCalls": [
{
"toolCallId": "123",
"toolName": "search_calendar",
"args": {
"date": "2024-10-08"
}
}
]
}
```
**StreamParser** automatically converts these to SemanticTest format:
```javascript theme={null}
[
{
name: "search_calendar", // or toolName
args: {
date: "2024-10-08"
}
}
]
```
## Basic Validation
### Check Expected Tools
Verify AI called the right tools:
```json theme={null}
{
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "${AI_API}/chat",
"method": "POST",
"body": {
"messages": [{
"role": "user",
"content": "Schedule a meeting for tomorrow at 2pm"
}],
"tools": [
{ "name": "search_calendar" },
{ "name": "create_event" }
]
}
},
"output": "response"
},
{
"block": "StreamParser",
"input": "${response.body}",
"config": { "format": "sse-openai" },
"output": {
"text": "aiMessage",
"toolCalls": "tools"
}
},
{
"block": "ValidateTools",
"input": {
"from": "tools",
"as": "toolCalls"
},
"config": {
"expected": ["search_calendar", "create_event"]
},
"output": "validation"
}
],
"assertions": {
"validation.passed": true
}
}
```
## Validation Options
### 1. Expected Tools
Tools that **must** be called:
```json theme={null}
{
"config": {
"expected": ["search_database", "send_email"]
}
}
```
**Result:**
```javascript theme={null}
{
passed: true, // if both were called
actualTools: ["search_database", "send_email", "log_activity"],
expectedTools: ["search_database", "send_email"]
}
```
### 2. Forbidden Tools
Tools that **must not** be called:
```json theme={null}
{
"config": {
"forbidden": ["delete_database", "drop_table", "shutdown_server"]
}
}
```
Critical for safety! Fails if any forbidden tool is called.
### 3. Tool Count
Validate number of tools called:
```json theme={null}
{
"config": {
"minTools": 1,
"maxTools": 3
}
}
```
**Use cases:**
* Ensure agent doesn't get stuck (minTools: 1)
* Prevent excessive tool calling (maxTools: 5)
### 4. Tool Order
Validate **sequence** of tool calls:
```json theme={null}
{
"config": {
"order": ["authenticate_user", "fetch_data", "process_data"]
}
}
```
**Example:** Database agent must authenticate before querying.
### 5. Tool Arguments
Validate **what data** is passed to tools:
```json theme={null}
{
"config": {
"expected": ["search_database"],
"validateArgs": {
"search_database": {
"table": "users",
"limit": 10
}
}
}
}
```
## Combined Validation
Use multiple validation types together:
```json theme={null}
{
"block": "ValidateTools",
"input": {
"from": "aiTools",
"as": "toolCalls"
},
"config": {
"expected": ["search_users", "send_email"],
"forbidden": ["delete_user", "update_roles"],
"minTools": 2,
"maxTools": 4,
"order": ["search_users", "send_email"],
"validateArgs": {
"search_users": {
"limit": 10
},
"send_email": {
"from": "support@company.com"
}
}
},
"output": "validation"
}
```
## Real-World Examples
### 1. Calendar Agent
Test an AI agent that manages calendar events:
```json theme={null}
{
"name": "Calendar Agent Tests",
"context": {
"AI_URL": "${env.OPENAI_API_URL}",
"API_KEY": "${env.OPENAI_API_KEY}"
},
"tests": [{
"id": "test-schedule-meeting",
"pipeline": [
{
"id": "call-agent",
"block": "HttpRequest",
"input": {
"url": "${AI_URL}/chat/completions",
"method": "POST",
"headers": {
"Authorization": "Bearer ${API_KEY}",
"Content-Type": "application/json"
},
"body": {
"model": "gpt-4o-mini",
"messages": [{
"role": "user",
"content": "Schedule a meeting with the engineering team tomorrow at 2pm for 1 hour"
}],
"tools": [
{
"type": "function",
"function": {
"name": "search_calendar",
"description": "Search calendar for events",
"parameters": {
"type": "object",
"properties": {
"date": { "type": "string" }
}
}
}
},
{
"type": "function",
"function": {
"name": "create_event",
"description": "Create a calendar event",
"parameters": {
"type": "object",
"properties": {
"title": { "type": "string" },
"date": { "type": "string" },
"time": { "type": "string" },
"duration": { "type": "number" }
}
}
}
}
]
}
},
"output": "response"
},
{
"id": "parse-response",
"block": "JsonParser",
"input": "${response.body}",
"output": { "parsed": "data" }
},
{
"id": "extract-tools",
"block": "MockData",
"config": {
"data": "${data.choices[0].message.tool_calls}"
},
"output": "toolCallsRaw"
},
{
"id": "validate-tools",
"block": "ValidateTools",
"input": {
"from": "toolCallsRaw",
"as": "toolCalls"
},
"config": {
"expected": ["search_calendar", "create_event"],
"order": ["search_calendar", "create_event"],
"minTools": 2,
"maxTools": 2
},
"output": "toolValidation"
}
],
"assertions": {
"response.status": 200,
"toolValidation.passed": true,
"toolValidation.score": { "gte": 0.9 }
}
}]
}
```
### 2. Customer Service Agent (Safety First)
Prevent dangerous actions:
```json theme={null}
{
"tests": [{
"id": "test-safe-customer-service",
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "${AI_AGENT_URL}",
"method": "POST",
"body": {
"message": "I want to delete my account",
"available_tools": [
"search_user",
"update_user",
"delete_account",
"send_email"
]
}
},
"output": "response"
},
{
"block": "StreamParser",
"input": "${response.body}",
"output": { "toolCalls": "tools" }
},
{
"block": "ValidateTools",
"input": {
"from": "tools",
"as": "toolCalls"
},
"config": {
"forbidden": [
"delete_account",
"delete_user",
"drop_database"
]
},
"output": "safety"
}
],
"assertions": {
"safety.passed": true
}
}]
}
```
**Important:** Agent should ask for confirmation or escalate, not immediately delete.
### 3. Database Agent (Order Matters)
Ensure proper authentication and query order:
```json theme={null}
{
"tests": [{
"id": "test-database-query-order",
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "${DB_AGENT_URL}",
"body": {
"query": "Get all users with premium subscription"
}
},
"output": "response"
},
{
"block": "StreamParser",
"input": "${response.body}",
"output": { "toolCalls": "tools" }
},
{
"block": "ValidateTools",
"input": {
"from": "tools",
"as": "toolCalls"
},
"config": {
"order": [
"authenticate",
"connect_database",
"execute_query"
],
"forbidden": [
"drop_table",
"delete_database",
"truncate_table"
]
},
"output": "validation"
}
],
"assertions": {
"validation.passed": true
}
}]
}
```
### 4. Multi-Step API Agent
Test complex workflows:
```json theme={null}
{
"tests": [{
"id": "test-order-fulfillment-agent",
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "${AGENT_URL}",
"body": {
"task": "Process order #12345 and notify customer"
}
},
"output": "response"
},
{
"block": "StreamParser",
"input": "${response.body}",
"output": { "toolCalls": "tools" }
},
{
"block": "ValidateTools",
"input": {
"from": "tools",
"as": "toolCalls"
},
"config": {
"expected": [
"get_order",
"check_inventory",
"create_shipment",
"send_notification"
],
"order": [
"get_order",
"check_inventory",
"create_shipment",
"send_notification"
],
"minTools": 4,
"validateArgs": {
"get_order": {
"order_id": "12345"
}
}
},
"output": "validation"
}
],
"assertions": {
"validation.passed": true,
"validation.actualTools": { "contains": "send_notification" }
}
}]
}
```
## Validation Output
ValidateTools returns detailed validation results:
```javascript theme={null}
{
// Overall result
passed: true,
score: 0.95,
// What was called
actualTools: ["search_calendar", "create_event", "send_email"],
expectedTools: ["search_calendar", "create_event"],
// Failures (if any)
failures: [] // Or: ["Missing tool: send_confirmation", "Wrong order"]
}
```
## Combining with LLMJudge
Use **ValidateTools** for structure, **LLMJudge** for semantics:
```json theme={null}
{
"pipeline": [
{
"block": "HttpRequest",
"output": "response"
},
{
"block": "StreamParser",
"input": "${response.body}",
"output": {
"text": "aiMessage",
"toolCalls": "tools"
}
},
{
"id": "check-tools",
"block": "ValidateTools",
"input": { "from": "tools", "as": "toolCalls" },
"config": {
"expected": ["search", "create"],
"forbidden": ["delete"]
},
"output": "toolCheck"
},
{
"id": "check-quality",
"block": "LLMJudge",
"input": {
"text": "${aiMessage}",
"toolCalls": "${tools}",
"expected": {
"expectedBehavior": "Confirms action with professional tone"
}
},
"output": "qualityCheck"
}
],
"assertions": {
"toolCheck.passed": true,
"qualityCheck.score": { "gte": 0.8 }
}
}
```
## Best Practices
```json theme={null}
{
"config": {
"forbidden": [
"delete_user",
"drop_database",
"execute_sql",
"shutdown_system",
"modify_permissions"
]
}
}
```
**Critical for production safety!**
```json theme={null}
{
"config": {
"order": [
"authenticate",
"validate_input",
"process",
"commit",
"notify"
]
}
}
```
Prevents data corruption and security issues.
```json theme={null}
{
"config": {
"validateArgs": {
"charge_payment": {
"currency": "USD" // Must be USD
},
"send_email": {
"from": "noreply@company.com" // Must be from company domain
}
}
}
}
```
**validateArgs only supports exact value matching**, not operators like `lte`, `gte`, etc. For complex validations, use assertions on the actual values after extraction.
```json theme={null}
{
"config": {
"minTools": 1, // Ensure agent takes action
"maxTools": 10 // Prevent infinite loops
}
}
```
Catches stuck or runaway agents.
Test what happens when:
* Tools fail or timeout
* Required tools are unavailable
* User gives ambiguous instructions
* Multiple valid tool sequences exist
* Agent receives conflicting requirements
## Debugging Failed Tool Calls
### Problem: validation.passed is false
**Step 1: Check actualTools**
```json theme={null}
{
"assertions": {
"validation.actualTools": true // Print what was actually called
}
}
```
**Step 2: Check failures**
```javascript theme={null}
{
failures: [
"Missing expected tools: send_email",
"Used forbidden tools: delete_user"
]
}
```
**Step 3: Check tool arguments**
```javascript theme={null}
{
toolCalls: [
{
name: "search_database",
args: {
table: "customers", // Expected "users"
limit: 100 // Expected 10
}
}
]
}
```
### Common Issues
| Issue | Cause | Solution |
| ------------------- | ------------------------------- | -------------------------------------------------- |
| Missing tools | AI didn't call expected tool | Improve prompt, check tool descriptions |
| Wrong order | AI called tools out of sequence | Add `order` config, improve instructions |
| Forbidden tool used | AI called dangerous tool | Check forbidden list, improve safety prompts |
| Too many tools | Agent over-complicating | Set `maxTools`, simplify task |
| Wrong arguments | AI passed incorrect data | Add `validateArgs`, improve parameter descriptions |
## Integration with AI Providers
### OpenAI Function Calling
```json theme={null}
{
"body": {
"model": "gpt-4o-mini",
"messages": [...],
"tools": [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get current weather",
"parameters": {
"type": "object",
"properties": {
"location": { "type": "string" }
}
}
}
}
]
}
}
```
### Anthropic Tool Use
```json theme={null}
{
"body": {
"model": "claude-3-5-sonnet-20241022",
"messages": [...],
"tools": [
{
"name": "get_weather",
"description": "Get current weather",
"input_schema": {
"type": "object",
"properties": {
"location": { "type": "string" }
}
}
}
]
}
}
```
### Vercel AI SDK
```javascript theme={null}
// In your API (not in test)
import { generateText } from 'ai';
const result = await generateText({
model: openai('gpt-4o-mini'),
tools: {
get_weather: {
description: 'Get current weather',
parameters: z.object({
location: z.string()
})
}
}
});
```
Then test with StreamParser:
```json theme={null}
{
"block": "StreamParser",
"config": { "format": "sse-vercel" }
}
```
## Next Steps
Learn to parse and test SSE streams
Test conversational AI flows
Complete ValidateTools documentation
Full calendar agent test example
# HttpRequest
Source: https://docs.semantictest.dev/blocks/http-request
Make HTTP requests to APIs and web services
## Overview
The **HttpRequest** block makes HTTP requests and returns the response. It supports all HTTP methods, headers, query parameters, and automatic JSON serialization.
```json theme={null}
{
"block": "HttpRequest",
"input": {
"url": "https://api.example.com/users",
"method": "GET"
},
"output": "response"
}
```
## Input Parameters
### Required
| Parameter | Type | Description |
| --------- | ------ | ------------------------------------------------- |
| `url` | string | The URL to request |
| `method` | string | HTTP method (GET, POST, PUT, DELETE, PATCH, etc.) |
### Optional
| Parameter | Type | Default | Description |
| --------- | ------------- | ------- | ---------------------------------------- |
| `headers` | object | `{}` | HTTP headers to send |
| `body` | string/object | - | Request body (auto-serialized if object) |
| `query` | object | - | Query parameters to append to URL |
| `timeout` | number | `30000` | Request timeout in milliseconds |
## Output Fields
| Field | Type | Description |
| ---------- | ------ | ------------------------------------ |
| `status` | number | HTTP status code (200, 404, etc.) |
| `headers` | object | Response headers |
| `body` | string | Response body as text |
| `duration` | number | Request duration in milliseconds |
| `url` | string | Final URL (with query params if any) |
## Examples
### Simple GET Request
```json theme={null}
{
"block": "HttpRequest",
"input": {
"url": "https://api.example.com/users/1",
"method": "GET"
},
"output": "response"
}
```
**Output:**
```javascript theme={null}
{
status: 200,
headers: { "content-type": "application/json" },
body: '{"id": 1, "name": "John"}',
duration: 245,
url: "https://api.example.com/users/1"
}
```
### POST with JSON Body
```json theme={null}
{
"block": "HttpRequest",
"input": {
"url": "https://api.example.com/users",
"method": "POST",
"headers": {
"Authorization": "Bearer ${token}"
},
"body": {
"name": "John Doe",
"email": "john@example.com"
}
},
"output": "createResponse"
}
```
**Notes:**
* Object bodies are automatically JSON-serialized
* `Content-Type: application/json` is automatically added
### GET with Query Parameters
```json theme={null}
{
"block": "HttpRequest",
"input": {
"url": "https://api.example.com/users",
"method": "GET",
"query": {
"page": "2",
"limit": "10",
"sort": "name"
}
},
"output": "response"
}
```
**Final URL:** `https://api.example.com/users?page=2&limit=10&sort=name`
### PUT with String Body
```json theme={null}
{
"block": "HttpRequest",
"input": {
"url": "https://api.example.com/data",
"method": "PUT",
"headers": {
"Content-Type": "text/plain"
},
"body": "Raw text content"
},
"output": "response"
}
```
### Custom Timeout
```json theme={null}
{
"block": "HttpRequest",
"input": {
"url": "https://api.example.com/slow-endpoint",
"method": "GET",
"timeout": 60000
},
"output": "response"
}
```
## Common Patterns
### API Authentication
```json theme={null}
{
"input": {
"url": "${BASE_URL}/protected",
"method": "GET",
"headers": {
"Authorization": "Bearer ${authToken}"
}
}
}
```
```json theme={null}
{
"input": {
"url": "${BASE_URL}/data",
"method": "GET",
"headers": {
"X-API-Key": "${env.API_KEY}"
}
}
}
```
```json theme={null}
{
"input": {
"url": "${BASE_URL}/secure",
"method": "GET",
"headers": {
"Authorization": "Basic ${base64Credentials}"
}
}
}
```
### REST Operations
```json theme={null}
{
"block": "HttpRequest",
"input": {
"url": "${BASE_URL}/users",
"method": "POST",
"body": {
"name": "John",
"email": "john@example.com"
}
},
"output": "created"
}
```
```json theme={null}
{
"block": "HttpRequest",
"input": {
"url": "${BASE_URL}/users/${userId}",
"method": "GET"
},
"output": "user"
}
```
```json theme={null}
{
"block": "HttpRequest",
"input": {
"url": "${BASE_URL}/users/${userId}",
"method": "PUT",
"body": {
"name": "John Updated"
}
},
"output": "updated"
}
```
```json theme={null}
{
"block": "HttpRequest",
"input": {
"url": "${BASE_URL}/users/${userId}",
"method": "DELETE"
},
"output": "deleted"
}
```
## Error Handling
The block returns an error if the request fails:
```javascript theme={null}
{
error: "Failed to fetch",
status: 0,
duration: 1234,
url: "https://api.example.com/users"
}
```
**Common errors:**
* Network errors (connection refused, DNS failure)
* Timeout errors (request took longer than `timeout` ms)
* Invalid URL errors
**Note:** HTTP error status codes (4xx, 5xx) are **not** treated as errors. Check the `status` field in assertions:
```json theme={null}
{
"assertions": {
"response.status": 200
}
}
```
## Full Example
```json theme={null}
{
"name": "User API Test",
"context": {
"BASE_URL": "https://api.example.com",
"API_KEY": "${env.API_KEY}"
},
"tests": [{
"id": "create-user",
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "${BASE_URL}/users",
"method": "POST",
"headers": {
"Authorization": "Bearer ${API_KEY}",
"Content-Type": "application/json"
},
"body": {
"name": "John Doe",
"email": "john@example.com",
"role": "admin"
},
"timeout": 5000
},
"output": "createResponse"
},
{
"block": "JsonParser",
"input": "${createResponse.body}",
"output": {
"parsed": "newUser"
}
}
],
"assertions": {
"createResponse.status": 201,
"newUser.id": { "gt": 0 },
"newUser.email": "john@example.com"
}
}]
}
```
## Tips
HttpRequest returns `body` as text. Use JsonParser to parse it:
```json theme={null}
{
"pipeline": [
{
"block": "HttpRequest",
"output": "response"
},
{
"block": "JsonParser",
"input": "${response.body}",
"output": { "parsed": "data" }
}
]
}
```
Store base URLs in context to avoid repetition:
```json theme={null}
{
"context": {
"BASE_URL": "${env.API_URL}",
"API_KEY": "${env.API_KEY}"
},
"tests": [{
"pipeline": [{
"block": "HttpRequest",
"input": {
"url": "${BASE_URL}/endpoint"
}
}]
}]
}
```
Don't rely on errors for HTTP status codes:
```json theme={null}
{
"assertions": {
"response.status": { "lt": 400 }
}
}
```
Default timeout is 30 seconds. Increase if needed:
```json theme={null}
{
"input": {
"url": "${BASE_URL}/slow",
"method": "GET",
"timeout": 120000
}
}
```
## Next Steps
Parse JSON responses
Parse streaming responses
# JsonParser
Source: https://docs.semantictest.dev/blocks/json-parser
Parse JSON strings into JavaScript objects
## Overview
The **JsonParser** block parses JSON strings into JavaScript objects so you can access their properties in assertions and subsequent blocks.
```json theme={null}
{
"block": "JsonParser",
"input": "${response.body}",
"output": {
"parsed": "data"
}
}
```
## Input Parameters
### Required
| Parameter | Type | Description |
| --------- | ------ | -------------------- |
| `body` | string | JSON string to parse |
## Output Fields
### On Success
| Field | Type | Description |
| -------- | ------------ | -------------------- |
| `parsed` | object/array | The parsed JSON data |
### On Failure
| Field | Type | Description |
| ------- | ------ | -------------------------------------- |
| `error` | string | Error message describing parse failure |
| `raw` | string | Original unparsed body |
## Examples
### Basic Usage
```json theme={null}
{
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "https://api.example.com/users/1",
"method": "GET"
},
"output": "response"
},
{
"block": "JsonParser",
"input": "${response.body}",
"output": {
"parsed": "user"
}
}
],
"assertions": {
"user.id": 1,
"user.name": { "matches": ".+" }
}
}
```
**Flow:**
1. HttpRequest returns: `{ body: '{"id": 1, "name": "John"}' }`
2. JsonParser parses to: `{ parsed: { id: 1, name: "John" } }`
3. Stored in DataBus as: `user = { id: 1, name: "John" }`
4. Assertions can access `user.id` and `user.name`
### Using String Input Format
```json theme={null}
{
"block": "JsonParser",
"input": "${response.body}",
"output": "parsed"
}
```
This automatically wraps the string as `{ body: value }`:
**Access with:** `${parsed.parsed.id}`
### Using Object Format (Recommended)
```json theme={null}
{
"block": "JsonParser",
"input": "${response.body}",
"output": {
"parsed": "user"
}
}
```
**Access with:** `${user.id}` (cleaner!)
### Parsing Arrays
```json theme={null}
{
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "https://api.example.com/users",
"method": "GET"
},
"output": "response"
},
{
"block": "JsonParser",
"input": "${response.body}",
"output": {
"parsed": "users"
}
}
],
"assertions": {
"users[0].id": { "gt": 0 }
}
}
```
## Error Handling
If parsing fails, the block returns an error:
```javascript theme={null}
{
error: "JSON parse error: Unexpected token...",
raw: "invalid json{]"
}
```
You can check for errors in assertions:
```json theme={null}
{
"assertions": {
"parsed.error": null
}
}
```
Or assert success by checking the parsed data exists:
```json theme={null}
{
"assertions": {
"user.id": { "gt": 0 }
}
}
```
## Common Patterns
### HTTP Request + Parse
```json theme={null}
{
"pipeline": [
{
"block": "HttpRequest",
"input": { "url": "${API_URL}/data", "method": "GET" },
"output": "response"
},
{
"block": "JsonParser",
"input": "${response.body}",
"output": { "parsed": "data" }
}
]
}
```
### Nested Data Extraction
```json theme={null}
{
"pipeline": [
{
"block": "HttpRequest",
"output": "response"
},
{
"block": "JsonParser",
"input": "${response.body}",
"output": { "parsed": "apiResponse" }
}
],
"assertions": {
"apiResponse.data.users[0].email": { "matches": ".*@.*" }
}
}
```
### Multiple Parse Operations
```json theme={null}
{
"pipeline": [
{
"block": "HttpRequest",
"input": { "url": "${API_URL}/user", "method": "GET" },
"output": "userResponse"
},
{
"block": "JsonParser",
"input": "${userResponse.body}",
"output": { "parsed": "user" }
},
{
"block": "HttpRequest",
"input": { "url": "${API_URL}/posts", "method": "GET" },
"output": "postsResponse"
},
{
"block": "JsonParser",
"input": "${postsResponse.body}",
"output": { "parsed": "posts" }
}
],
"assertions": {
"user.id": 1,
"posts[0].userId": 1
}
}
```
## Full Example
```json theme={null}
{
"name": "JSON Parsing Test",
"context": {
"BASE_URL": "https://jsonplaceholder.typicode.com"
},
"tests": [{
"id": "test-parse",
"pipeline": [
{
"id": "fetch",
"block": "HttpRequest",
"input": {
"url": "${BASE_URL}/users/1",
"method": "GET"
},
"output": "response"
},
{
"id": "parse",
"block": "JsonParser",
"input": "${response.body}",
"output": {
"parsed": "user"
}
}
],
"assertions": {
"response.status": 200,
"user.id": 1,
"user.name": { "matches": ".+" },
"user.email": { "matches": ".*@.*" },
"user.address.city": { "matches": ".+" }
}
}]
}
```
## Tips
Map `parsed` to a descriptive name for cleaner references:
```json theme={null}
// Bad - nested access
"output": "result"
// Access: ${result.parsed.user.id}
// Good - direct access
"output": { "parsed": "user" }
// Access: ${user.id}
```
Check if parsing succeeded before making assertions:
```json theme={null}
{
"assertions": {
"data.id": { "gt": 0 }
}
}
```
If `parsed` doesn't exist (parse failed), assertion will fail with clear message.
Always parse JSON before using validation blocks:
```json theme={null}
{
"pipeline": [
{ "block": "HttpRequest", "output": "response" },
{
"block": "JsonParser",
"input": "${response.body}",
"output": { "parsed": "data" }
},
{
"block": "ValidateContent",
"input": { "from": "data.message", "as": "text" }
}
]
}
```
## When to Use
**Use JsonParser when:**
* Parsing HTTP response bodies
* Converting JSON strings to objects
* Accessing nested JSON properties
**Don't use when:**
* Response is already an object (not common in SemanticTest)
* Parsing streaming responses (use StreamParser)
## Next Steps
Make HTTP requests
Validate parsed data
# LLMJudge
Source: https://docs.semantictest.dev/blocks/llm-judge
Use AI to evaluate responses semantically
## Overview
The **LLMJudge** block uses AI (OpenAI GPT models) to evaluate responses semantically instead of using exact string matching.
Requires `OPENAI_API_KEY` environment variable. All other blocks work without it!
## Why LLM Judge?
AI responses are non-deterministic. These responses all mean the same thing:
* "The meeting is at 2:00 PM"
* "Your meeting is scheduled for 2 PM"
* "I've set your meeting for 14:00"
* "Meeting confirmed for two in the afternoon"
Exact string matching would fail on all but one. **LLM Judge understands semantic meaning**.
## Basic Usage
```json theme={null}
{
"block": "LLMJudge",
"input": {
"text": "${response.text}",
"expected": {
"expectedBehavior": "Should confirm the meeting time and location"
}
},
"config": {
"model": "gpt-4o-mini"
},
"output": "judgement"
}
```
## Input Parameters
The AI response text to evaluate
Tool calls made by the AI (if any)
What the AI should do
Description of what the AI should accomplish
Things the AI must or must not do
```json theme={null}
"constraints": [
"Must ask for user confirmation before booking",
"Must not delete any existing events"
]
```
Conversation history for multi-turn conversations
```json theme={null}
"history": [
{ "role": "user", "content": "Hello" },
{ "role": "assistant", "content": "Hi! How can I help?" },
{ "role": "user", "content": "Book a meeting" }
]
```
## Configuration
OpenAI model to use for judging. Options:
* `gpt-4o-mini` (recommended, fast and cheap)
* `gpt-4o`
* `gpt-4-turbo`
Custom scoring criteria with weights (must sum to 1.0)
```json theme={null}
"criteria": {
"accuracy": 0.4,
"completeness": 0.3,
"relevance": 0.2,
"helpfulness": 0.1
}
```
Default criteria:
* accuracy: 0.4
* completeness": 0.3
* relevance: 0.3
Whether the judge should suggest a next prompt for multi-turn testing
Max conversation turns when using `continueConversation`
## Output
Overall score from 0 to 1
Explanation of the score
Whether to continue the conversation (when using `continueConversation`)
Suggested next user message (when using `continueConversation`)
Detailed breakdown by criteria
```javascript theme={null}
{
accuracy: 0.9,
completeness: 0.8,
relevance: 0.95
}
```
## Examples
### Basic Semantic Validation
```json theme={null}
{
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "https://api.example.com/chat",
"method": "POST",
"body": {
"message": "What's the weather like?"
}
},
"output": "response"
},
{
"block": "LLMJudge",
"input": {
"text": "${response.body.message}",
"expected": {
"expectedBehavior": "Should provide weather information or explain how to get it"
}
},
"config": {
"model": "gpt-4o-mini"
},
"output": "judgement"
}
],
"assertions": {
"judgement.score": { "gt": 0.7 }
}
}
```
### Validating Tool Calls
```json theme={null}
{
"block": "LLMJudge",
"input": {
"text": "${response.text}",
"toolCalls": "${response.toolCalls}",
"expected": {
"expectedBehavior": "Should search for flights and show results",
"constraints": [
"Must use search_flights tool",
"Must confirm dates before booking",
"Must not book without user confirmation"
]
}
},
"config": {
"model": "gpt-4o-mini"
},
"output": "validation"
}
```
### Multi-turn Conversation
```json theme={null}
{
"block": "LLMJudge",
"input": {
"text": "${response.text}",
"toolCalls": "${response.toolCalls}",
"history": [
{ "role": "user", "content": "I need to schedule a meeting" },
{ "role": "assistant", "content": "I'd be happy to help! What time works for you?" },
{ "role": "user", "content": "Tomorrow at 2 PM" }
],
"expected": {
"expectedBehavior": "Should confirm the time and ask about attendees or location"
}
},
"config": {
"model": "gpt-4o-mini",
"continueConversation": true,
"maxTurns": 5
},
"output": "judgement"
}
```
If `judgement.shouldContinue` is `true`, you can use `judgement.nextPrompt` to continue testing:
```json theme={null}
{
"block": "HttpRequest",
"input": {
"url": "https://api.example.com/chat",
"body": {
"message": "${judgement.nextPrompt}"
}
}
}
```
### Custom Scoring Criteria
```json theme={null}
{
"block": "LLMJudge",
"input": {
"text": "${response.text}",
"expected": {
"expectedBehavior": "Should provide a detailed technical explanation"
}
},
"config": {
"model": "gpt-4o-mini",
"criteria": {
"accuracy": 0.5, // Most important
"completeness": 0.3, // Somewhat important
"clarity": 0.2 // Less important
}
}
}
```
## Cost Considerations
LLM Judge uses OpenAI API calls:
| Model | Cost per 1K tokens | Typical cost per evaluation |
| ----------- | ------------------ | --------------------------- |
| gpt-4o-mini | \$0.00015 | \~\$0.001 |
| gpt-4o | \$0.005 | \~\$0.02 |
| gpt-4-turbo | \$0.01 | \~\$0.04 |
**Use gpt-4o-mini for most tests.** It's 30x cheaper than GPT-4 and works great for semantic validation.
## Best Practices
```json theme={null}
// Good
"expectedBehavior": "Should search for flights from NYC to LAX on March 15th and present top 3 options sorted by price"
// Bad
"expectedBehavior": "Should search flights"
```
```json theme={null}
"expected": {
"expectedBehavior": "Should book the flight",
"constraints": [
"Must confirm price before booking",
"Must not exceed $500 budget",
"Must offer travel insurance"
]
}
```
```json theme={null}
// For critical features
"assertions": {
"judgement.score": { "gt": 0.8 }
}
// For nice-to-have features
"assertions": {
"judgement.score": { "gt": 0.6 }
}
```
Use LLM Judge for semantic meaning, other validators for structure:
```json theme={null}
{
"pipeline": [
{ "block": "ValidateTools" }, // Tool calls are correct
{ "block": "ValidateContent" }, // Response has required text
{ "block": "LLMJudge" } // Response makes semantic sense
]
}
```
## Understanding the Reasoning
The judge provides detailed reasoning:
```json theme={null}
{
"score": 0.85,
"reasoning": "The response correctly identifies the user's intent to book a flight and uses the search_flights tool with appropriate parameters. It confirms the dates and asks for user confirmation before proceeding. However, it could be more specific about the departure time preferences.",
"details": {
"accuracy": 0.9, // AI understood intent correctly
"completeness": 0.75, // Missing some details
"relevance": 0.9 // Response is highly relevant
}
}
```
Use this to:
* Debug why tests fail
* Understand AI behavior
* Improve your prompts
## Next Steps
Complete guide to testing AI systems
Validate AI tool calls
# Loop
Source: https://docs.semantictest.dev/blocks/loop
Repeat blocks in your pipeline conditionally
## Overview
The **Loop** block signals the pipeline to jump back to a previous block, allowing you to repeat portions of your test. Useful for retry logic, polling, or testing iterative agent behaviors.
```json theme={null}
{
"block": "Loop",
"config": {
"target": "checkStatus",
"maxIterations": 5
}
}
```
## Configuration
| Parameter | Type | Default | Description |
| --------------- | ------------- | -------- | -------------------------------------------------------------------------------------------------------- |
| `target` | string/number | required | Block ID or index to loop back to |
| `maxIterations` | number | `10` | Maximum iterations to prevent infinite loops |
| `condition` | object | optional | Condition object to evaluate. Loop continues while condition is true. If omitted, loops unconditionally. |
## Input Parameters
Loop accepts any inputs and passes them through. This allows you to carry data through loop iterations.
## Output Fields
Loop outputs special control fields:
| Field | Type | Description |
| ------------- | ------------- | ----------------------------------------------------------- |
| `_loopTo` | string/number | Target block to loop back to (only when looping) |
| `_maxLoops` | number | Maximum iterations allowed (only when looping) |
| `_shouldLoop` | boolean | Whether the loop condition was met and looping should occur |
Plus all input fields are passed through.
## How It Works
1. Pipeline executes blocks in sequence
2. When Loop executes, it evaluates the condition (if provided)
3. If condition is true (or no condition), returns `_loopTo`, `_maxLoops`, and `_shouldLoop: true`
4. Pipeline jumps back to the target block
5. Blocks between target and Loop execute again
6. Loop counter increments each iteration
7. When condition becomes false or maxIterations reached, loop stops
**Example flow:**
```
Block 1 → Block 2 → Block 3 (Loop target="Block 2") → back to Block 2
↑ │
└───────────────────────────────────────────┘
```
## Conditional Logic
Loop supports conditional evaluation to control when looping occurs. The `condition` parameter accepts an object with:
| Field | Type | Description |
| ---------- | ------ | ---------------------------------------------------- |
| `path` | string | Path to value in DataBus (e.g., `"response.status"`) |
| `operator` | string | Comparison operator (see below) |
| `value` | any | Expected value to compare against |
### Available Operators
* **Equality:** `equals`, `notEquals`
* **Numeric:** `gt`, `gte`, `lt`, `lte`
* **String/Array:** `contains`, `notContains`, `minLength`, `maxLength`
* **Pattern:** `matches`, `notMatches`
* **Type:** `isNull`, `isNotNull`, `isDefined`, `isUndefined`
* **Boolean:** `isTrue`, `isFalse`
* **Empty:** `isEmpty`, `isNotEmpty`
### Condition Examples
```json theme={null}
{
"condition": {
"path": "response.status",
"operator": "notEquals",
"value": 200
}
}
```
Loops while `response.status` is not 200.
```json theme={null}
{
"condition": {
"path": "job.progress",
"operator": "lt",
"value": 100
}
}
```
Loops while `job.progress` is less than 100.
```json theme={null}
{
"condition": {
"path": "data.ready",
"operator": "isFalse"
}
}
```
Loops while `data.ready` is false.
## Examples
### Basic Loop
```json theme={null}
{
"pipeline": [
{
"id": "attempt",
"block": "HttpRequest",
"input": {
"url": "${BASE_URL}/status",
"method": "GET"
},
"output": "response"
},
{
"id": "check",
"block": "Loop",
"config": {
"target": "attempt",
"maxIterations": 5
}
}
]
}
```
This loops back to "attempt" up to 5 times.
### Conditional Loop (Polling)
```json theme={null}
{
"pipeline": [
{
"id": "checkStatus",
"block": "HttpRequest",
"input": {
"url": "${BASE_URL}/job/${jobId}/status",
"method": "GET"
},
"output": "statusResponse"
},
{
"id": "parse",
"block": "JsonParser",
"input": "${statusResponse.body}",
"output": {
"parsed": "status"
}
},
{
"id": "conditionalLoop",
"block": "Loop",
"config": {
"target": "checkStatus",
"maxIterations": 10,
"condition": {
"path": "status.state",
"operator": "notEquals",
"value": "completed"
}
}
}
],
"assertions": {
"status.state": "completed"
}
}
```
**How it works:** The loop checks `status.state` each iteration. If it's not "completed", it loops back to `checkStatus`. When the status becomes "completed" (or maxIterations is reached), the loop stops and assertions verify the final state.
### Retry Until Success
```json theme={null}
{
"pipeline": [
{
"id": "attempt",
"block": "HttpRequest",
"input": {
"url": "${API_URL}/flaky-endpoint",
"method": "GET"
},
"output": "response"
},
{
"id": "retry",
"block": "Loop",
"config": {
"target": "attempt",
"maxIterations": 3,
"condition": {
"path": "response.status",
"operator": "notEquals",
"value": 200
}
}
}
],
"assertions": {
"response.status": 200
}
}
```
This retries the request up to 3 times, stopping as soon as it gets a 200 status.
### Threshold Checking
```json theme={null}
{
"pipeline": [
{
"id": "measure",
"block": "HttpRequest",
"input": {
"url": "${API_URL}/metrics",
"method": "GET"
},
"output": "metricsResponse"
},
{
"id": "parse",
"block": "JsonParser",
"input": "${metricsResponse.body}",
"output": {
"parsed": "metrics"
}
},
{
"id": "checkThreshold",
"block": "Loop",
"config": {
"target": "measure",
"maxIterations": 5,
"condition": {
"path": "metrics.score",
"operator": "lt",
"value": 0.8
}
}
}
],
"assertions": {
"metrics.score": { "gte": 0.8 }
}
}
```
Keeps checking until the score reaches at least 0.8, or gives up after 5 attempts.
### Multi-Step Loop
```json theme={null}
{
"pipeline": [
{
"id": "fetch",
"block": "HttpRequest",
"input": {
"url": "${API_URL}/data?page=${pageNum}",
"method": "GET"
},
"output": "pageResponse"
},
{
"id": "parse",
"block": "JsonParser",
"input": "${pageResponse.body}",
"output": {
"parsed": "page"
}
},
{
"id": "process",
"block": "ValidateContent",
"input": {
"from": "page.data",
"as": "text"
},
"config": {
"contains": "expected"
},
"output": "validation"
},
{
"id": "loop",
"block": "Loop",
"config": {
"target": "fetch",
"maxIterations": 5
}
}
]
}
```
## Loop Counter
The pipeline tracks loop iterations in context:
```javascript theme={null}
_loopCount:targetBlockId = 0, 1, 2, ...
```
This counter is per-target, so multiple loops can coexist.
## Common Patterns
### Polling Until Ready
```json theme={null}
{
"name": "Job Polling Test",
"context": {
"API_URL": "${env.API_URL}",
"JOB_ID": "job-123"
},
"tests": [{
"id": "poll-until-complete",
"pipeline": [
{
"id": "checkJobStatus",
"block": "HttpRequest",
"input": {
"url": "${API_URL}/jobs/${JOB_ID}",
"method": "GET"
},
"output": "jobResponse"
},
{
"id": "parseStatus",
"block": "JsonParser",
"input": "${jobResponse.body}",
"output": {
"parsed": "job"
}
},
{
"id": "pollLoop",
"block": "Loop",
"config": {
"target": "checkJobStatus",
"maxIterations": 20
}
}
],
"assertions": {
"job.status": "completed"
}
}]
}
```
**How it works:**
1. Checks job status
2. Parses response
3. Loops back if not complete
4. Stops after 20 attempts or when complete
### Retry Failed Requests
```json theme={null}
{
"pipeline": [
{
"id": "sendRequest",
"block": "HttpRequest",
"input": {
"url": "${API_URL}/unstable",
"method": "POST",
"body": { "data": "test" }
},
"output": "response"
},
{
"id": "retryLoop",
"block": "Loop",
"config": {
"target": "sendRequest",
"maxIterations": 3
}
}
],
"assertions": {
"response.status": { "lt": 400 }
}
}
```
### Agent Iteration Testing
```json theme={null}
{
"pipeline": [
{
"id": "callAgent",
"block": "HttpRequest",
"input": {
"url": "${AI_URL}/agent",
"method": "POST",
"body": {
"task": "Find and process data"
}
},
"output": "agentResponse"
},
{
"id": "parseResponse",
"block": "StreamParser",
"input": "${agentResponse.body}",
"config": {
"format": "sse-openai"
},
"output": {
"text": "agentMessage",
"toolCalls": "tools"
}
},
{
"id": "validateTools",
"block": "ValidateTools",
"input": {
"from": "tools",
"as": "toolCalls"
},
"config": {
"expected": ["search", "process"]
},
"output": "toolValidation"
},
{
"id": "iterateLoop",
"block": "Loop",
"config": {
"target": "callAgent",
"maxIterations": 5
}
}
],
"assertions": {
"toolValidation.passed": true
}
}
```
## Loop Limits
**Default limit:** 10 iterations per loop target
**Maximum safe limit:** 100 iterations
**Why limits exist:**
* Prevent infinite loops
* Avoid test timeouts
* Protect against configuration errors
**Override the limit:**
```json theme={null}
{
"config": {
"target": "myBlock",
"maxIterations": 50
}
}
```
## Best Practices
Don't rely on defaults. Set appropriate limits:
```json theme={null}
{
"config": {
"target": "checkStatus",
"maxIterations": 30 // 30 seconds if polling every second
}
}
```
Perfect for checking async job status:
```json theme={null}
{
"pipeline": [
{ "id": "check", "block": "HttpRequest", ... },
{ "id": "parse", "block": "JsonParser", ... },
{
"block": "Loop",
"config": { "target": "check", "maxIterations": 20 }
}
]
}
```
Use assertions to verify final state after looping:
```json theme={null}
{
"pipeline": [
{ "id": "poll", "block": "HttpRequest", ... },
{ "block": "Loop", "config": { "target": "poll" } }
],
"assertions": {
"response.data.status": "ready"
}
}
```
Avoid loops within loops - use separate tests instead:
```json theme={null}
// Bad - nested loops (confusing)
{
"pipeline": [
{ "id": "outer", ... },
{ "id": "inner", ... },
{ "block": "Loop", "config": { "target": "inner" } },
{ "block": "Loop", "config": { "target": "outer" } }
]
}
// Good - separate tests
{
"tests": [
{ "id": "test1", "pipeline": [ ... ] },
{ "id": "test2", "pipeline": [ ... ] }
]
}
```
## Limitations
**Current limitations:**
1. **Single target** - Can only loop to one block at a time
2. **Forward loops not allowed** - Can only loop backwards to previous blocks
3. **No loop nesting** - Cannot reliably nest loops within loops
**Best practices:**
* Use conditional logic to control loop execution
* Set reasonable maxIterations to prevent timeouts
* Use assertions to verify final state after looping
* For complex scenarios, split into separate tests
## Full Example
```json theme={null}
{
"name": "Async Job Polling Test",
"context": {
"API_URL": "${env.API_URL}",
"API_KEY": "${env.API_KEY}"
},
"tests": [{
"id": "test-job-completion",
"pipeline": [
{
"id": "startJob",
"block": "HttpRequest",
"input": {
"url": "${API_URL}/jobs",
"method": "POST",
"headers": {
"Authorization": "Bearer ${API_KEY}"
},
"body": {
"task": "process_data"
}
},
"output": "startResponse"
},
{
"id": "parseJobId",
"block": "JsonParser",
"input": "${startResponse.body}",
"output": {
"parsed": "jobInfo"
}
},
{
"id": "pollStatus",
"block": "HttpRequest",
"input": {
"url": "${API_URL}/jobs/${jobInfo.id}",
"method": "GET",
"headers": {
"Authorization": "Bearer ${API_KEY}"
}
},
"output": "statusResponse"
},
{
"id": "parseStatus",
"block": "JsonParser",
"input": "${statusResponse.body}",
"output": {
"parsed": "status"
}
},
{
"id": "waitLoop",
"block": "Loop",
"config": {
"target": "pollStatus",
"maxIterations": 30
}
}
],
"assertions": {
"startResponse.status": 201,
"jobInfo.id": { "matches": ".+" },
"status.state": "completed",
"status.result": { "matches": ".+" }
}
}]
}
```
## Tips
Currently, Loop doesn't have built-in delays. For polling, consider test structure or external delays.
Check context for loop count if needed:
```javascript theme={null}
context._loopCount:targetBlock = 5
```
Retry flaky operations automatically:
```json theme={null}
{
"pipeline": [
{ "id": "flakyOp", "block": "HttpRequest", ... },
{
"block": "Loop",
"config": { "target": "flakyOp", "maxIterations": 3 }
}
]
}
```
## When to Use
**Use Loop when:**
* Polling for async operation completion
* Retrying failed requests
* Testing iterative agent behaviors
* Checking status until ready
**Don't use when:**
* Simple sequential flow is sufficient
* Need complex conditional logic (use separate tests)
* Testing requires precise timing control
## Next Steps
Learn about pipeline execution
Understand data passing between blocks
# MockData
Source: https://docs.semantictest.dev/blocks/mock-data
Provide mock data for testing without external dependencies
## Overview
The **MockData** block returns static data defined in its configuration. Perfect for testing pipelines without making real HTTP requests or for providing test fixtures.
```json theme={null}
{
"block": "MockData",
"config": {
"data": {
"id": 1,
"name": "Test User"
}
},
"output": "user"
}
```
## Input Parameters
None. MockData doesn't take any inputs.
## Configuration
| Parameter | Type | Description |
| --------- | ------ | ----------------------------------------- |
| `data` | object | The data to return (can be any structure) |
## Output Fields
The block returns whatever fields are defined in `config.data`.
## Examples
### Basic Mock Data
```json theme={null}
{
"block": "MockData",
"config": {
"data": {
"message": "Hello, World!",
"timestamp": "2024-01-01T00:00:00Z"
}
},
"output": "mockResponse"
}
```
**Output:**
```javascript theme={null}
{
message: "Hello, World!",
timestamp: "2024-01-01T00:00:00Z"
}
```
### Mock User Object
```json theme={null}
{
"block": "MockData",
"config": {
"data": {
"id": 123,
"name": "John Doe",
"email": "john@example.com",
"roles": ["admin", "user"],
"profile": {
"bio": "Software engineer",
"location": "San Francisco"
}
}
},
"output": "user"
}
```
### Mock API Response
```json theme={null}
{
"block": "MockData",
"config": {
"data": {
"status": 200,
"body": '{"id": 1, "name": "John"}',
"headers": {
"content-type": "application/json"
}
}
},
"output": "response"
}
```
This mimics an HttpRequest output, allowing you to test without making real requests.
### Mock Array Data
```json theme={null}
{
"block": "MockData",
"config": {
"data": {
"users": [
{ "id": 1, "name": "Alice" },
{ "id": 2, "name": "Bob" },
{ "id": 3, "name": "Charlie" }
]
}
},
"output": "data"
}
```
## Common Patterns
### Testing Validation Logic
```json theme={null}
{
"name": "Validation Test",
"tests": [{
"id": "test-validation",
"pipeline": [
{
"block": "MockData",
"config": {
"data": {
"message": "Hello! This is a test message."
}
},
"output": "mockData"
},
{
"block": "ValidateContent",
"input": {
"from": "mockData.message",
"as": "text"
},
"config": {
"contains": ["Hello", "test"]
},
"output": "validation"
}
],
"assertions": {
"validation.passed": true
}
}]
}
```
### Testing LLM Judge
```json theme={null}
{
"pipeline": [
{
"block": "MockData",
"config": {
"data": {
"response": "The meeting is scheduled for 2:00 PM tomorrow."
}
},
"output": "aiResponse"
},
{
"block": "LLMJudge",
"input": {
"text": "${aiResponse.response}",
"expected": {
"expectedBehavior": "Should mention a time and date"
}
},
"output": "judgement"
}
],
"assertions": {
"judgement.score": { "gt": 0.7 }
}
}
```
### Mock Multiple Blocks
```json theme={null}
{
"pipeline": [
{
"id": "mock-user",
"block": "MockData",
"config": {
"data": {
"id": 1,
"name": "Alice"
}
},
"output": "user"
},
{
"id": "mock-posts",
"block": "MockData",
"config": {
"data": {
"posts": [
{ "id": 101, "userId": 1, "title": "First Post" },
{ "id": 102, "userId": 1, "title": "Second Post" }
]
}
},
"output": "posts"
}
],
"assertions": {
"user.id": 1,
"posts.posts[0].userId": 1
}
}
```
### Progressive Enhancement
Start with mock data, then replace with real requests:
```json theme={null}
{
"tests": [
{
"id": "test-with-mock",
"pipeline": [
{
"block": "MockData",
"config": {
"data": { "id": 1, "name": "Test User" }
},
"output": "user"
}
],
"assertions": {
"user.name": { "matches": ".+" }
}
},
{
"id": "test-with-real-api",
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "${BASE_URL}/users/1",
"method": "GET"
},
"output": "response"
},
{
"block": "JsonParser",
"input": "${response.body}",
"output": { "parsed": "user" }
}
],
"assertions": {
"user.name": { "matches": ".+" }
}
}
]
}
```
## Full Example
```json theme={null}
{
"name": "Mock Data Example",
"tests": [{
"id": "test-user-validation",
"pipeline": [
{
"id": "create-mock-user",
"block": "MockData",
"config": {
"data": {
"id": 123,
"name": "John Doe",
"email": "john@example.com",
"age": 30,
"roles": ["admin", "user"],
"profile": {
"bio": "Software engineer with 10 years experience",
"website": "https://johndoe.com"
}
}
},
"output": "user"
},
{
"id": "validate-email",
"block": "ValidateContent",
"input": {
"from": "user.email",
"as": "text"
},
"config": {
"matches": "^[\\w.-]+@[\\w.-]+\\.\\w+$"
},
"output": "emailValidation"
},
{
"id": "validate-bio",
"block": "ValidateContent",
"input": {
"from": "user.profile.bio",
"as": "text"
},
"config": {
"contains": ["engineer"]
},
"output": "bioValidation"
}
],
"assertions": {
"user.id": 123,
"user.age": { "gt": 18 },
"user.roles": { "contains": "admin" },
"emailValidation.passed": true,
"bioValidation.passed": true
}
}]
}
```
## Tips
Mock data is perfect for testing validation logic without external dependencies:
```json theme={null}
{
"pipeline": [
{
"block": "MockData",
"config": { "data": { "text": "test data" } },
"output": "mock"
},
{
"block": "ValidateContent",
"input": { "from": "mock.text", "as": "text" },
"config": { "contains": "test" },
"output": "validation"
}
]
}
```
Structure mock data to match real API responses:
```json theme={null}
{
"config": {
"data": {
"status": 200,
"body": '{"result": "success"}',
"headers": { "content-type": "application/json" }
}
}
}
```
Create mocks for edge cases and error scenarios:
```json theme={null}
{
"tests": [
{
"id": "test-empty-data",
"pipeline": [{
"block": "MockData",
"config": { "data": { "items": [] } },
"output": "empty"
}]
},
{
"id": "test-null-fields",
"pipeline": [{
"block": "MockData",
"config": { "data": { "name": null } },
"output": "nullData"
}]
}
]
}
```
Mix mock and real blocks to test specific parts of your pipeline:
```json theme={null}
{
"pipeline": [
{
"block": "MockData",
"config": { "data": { "apiKey": "test-key" } },
"output": "config"
},
{
"block": "HttpRequest",
"input": {
"url": "${API_URL}/data",
"method": "GET",
"headers": {
"Authorization": "Bearer ${config.apiKey}"
}
},
"output": "response"
}
]
}
```
## When to Use
**Use MockData when:**
* Testing validation logic without external APIs
* Creating reproducible test fixtures
* Developing tests before APIs are available
* Testing edge cases and error scenarios
* Running tests without network access
**Don't use when:**
* Testing actual API integration (use HttpRequest)
* Verifying real system behavior
* Load testing or performance testing
## Next Steps
Validate mock data
Use AI to validate mock responses
# Blocks Overview
Source: https://docs.semantictest.dev/blocks/overview
All available building blocks for your test pipelines
## What are Blocks?
Blocks are the building pieces of test pipelines. Each block does one thing well and can be combined with other blocks to create complex test scenarios.
## Available Blocks
Make HTTP requests to APIs with full control over method, headers, and body
Parse JSON strings into structured data
Parse streaming SSE responses from OpenAI, Vercel AI SDK, and custom formats
Validate text content with contains, regex, length checks, and more
Validate AI tool calls, order, arguments, and constraints
Use AI to evaluate responses semantically with configurable criteria
Loop back to previous blocks with max iteration control
Return mock data for testing without external dependencies
## Block Categories
### HTTP & Data Fetching
* **HttpRequest**: Make API calls
* **MockData**: Test without external dependencies
### Parsing & Transformation
* **JsonParser**: Parse JSON responses
* **StreamParser**: Handle streaming data
### Validation
* **ValidateContent**: Text and content validation
* **ValidateTools**: AI tool call validation
* **LLMJudge**: Semantic validation with AI
### Control Flow
* **Loop**: Retry logic and iterations
## Common Patterns
### API Testing
```json theme={null}
{
"pipeline": [
{ "block": "HttpRequest" }, // Call API
{ "block": "JsonParser" }, // Parse response
{ "block": "ValidateContent" } // Validate content
]
}
```
### AI Agent Testing
```json theme={null}
{
"pipeline": [
{ "block": "HttpRequest" }, // Call AI API
{ "block": "StreamParser" }, // Parse streaming response
{ "block": "ValidateTools" }, // Check tool calls
{ "block": "LLMJudge" } // Semantic validation
]
}
```
### Retry Logic
```json theme={null}
{
"pipeline": [
{ "id": "request", "block": "HttpRequest" },
{ "block": "ValidateContent" },
{
"block": "Loop",
"config": {
"target": "request",
"maxIterations": 3
}
}
]
}
```
## Creating Custom Blocks
Don't see what you need? Create custom blocks:
```javascript theme={null}
import { Block } from '@blade47/semantic-test';
export class MyCustomBlock extends Block {
static get inputs() {
return {
required: ['data'],
optional: ['config']
};
}
static get outputs() {
return {
produces: ['result']
};
}
async process(inputs, context) {
// Your logic here
return { result: processedData };
}
}
```
Complete guide to creating and registering custom blocks
## Block Composition
Blocks are designed to be composable. Here are common combinations:
```json theme={null}
HttpRequest → JsonParser → ValidateContent → Assertions
```
```json theme={null}
HttpRequest → StreamParser → ValidateTools → LLMJudge → Assertions
```
```json theme={null}
HttpRequest → ValidateContent → Loop (if failed) → Assertions
```
```json theme={null}
MockData → ValidateContent → LLMJudge → Assertions
```
## Next Steps
Click on any block above to see detailed documentation
See blocks in action with real examples
Extend SemanticTest with your own blocks
# StreamParser
Source: https://docs.semantictest.dev/blocks/stream-parser
Parse Server-Sent Events (SSE) streaming responses from AI APIs
## Overview
The **StreamParser** block parses streaming responses (Server-Sent Events) from AI APIs. It extracts text content and tool calls from streaming formats used by OpenAI, Vercel AI SDK, and other providers.
```json theme={null}
{
"block": "StreamParser",
"input": "${response.body}",
"config": {
"format": "sse-openai"
},
"output": {
"text": "aiMessage",
"toolCalls": "aiTools"
}
}
```
## Input Parameters
### Required
| Parameter | Type | Description |
| --------- | ------ | ---------------------------- |
| `body` | string | Raw SSE stream response body |
### Optional
| Parameter | Type | Default | Description |
| --------- | ------ | -------- | ------------------------------------------- |
| `format` | string | `'text'` | Stream format (see supported formats below) |
## Configuration
| Parameter | Type | Default | Description |
| --------- | ------ | -------- | ------------------------- |
| `format` | string | `'text'` | Streaming format to parse |
## Supported Formats
| Format | Description |
| ------------ | ------------------------------------- |
| `sse` | Generic Server-Sent Events |
| `sse-openai` | OpenAI streaming format (ChatGPT API) |
| `sse-vercel` | Vercel AI SDK streaming format |
| `text` | Plain text (no parsing) |
## Output Fields
| Field | Type | Description |
| ----------- | ------ | ---------------------------------------- |
| `text` | string | Combined text content from all chunks |
| `toolCalls` | array | Tool/function calls made by the AI |
| `chunks` | array | Individual stream chunks (for debugging) |
| `metadata` | object | Parse metadata (format, counts, errors) |
### Tool Call Format
```javascript theme={null}
{
id: "call_123",
name: "search_database",
args: { query: "users", limit: 10 }
}
```
## Examples
### Parse OpenAI Streaming Response
```json theme={null}
{
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "https://api.openai.com/v1/chat/completions",
"method": "POST",
"headers": {
"Authorization": "Bearer ${env.OPENAI_API_KEY}"
},
"body": {
"model": "gpt-4",
"messages": [{ "role": "user", "content": "Hello!" }],
"stream": true
}
},
"output": "response"
},
{
"block": "StreamParser",
"input": "${response.body}",
"config": {
"format": "sse-openai"
},
"output": {
"text": "aiMessage",
"toolCalls": "aiTools",
"metadata": "streamMeta"
}
}
],
"assertions": {
"aiMessage": { "matches": ".+" }
}
}
```
### Parse Vercel AI SDK Stream
```json theme={null}
{
"block": "StreamParser",
"input": "${response.body}",
"config": {
"format": "sse-vercel"
},
"output": {
"text": "aiText",
"toolCalls": "tools"
}
}
```
### Extract Tool Calls
```json theme={null}
{
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "${AI_API_URL}/chat",
"method": "POST",
"body": {
"message": "Search for users named John",
"tools": [
{ "name": "search_database" }
]
}
},
"output": "response"
},
{
"block": "StreamParser",
"input": "${response.body}",
"config": {
"format": "sse-openai"
},
"output": {
"text": "message",
"toolCalls": "tools"
}
},
{
"block": "ValidateTools",
"input": {
"from": "tools",
"as": "toolCalls"
},
"config": {
"expected": ["search_database"]
},
"output": "toolValidation"
}
],
"assertions": {
"toolValidation.passed": true
}
}
```
### Generic SSE Stream
```json theme={null}
{
"block": "StreamParser",
"input": "${response.body}",
"config": {
"format": "sse"
},
"output": {
"text": "content",
"chunks": "allChunks"
}
}
```
## Common Patterns
### OpenAI Chat with Function Calling
```json theme={null}
{
"name": "OpenAI Function Test",
"context": {
"OPENAI_URL": "https://api.openai.com/v1",
"API_KEY": "${env.OPENAI_API_KEY}"
},
"tests": [{
"id": "test-function-calling",
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "${OPENAI_URL}/chat/completions",
"method": "POST",
"headers": {
"Authorization": "Bearer ${API_KEY}",
"Content-Type": "application/json"
},
"body": {
"model": "gpt-4-turbo",
"messages": [{
"role": "user",
"content": "Search for users with the name Alice"
}],
"tools": [{
"type": "function",
"function": {
"name": "search_users",
"description": "Search for users",
"parameters": {
"type": "object",
"properties": {
"query": { "type": "string" }
}
}
}
}],
"stream": true
}
},
"output": "response"
},
{
"block": "StreamParser",
"input": "${response.body}",
"config": {
"format": "sse-openai"
},
"output": {
"text": "message",
"toolCalls": "tools",
"metadata": "meta"
}
},
{
"block": "ValidateTools",
"input": {
"from": "tools",
"as": "toolCalls"
},
"config": {
"expected": ["search_users"]
},
"output": "validation"
}
],
"assertions": {
"response.status": 200,
"validation.passed": true,
"tools[0].name": "search_users"
}
}]
}
```
### Vercel AI SDK Stream
```json theme={null}
{
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "${APP_URL}/api/chat",
"method": "POST",
"body": {
"messages": [{
"role": "user",
"content": "What's the weather?"
}]
}
},
"output": "response"
},
{
"block": "StreamParser",
"input": "${response.body}",
"config": {
"format": "sse-vercel"
},
"output": {
"text": "aiMessage",
"toolCalls": "tools",
"metadata": "streamInfo"
}
}
],
"assertions": {
"aiMessage": { "contains": "weather" },
"tools": { "contains": "get_weather" }
}
}
```
### Multi-Turn Conversation
```json theme={null}
{
"pipeline": [
{
"id": "first-message",
"block": "HttpRequest",
"input": {
"url": "${AI_URL}/chat",
"method": "POST",
"body": {
"messages": [{
"role": "user",
"content": "Hello!"
}]
}
},
"output": "response1"
},
{
"id": "parse1",
"block": "StreamParser",
"input": "${response1.body}",
"config": { "format": "sse-openai" },
"output": { "text": "message1" }
},
{
"id": "second-message",
"block": "HttpRequest",
"input": {
"url": "${AI_URL}/chat",
"method": "POST",
"body": {
"messages": [
{ "role": "user", "content": "Hello!" },
{ "role": "assistant", "content": "${message1}" },
{ "role": "user", "content": "Tell me a joke" }
]
}
},
"output": "response2"
},
{
"id": "parse2",
"block": "StreamParser",
"input": "${response2.body}",
"config": { "format": "sse-openai" },
"output": { "text": "message2" }
}
],
"assertions": {
"message1": { "matches": ".+" },
"message2": { "matches": ".+" }
}
}
```
## Stream Metadata
The `metadata` output includes useful information:
```javascript theme={null}
{
format: "sse-openai",
totalChunks: 15,
totalTools: 2,
toolErrorCount: 0 // Vercel format only
}
```
Use for debugging or assertions:
```json theme={null}
{
"assertions": {
"streamMeta.totalChunks": { "gt": 0 },
"streamMeta.totalTools": { "gt": 0 }
}
}
```
## Chunks Array
The `chunks` array contains individual stream events:
```javascript theme={null}
[
{ type: "text", content: "Hello" },
{ type: "text", content: " there" },
{ type: "tool-call", toolCallId: "123", toolName: "search" },
{ type: "finish", reason: "stop" }
]
```
Useful for debugging stream parsing issues.
## Error Handling
If parsing fails, StreamParser returns empty data:
```javascript theme={null}
{
text: "",
toolCalls: [],
chunks: [],
metadata: { format: "sse-openai", error: "Parse error message" }
}
```
Check for errors:
```json theme={null}
{
"assertions": {
"streamMeta.error": null
}
}
```
## Custom Parsers
You can register custom parsers programmatically:
```javascript theme={null}
import { StreamParser } from '@blade47/semantic-test';
function parseCustomFormat(body) {
// Parse your custom format
return {
text: "parsed text",
toolCalls: [],
chunks: [],
metadata: { format: "custom" }
};
}
StreamParser.register('custom', parseCustomFormat);
```
Then use in tests:
```json theme={null}
{
"config": {
"format": "custom"
}
}
```
## Full Example
```json theme={null}
{
"name": "AI Agent Stream Test",
"context": {
"AI_URL": "${env.AI_API_URL}",
"API_KEY": "${env.API_KEY}"
},
"tests": [{
"id": "test-streaming-agent",
"pipeline": [
{
"id": "call-agent",
"block": "HttpRequest",
"input": {
"url": "${AI_URL}/chat",
"method": "POST",
"headers": {
"Authorization": "Bearer ${API_KEY}"
},
"body": {
"messages": [{
"role": "user",
"content": "Search for users named Alice and send them an email"
}],
"tools": [
{ "name": "search_users" },
{ "name": "send_email" }
],
"stream": true
}
},
"output": "response"
},
{
"id": "parse",
"block": "StreamParser",
"input": "${response.body}",
"config": {
"format": "sse-openai"
},
"output": {
"text": "aiMessage",
"toolCalls": "tools",
"chunks": "streamChunks",
"metadata": "meta"
}
},
{
"id": "validate-message",
"block": "ValidateContent",
"input": {
"from": "aiMessage",
"as": "text"
},
"config": {
"contains": ["Alice", "email"]
},
"output": "contentCheck"
},
{
"id": "validate-tools",
"block": "ValidateTools",
"input": {
"from": "tools",
"as": "toolCalls"
},
"config": {
"expected": ["search_users", "send_email"],
"order": ["search_users", "send_email"]
},
"output": "toolCheck"
}
],
"assertions": {
"response.status": 200,
"aiMessage": { "matches": ".+" },
"contentCheck.passed": true,
"toolCheck.passed": true,
"meta.totalTools": 2
}
}]
}
```
## Tips
Map outputs to descriptive names:
```json theme={null}
{
"output": {
"text": "aiMessage",
"toolCalls": "tools"
}
}
```
Then access as `${aiMessage}` and `${tools}`.
* `sse-openai` - For OpenAI, Azure OpenAI
* `sse-vercel` - For Vercel AI SDK apps
* `sse` - For generic SSE streams
Combine with ValidateTools:
```json theme={null}
{
"pipeline": [
{ "block": "HttpRequest", "output": "response" },
{
"block": "StreamParser",
"input": "${response.body}",
"output": { "toolCalls": "tools" }
},
{
"block": "ValidateTools",
"input": { "from": "tools", "as": "toolCalls" },
"output": "validation"
}
]
}
```
Output chunks for debugging stream issues:
```json theme={null}
{
"output": {
"text": "message",
"chunks": "debugChunks"
}
}
```
Then inspect `${debugChunks}` to see individual events.
## When to Use
**Use StreamParser when:**
* Parsing streaming responses from AI APIs
* Extracting tool calls from function-calling models
* Testing real-time AI responses
* Working with Server-Sent Events (SSE)
**Don't use when:**
* Response is regular JSON (use JsonParser)
* Non-streaming HTTP response
* Testing non-AI APIs
## Next Steps
Validate extracted tool calls
Validate AI message quality
# ValidateContent
Source: https://docs.semantictest.dev/blocks/validate-content
Validate text content against rules and patterns
## Overview
The **ValidateContent** block validates text content against various rules like required substrings, forbidden words, length constraints, and regex patterns.
```json theme={null}
{
"block": "ValidateContent",
"input": {
"from": "response.message",
"as": "text"
},
"config": {
"contains": ["hello", "world"]
},
"output": "validation"
}
```
## Input Parameters
### Required
| Parameter | Type | Description |
| --------- | ------ | ---------------------------- |
| `text` | string | The text content to validate |
**Note:** Use the `from`/`as` input format to map your data to the `text` parameter.
## Configuration
All config parameters are optional. Use the ones relevant to your validation:
| Parameter | Type | Description |
| ------------- | ------------ | ------------------------------------------------- |
| `contains` | string/array | Required substring(s) - case insensitive |
| `notContains` | string/array | Forbidden substring(s) - case insensitive |
| `minLength` | number | Minimum text length |
| `maxLength` | number | Maximum text length |
| `matches` | string | Regex pattern the text must match |
| `matchFlags` | string | Regex flags (default: `'i'` for case-insensitive) |
## Output Fields
| Field | Type | Description |
| ---------- | ------- | ------------------------------ |
| `passed` | boolean | Whether all validations passed |
| `failures` | array | List of validation failures |
| `score` | number | Validation score (0.0 - 1.0) |
| `checks` | object | Individual check results |
## Examples
### Validate Required Keywords
```json theme={null}
{
"block": "ValidateContent",
"input": {
"from": "aiResponse.message",
"as": "text"
},
"config": {
"contains": ["appointment", "time"]
},
"output": "validation"
}
```
**Success:**
```javascript theme={null}
{
passed: true,
failures: [],
score: 1.0,
checks: {
contains_appointment: true,
contains_time: true
}
}
```
**Failure:**
```javascript theme={null}
{
passed: false,
failures: ["Missing expected content: time"],
score: 0.5,
checks: {
contains_appointment: true,
contains_time: false
}
}
```
### Forbid Inappropriate Content
```json theme={null}
{
"block": "ValidateContent",
"input": {
"from": "userInput.comment",
"as": "text"
},
"config": {
"notContains": ["spam", "offensive", "inappropriate"]
},
"output": "validation"
}
```
### Validate Text Length
```json theme={null}
{
"block": "ValidateContent",
"input": {
"from": "user.bio",
"as": "text"
},
"config": {
"minLength": 10,
"maxLength": 500
},
"output": "validation"
}
```
### Regex Pattern Matching
```json theme={null}
{
"block": "ValidateContent",
"input": {
"from": "user.email",
"as": "text"
},
"config": {
"matches": "^[\\w.-]+@[\\w.-]+\\.\\w+$"
},
"output": "emailValidation"
}
```
### Combined Validations
```json theme={null}
{
"block": "ValidateContent",
"input": {
"from": "aiResponse.message",
"as": "text"
},
"config": {
"contains": ["meeting", "time"],
"notContains": ["error", "failed"],
"minLength": 20,
"maxLength": 500,
"matches": "\\d{1,2}:\\d{2}"
},
"output": "validation"
}
```
## Common Patterns
### Validate AI Response Quality
```json theme={null}
{
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "${AI_API_URL}/chat",
"method": "POST",
"body": {
"message": "What time is the meeting?"
}
},
"output": "response"
},
{
"block": "JsonParser",
"input": "${response.body}",
"output": { "parsed": "aiResponse" }
},
{
"block": "ValidateContent",
"input": {
"from": "aiResponse.message",
"as": "text"
},
"config": {
"contains": ["meeting", "time"],
"minLength": 10
},
"output": "validation"
}
],
"assertions": {
"validation.passed": true
}
}
```
### Multi-Field Validation
```json theme={null}
{
"pipeline": [
{
"block": "MockData",
"config": {
"data": {
"name": "John Doe",
"email": "john@example.com",
"bio": "Software engineer"
}
},
"output": "user"
},
{
"id": "validate-name",
"block": "ValidateContent",
"input": {
"from": "user.name",
"as": "text"
},
"config": {
"minLength": 2,
"maxLength": 100
},
"output": "nameValidation"
},
{
"id": "validate-email",
"block": "ValidateContent",
"input": {
"from": "user.email",
"as": "text"
},
"config": {
"matches": ".*@.*\\..*"
},
"output": "emailValidation"
},
{
"id": "validate-bio",
"block": "ValidateContent",
"input": {
"from": "user.bio",
"as": "text"
},
"config": {
"minLength": 10
},
"output": "bioValidation"
}
],
"assertions": {
"nameValidation.passed": true,
"emailValidation.passed": true,
"bioValidation.passed": true
}
}
```
### Content Safety Check
```json theme={null}
{
"block": "ValidateContent",
"input": {
"from": "userComment.text",
"as": "text"
},
"config": {
"notContains": [
"spam",
"offensive",
"inappropriate",
"abuse"
],
"minLength": 5,
"maxLength": 10000
},
"output": "safetyCheck"
}
```
## Validation Score
The `score` field represents how well the content passed validation:
* **1.0** - All validations passed
* **0.5** - Some validations failed
* **\< 0.5** - Multiple validations failed
The score is calculated based on:
* Missing required content
* Forbidden content found
* Length violations
* Pattern mismatches
Use the score for assertions:
```json theme={null}
{
"assertions": {
"validation.score": { "gt": 0.8 }
}
}
```
## Error Messages
Common failure messages:
| Message | Cause |
| ----------------------------------- | ---------------------------- |
| `Missing expected content: word` | Required substring not found |
| `Found forbidden content: word` | Forbidden substring found |
| `Text too short: 5 < 10` | Below minLength |
| `Text too long: 600 > 500` | Above maxLength |
| `Text doesn't match pattern: regex` | Regex pattern didn't match |
## Full Example
```json theme={null}
{
"name": "Content Validation Test",
"context": {
"AI_API_URL": "${env.AI_API_URL}"
},
"tests": [{
"id": "validate-ai-response",
"pipeline": [
{
"id": "call-ai",
"block": "HttpRequest",
"input": {
"url": "${AI_API_URL}/chat",
"method": "POST",
"headers": {
"Authorization": "Bearer ${env.API_KEY}"
},
"body": {
"messages": [{
"role": "user",
"content": "Schedule a meeting for tomorrow at 2pm"
}]
}
},
"output": "response"
},
{
"id": "parse",
"block": "JsonParser",
"input": "${response.body}",
"output": {
"parsed": "aiResponse"
}
},
{
"id": "validate-response",
"block": "ValidateContent",
"input": {
"from": "aiResponse.message",
"as": "text"
},
"config": {
"contains": ["meeting", "tomorrow", "2pm"],
"notContains": ["error", "sorry", "cannot"],
"minLength": 20,
"maxLength": 500
},
"output": "validation"
}
],
"assertions": {
"response.status": 200,
"validation.passed": true,
"validation.score": { "gt": 0.9 }
}
}]
}
```
## Tips
ValidateContent expects a `text` parameter. Use the from/as format:
```json theme={null}
{
"input": {
"from": "response.message",
"as": "text"
}
}
```
The `contains` and `notContains` checks are case-insensitive:
```json theme={null}
{
"config": {
"contains": "hello"
}
}
// Matches: "Hello", "HELLO", "hello"
```
Use ValidateContent for structural checks, LLMJudge for semantic validation:
```json theme={null}
{
"pipeline": [
{
"block": "ValidateContent",
"input": { "from": "response.text", "as": "text" },
"config": { "minLength": 10, "contains": "meeting" },
"output": "structureCheck"
},
{
"block": "LLMJudge",
"input": {
"text": "${response.text}",
"expected": { "expectedBehavior": "Professional tone" }
},
"output": "semanticCheck"
}
]
}
```
Regex patterns are powerful for complex validation:
```json theme={null}
{
"config": {
"matches": "^Meeting at \\d{1,2}(am|pm)$"
}
}
```
## When to Use
**Use ValidateContent when:**
* Checking for required keywords
* Filtering inappropriate content
* Validating text length
* Pattern matching (emails, URLs, etc.)
* Structural content validation
**Use LLMJudge instead when:**
* Checking semantic meaning
* Evaluating tone or style
* Comparing similarity to expected content
* Complex reasoning about content
## Next Steps
Semantic content validation
Validate AI tool calls
# ValidateTools
Source: https://docs.semantictest.dev/blocks/validate-tools
Validate AI tool/function calls
## Overview
The **ValidateTools** block validates that AI models call the correct tools/functions with the right arguments. Essential for testing AI agents and function-calling models.
```json theme={null}
{
"block": "ValidateTools",
"input": {
"from": "aiResponse.toolCalls",
"as": "toolCalls"
},
"config": {
"expected": ["search_database", "send_email"]
},
"output": "toolValidation"
}
```
## Input Parameters
### Required
| Parameter | Type | Description |
| ----------- | ----- | -------------------------- |
| `toolCalls` | array | Array of tool call objects |
Each tool call object should have:
```javascript theme={null}
{
name: "tool_name", // or toolName
args: { /* ... */ } // optional arguments
}
```
## Configuration
All config parameters are optional:
| Parameter | Type | Description |
| -------------- | ------------ | --------------------------------------------- |
| `expected` | string/array | Required tool names |
| `forbidden` | string/array | Tools that should NOT be called |
| `minTools` | number | Minimum number of tools that should be called |
| `maxTools` | number | Maximum number of tools allowed |
| `order` | array | Expected order of tool calls |
| `validateArgs` | object | Expected arguments for each tool |
## Output Fields
| Field | Type | Description |
| --------------- | ------- | ------------------------------------ |
| `passed` | boolean | Whether all validations passed |
| `failures` | array | List of validation failures |
| `score` | number | Validation score (0.0 - 1.0) |
| `actualTools` | array | Tool names that were actually called |
| `expectedTools` | array | Tool names that were expected |
## Examples
### Validate Expected Tools
```json theme={null}
{
"block": "ValidateTools",
"input": {
"from": "aiResponse.toolCalls",
"as": "toolCalls"
},
"config": {
"expected": ["search_database", "send_email"]
},
"output": "validation"
}
```
**Success:**
```javascript theme={null}
{
passed: true,
failures: [],
score: 1.0,
actualTools: ["search_database", "send_email"],
expectedTools: ["search_database", "send_email"]
}
```
**Failure:**
```javascript theme={null}
{
passed: false,
failures: ["Missing expected tools: send_email"],
score: 0.5,
actualTools: ["search_database"],
expectedTools: ["search_database", "send_email"]
}
```
### Forbid Dangerous Tools
```json theme={null}
{
"block": "ValidateTools",
"input": {
"from": "aiResponse.toolCalls",
"as": "toolCalls"
},
"config": {
"forbidden": ["delete_database", "drop_table"]
},
"output": "validation"
}
```
### Validate Tool Count
```json theme={null}
{
"block": "ValidateTools",
"input": {
"from": "aiResponse.toolCalls",
"as": "toolCalls"
},
"config": {
"minTools": 1,
"maxTools": 3
},
"output": "validation"
}
```
### Validate Tool Order
```json theme={null}
{
"block": "ValidateTools",
"input": {
"from": "aiResponse.toolCalls",
"as": "toolCalls"
},
"config": {
"order": ["get_user", "update_profile", "send_notification"]
},
"output": "validation"
}
```
### Validate Tool Arguments
```json theme={null}
{
"block": "ValidateTools",
"input": {
"from": "aiResponse.toolCalls",
"as": "toolCalls"
},
"config": {
"expected": ["search_database"],
"validateArgs": {
"search_database": {
"table": "users",
"limit": 10
}
}
},
"output": "validation"
}
```
### Combined Validations
```json theme={null}
{
"block": "ValidateTools",
"input": {
"from": "aiResponse.toolCalls",
"as": "toolCalls"
},
"config": {
"expected": ["search_users", "send_email"],
"forbidden": ["delete_user"],
"minTools": 2,
"maxTools": 3,
"validateArgs": {
"search_users": {
"limit": 10
}
}
},
"output": "validation"
}
```
## Common Patterns
### Test AI Agent Tool Usage
```json theme={null}
{
"name": "AI Agent Tool Test",
"tests": [{
"id": "test-calendar-agent",
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "${AI_API_URL}/chat",
"method": "POST",
"body": {
"messages": [{
"role": "user",
"content": "Schedule a meeting for tomorrow at 2pm"
}],
"tools": [
{ "name": "search_calendar" },
{ "name": "create_event" }
]
}
},
"output": "response"
},
{
"block": "StreamParser",
"input": "${response.body}",
"config": { "format": "sse-openai" },
"output": {
"toolCalls": "toolCalls"
}
},
{
"block": "ValidateTools",
"input": {
"from": "toolCalls",
"as": "toolCalls"
},
"config": {
"expected": ["search_calendar", "create_event"],
"order": ["search_calendar", "create_event"]
},
"output": "toolValidation"
}
],
"assertions": {
"toolValidation.passed": true
}
}]
}
```
### Ensure Safe Tool Usage
```json theme={null}
{
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "${AI_URL}/chat",
"method": "POST",
"body": {
"message": "Delete my account"
}
},
"output": "response"
},
{
"block": "StreamParser",
"input": "${response.body}",
"output": { "toolCalls": "tools" }
},
{
"block": "ValidateTools",
"input": {
"from": "tools",
"as": "toolCalls"
},
"config": {
"forbidden": ["delete_account", "drop_database"]
},
"output": "safety"
}
],
"assertions": {
"safety.passed": true
}
}
```
### Validate Multi-Step Agent
```json theme={null}
{
"block": "ValidateTools",
"input": {
"from": "aiResponse.toolCalls",
"as": "toolCalls"
},
"config": {
"order": [
"authenticate_user",
"fetch_data",
"process_data",
"send_response"
]
},
"output": "validation"
}
```
## Tool Call Format
The block expects tool calls in this format:
```javascript theme={null}
[
{
name: "search_database", // or toolName
args: {
query: "users",
limit: 10
}
},
{
name: "send_email",
args: {
to: "user@example.com",
subject: "Hello"
}
}
]
```
**Compatible with:**
* OpenAI function calling format
* Anthropic tool use format
* Vercel AI SDK tool calls
StreamParser automatically extracts tool calls into this format from streaming responses.
## Validation Score
The `score` field represents validation quality:
* **1.0** - All validations passed
* **0.5 - 0.9** - Some issues (missing tools, wrong order)
* **\< 0.5** - Significant issues (forbidden tools used, major mismatches)
Factors that reduce score:
* Missing expected tools
* Using forbidden tools
* Wrong tool count
* Incorrect order
* Argument mismatches
## Error Messages
Common failure messages:
| Message | Cause |
| -------------------------------------- | ------------------------------------------ |
| `Missing expected tools: tool1, tool2` | Expected tools not called |
| `Used forbidden tools: tool1` | Forbidden tool was called |
| `Too few tools: 1 < 2` | Below minTools |
| `Too many tools: 5 > 3` | Above maxTools |
| `Tool order incorrect` | Tools called in wrong order |
| `Tool 'search' arg 'limit' mismatch` | Tool argument doesn't match expected value |
## Full Example
```json theme={null}
{
"name": "Customer Service Agent Test",
"context": {
"AI_URL": "${env.AI_API_URL}",
"API_KEY": "${env.API_KEY}"
},
"tests": [{
"id": "test-customer-query",
"pipeline": [
{
"id": "call-agent",
"block": "HttpRequest",
"input": {
"url": "${AI_URL}/chat",
"method": "POST",
"headers": {
"Authorization": "Bearer ${API_KEY}"
},
"body": {
"messages": [{
"role": "user",
"content": "Look up my order #12345 and send me an update"
}],
"tools": [
{ "name": "search_orders", "description": "Search orders" },
{ "name": "send_email", "description": "Send email" }
]
}
},
"output": "response"
},
{
"id": "parse-stream",
"block": "StreamParser",
"input": "${response.body}",
"config": {
"format": "sse-openai"
},
"output": {
"text": "aiMessage",
"toolCalls": "toolCalls"
}
},
{
"id": "validate-tools",
"block": "ValidateTools",
"input": {
"from": "toolCalls",
"as": "toolCalls"
},
"config": {
"expected": ["search_orders", "send_email"],
"order": ["search_orders", "send_email"],
"minTools": 2,
"maxTools": 2,
"validateArgs": {
"search_orders": {
"order_id": "12345"
}
}
},
"output": "toolValidation"
}
],
"assertions": {
"response.status": 200,
"toolValidation.passed": true,
"toolValidation.score": { "gt": 0.9 }
}
}]
}
```
## Tips
StreamParser extracts tool calls from streaming responses:
```json theme={null}
{
"pipeline": [
{
"block": "HttpRequest",
"output": "response"
},
{
"block": "StreamParser",
"input": "${response.body}",
"output": { "toolCalls": "tools" }
},
{
"block": "ValidateTools",
"input": { "from": "tools", "as": "toolCalls" },
"config": { "expected": ["search"] },
"output": "validation"
}
]
}
```
Use `order` to validate sequential tool usage:
```json theme={null}
{
"config": {
"order": ["auth", "fetch", "process", "respond"]
}
}
```
ValidateTools checks structure, LLMJudge checks reasoning:
```json theme={null}
{
"pipeline": [
{
"block": "ValidateTools",
"config": { "expected": ["search"] },
"output": "toolCheck"
},
{
"block": "LLMJudge",
"input": {
"text": "${aiMessage}",
"expected": {
"expectedBehavior": "Polite customer service"
}
},
"output": "qualityCheck"
}
],
"assertions": {
"toolCheck.passed": true,
"qualityCheck.score": { "gt": 0.7 }
}
}
```
Prevent dangerous operations:
```json theme={null}
{
"config": {
"forbidden": [
"delete_user",
"drop_table",
"execute_sql",
"send_money"
]
}
}
```
## When to Use
**Use ValidateTools when:**
* Testing AI function calling
* Validating agent tool usage
* Ensuring safe tool execution
* Testing multi-step workflows
* Verifying tool arguments
**Don't use when:**
* No tool/function calling involved
* Testing pure text generation
* Validating non-AI responses
## Next Steps
Parse streaming responses with tool calls
Validate AI response quality
# Assertions
Source: https://docs.semantictest.dev/concepts/assertions
Validate your test results with powerful operators
## Overview
**Assertions** are the final step in your test pipeline. After all blocks execute, assertions check if the data in the DataBus matches your expectations.
```json theme={null}
{
"pipeline": [
],
"assertions": {
"response.status": 200,
"user.id": 1,
"validation.passed": true
}
}
```
## Simple Assertions (Equality)
The simplest form checks for exact equality:
```json theme={null}
{
"assertions": {
"response.status": 200,
"user.name": "John Doe",
"data.count": 10,
"validation.passed": true
}
}
```
**Rules:**
* String values must match exactly
* Numbers must match exactly
* Booleans must match exactly
* Null values must match exactly
## Operator Assertions
For more complex validation, use operator objects with 20+ powerful operators.
### Equality Operators
```json theme={null}
{
"assertions": {
"response.status": { "equals": 200 },
"user.id": { "notEquals": 0 }
}
}
```
### Numeric Comparisons
```json theme={null}
{
"assertions": {
"user.age": { "gt": 18 },
"user.age": { "gte": 18 },
"response.time": { "lt": 1000 },
"data.count": { "lte": 100 }
}
}
```
* `gt`: Greater than
* `gte`: Greater than or equal
* `lt`: Less than
* `lte`: Less than or equal
### String/Array Operations
```json theme={null}
{
"assertions": {
"response.message": { "contains": "success" },
"user.roles": { "contains": "admin" },
"response.error": { "notContains": "fatal" },
"user.name": { "minLength": 1 },
"user.bio": { "maxLength": 500 }
}
}
```
**For text:** Checks if the string contains the substring or meets length requirements
**For arrays:** Checks if the array contains the value or meets length requirements
### Pattern Matching
```json theme={null}
{
"assertions": {
"user.email": { "matches": ".*@example\\.com" },
"data.phone": { "matches": "^\\d{3}-\\d{3}-\\d{4}$" },
"response.id": { "notMatches": "^temp-.*" }
}
}
```
Uses JavaScript regex syntax.
### Type Checks
```json theme={null}
{
"assertions": {
"data.value": { "isNull": true },
"user.name": { "isNotNull": true },
"config.setting": { "isDefined": true },
"optional.field": { "isUndefined": true }
}
}
```
### Boolean Checks
```json theme={null}
{
"assertions": {
"validation.passed": { "isTrue": true },
"hasError": { "isFalse": true }
}
}
```
### Empty Checks
```json theme={null}
{
"assertions": {
"user.name": { "isNotEmpty": true },
"errors": { "isEmpty": true }
}
}
```
Works with strings, arrays, and objects.
## Complete Example
```json theme={null}
{
"name": "User API Test",
"tests": [{
"id": "test-user",
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "https://api.example.com/users/1",
"method": "GET"
},
"output": "response"
},
{
"block": "JsonParser",
"input": "${response.body}",
"output": {
"parsed": "user"
}
},
{
"block": "LLMJudge",
"input": {
"text": "${user}",
"expected": {
"expectedBehavior": "Should have complete user profile"
}
},
"output": "judgement"
}
],
"assertions": {
"response.status": { "gte": 200, "lt": 300 },
"user.id": { "gt": 0 },
"user.email": { "matches": ".*@.*\\..*", "isNotEmpty": true },
"user.age": { "gte": 0, "lte": 150 },
"user.roles": { "contains": "user", "isNotEmpty": true },
"user.name": { "minLength": 1, "maxLength": 100 },
"judgement.score": { "gte": 0.7 },
"judgement.reasoning": { "isNotEmpty": true }
}
}]
}
```
## Multiple Operators
You can combine multiple operators on the same field - all must pass:
```json theme={null}
{
"assertions": {
"user.age": {
"gte": 18,
"lte": 100
},
"response.message": {
"contains": "success",
"matches": "^[A-Z].*",
"isNotEmpty": true
},
"user.email": {
"matches": ".*@.*\\..*",
"isNotEmpty": true,
"minLength": 5
}
}
}
```
**Example explanations:**
* `user.age`: Must be between 18 and 100
* `response.message`: Must contain "success" AND start with capital letter
* `user.email`: Must be valid email format AND not empty AND at least 5 characters
All operators in a group are evaluated with AND logic - every operator must pass for the assertion to succeed.
## Common Patterns
```json theme={null}
{
"assertions": {
"response.status": { "gte": 200, "lt": 300 },
"response.body": { "isNotEmpty": true }
}
}
```
Check for success status range (200-299) and non-empty response body.
```json theme={null}
{
"assertions": {
"judgement.score": { "gte": 0.7, "lte": 1.0 },
"judgement.reasoning": { "isNotEmpty": true }
}
}
```
Ensure quality score is between 0.7 and 1.0 and reasoning is provided.
```json theme={null}
{
"assertions": {
"user.email": {
"matches": ".*@.*\\..*",
"isNotEmpty": true
},
"user.id": { "gt": 0 },
"user.name": { "minLength": 1, "maxLength": 100 },
"user.bio": { "maxLength": 500 },
"user.createdAt": { "isDefined": true },
"user.nickname": { "maxLength": 50 }
}
}
```
Validate email format, required fields, and field length constraints.
```json theme={null}
{
"assertions": {
"toolValidation.passed": true,
"toolValidation.actualTools": { "contains": "search_users" }
}
}
```
Ensure tool validation passed and expected tools were called.
## Assertion Failures
When an assertion fails, you get detailed error messages:
```bash theme={null}
❌ test-user: User API Test (234ms)
✅ response.status = 200
❌ user.age: Expected 25 to be > 30
✅ user.email matches ".*@.*\\..*"
❌ judgement.score: Expected 0.6 to be > 0.7
```
## Best Practices
Test successful response:
```json theme={null}
{
"id": "success-case",
"pipeline": [],
"assertions": {
"response.status": 200,
"validation.passed": true
}
}
```
Test error handling:
```json theme={null}
{
"id": "error-case",
"pipeline": [],
"assertions": {
"response.status": 400,
"validation.passed": false
}
}
```
Bad - too specific:
```json theme={null}
{
"judgement.score": 0.85
}
```
Good - allows for variance:
```json theme={null}
{
"judgement.score": { "gt": 0.7 }
}
```
```json theme={null}
{
"assertions": {
"user.id": { "gt": 0 },
"user.email": { "isDefined": true, "isNotEmpty": true },
"user.name": { "isNotEmpty": true },
"user.email": { "matches": ".*@.*\\..*" },
"user.name": { "minLength": 1, "maxLength": 100 }
}
}
```
**Structure checks:** Required fields exist and are not empty
**Content checks:** Data values are valid (format, length, etc.)
Bad - too many assertions make tests fragile:
```json theme={null}
{
"user.name": "John Doe",
"user.age": 30,
"user.email": "john@example.com",
"user.address.street": "123 Main St",
"user.address.city": "Springfield"
}
```
Good - test what matters:
```json theme={null}
{
"user.id": 1,
"user.email": { "matches": ".*@.*\\..*" },
"response.status": 200
}
```
If any specific detail changes, overly specific tests will break unnecessarily.
## Operator Reference
### Equality Operators
| Operator | Description | Example |
| ----------- | ----------------------------- | ---------------------- |
| `equals` | Value equals expected | `{ "equals": 200 }` |
| `notEquals` | Value does not equal expected | `{ "notEquals": 404 }` |
### Numeric Operators
| Operator | Description | Example |
| -------- | ------------------------ | ---------------- |
| `gt` | Greater than | `{ "gt": 10 }` |
| `gte` | Greater than or equal to | `{ "gte": 18 }` |
| `lt` | Less than | `{ "lt": 100 }` |
| `lte` | Less than or equal to | `{ "lte": 200 }` |
### String/Array Operators
| Operator | Description | Example |
| ------------- | ----------------------------------- | ---------------------------- |
| `contains` | String/array contains value | `{ "contains": "text" }` |
| `notContains` | String/array does not contain value | `{ "notContains": "error" }` |
| `minLength` | String/array length >= value | `{ "minLength": 1 }` |
| `maxLength` | String/array length \<= value | `{ "maxLength": 100 }` |
### Pattern Operators
| Operator | Description | Example |
| ------------ | ---------------------------- | ------------------------------ |
| `matches` | String matches regex pattern | `{ "matches": "^\\d+$" }` |
| `notMatches` | String does not match regex | `{ "notMatches": "^temp-.*" }` |
### Type Operators
| Operator | Description | Example |
| ------------- | -------------------------------- | ------------------------- |
| `isNull` | Value is null | `{ "isNull": true }` |
| `isNotNull` | Value is not null | `{ "isNotNull": true }` |
| `isDefined` | Value is defined (not undefined) | `{ "isDefined": true }` |
| `isUndefined` | Value is undefined | `{ "isUndefined": true }` |
### Boolean Operators
| Operator | Description | Example |
| --------- | ---------------------- | --------------------- |
| `isTrue` | Value is boolean true | `{ "isTrue": true }` |
| `isFalse` | Value is boolean false | `{ "isFalse": true }` |
### Empty Operators
| Operator | Description | Example |
| ------------ | -------------------------------- | ------------------------ |
| `isEmpty` | String/array/object is empty | `{ "isEmpty": true }` |
| `isNotEmpty` | String/array/object is not empty | `{ "isNotEmpty": true }` |
## Next Steps
See what data each block produces
See assertions in real tests
# Data Flow
Source: https://docs.semantictest.dev/concepts/data-flow
How data moves through your test pipelines
## The DataBus
The **DataBus** is a key-value store that holds all data during test execution. Think of it as a shared memory where blocks can read and write data using **named slots**.
```
┌─────────────┐
│ DataBus │
├─────────────┤
│ response │ ← HttpRequest writes here
│ parsed │ ← JsonParser writes here
│ validation │ ← Validate writes here
└─────────────┘
```
## Writing to the DataBus
Blocks write their output to named slots using the `output` field:
```json theme={null}
{
"block": "HttpRequest",
"input": { "url": "https://api.example.com/users/1" },
"output": "response" // ← Writes to 'response' slot
}
```
After this block executes, the DataBus contains:
```javascript theme={null}
{
response: {
status: 200,
body: '{"id": 1, "name": "John"}',
headers: { /* ... */ }
}
}
```
## Reading from the DataBus
Use `${}` syntax to read from slots:
```json theme={null}
{
"block": "JsonParser",
"input": "${response.body}", // ← Reads from 'response.body'
"output": "parsed"
}
```
You can access nested properties using dot notation:
```json theme={null}
"input": "${response.headers.content-type}"
"input": "${parsed.data.user.email}"
"input": "${validation.passed}"
```
## Variable Resolution
The DataBus resolves variables when blocks execute:
```json theme={null}
{
"pipeline": [
{
"id": "fetch",
"block": "HttpRequest",
"input": { "url": "${BASE_URL}/users" }, // ← Resolves ${BASE_URL}
"output": "users"
},
{
"id": "process",
"block": "JsonParser",
"input": "${users.body}", // ← Resolves ${users.body}
"output": "parsed"
}
]
}
```
### Resolution Order
Variables are resolved from multiple sources in this priority:
1. **Environment** - From `.env` file (accessed via `${env.VARIABLE}`)
2. **Context** - Defined in the test file
3. **DataBus** - From previous blocks
```json theme={null}
{
"context": {
"BASE_URL": "${env.API_URL}", // From .env
"API_KEY": "hardcoded-key"
},
"tests": [{
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "${BASE_URL}/data", // From context
"headers": {
"Authorization": "${API_KEY}" // From context
}
},
"output": "response"
},
{
"block": "JsonParser",
"input": "${response.body}", // From DataBus
"output": "data"
}
]
}]
}
```
## Data Transformation
Data can be transformed as it flows:
```json theme={null}
{
"pipeline": [
{
"block": "HttpRequest",
"output": "raw" // Raw HTTP response
},
{
"block": "JsonParser",
"input": "${raw.body}",
"output": "parsed" // Parsed JSON
},
{
"block": "ValidateContent",
"input": {
"from": "parsed.message",
"as": "text"
},
"output": "validation" // Validation result
}
]
}
```
DataBus state after each block:
```javascript theme={null}
{
raw: {
status: 200,
body: '{"message": "Hello"}',
headers: {}
}
}
```
```javascript theme={null}
{
raw: { /* ... */ },
parsed: {
message: "Hello"
}
}
```
```javascript theme={null}
{
raw: { /* ... */ },
parsed: { /* ... */ },
validation: {
passed: true,
checks: { /* ... */ }
}
}
```
## Context Variables
Define reusable variables in `context`:
```json theme={null}
{
"context": {
"BASE_URL": "https://api.example.com",
"API_VERSION": "v1",
"ENDPOINT": "${BASE_URL}/${API_VERSION}/users"
},
"tests": [{
"pipeline": [{
"block": "HttpRequest",
"input": { "url": "${ENDPOINT}" }
}]
}]
}
```
## Environment Variables
Reference environment variables:
```bash .env theme={null}
API_URL=https://api.example.com
API_KEY=secret123
```
```json theme={null}
{
"context": {
"BASE_URL": "${env.API_URL}",
"API_KEY": "${env.API_KEY}"
}
}
```
## Debugging Data Flow
Enable debug logging to see data flow:
```bash theme={null}
export LOG_LEVEL=DEBUG
npx semtest test.json
```
You'll see:
```
[DEBUG] DataBus.set('response', {...})
[DEBUG] DataBus.get('response.body') → '{"id": 1}'
[DEBUG] Resolving ${response.body} → {"id": 1}
```
## Best Practices
```json theme={null}
// Good
"output": "userProfile"
"output": "authToken"
"output": "validationResult"
// Bad
"output": "data"
"output": "result"
"output": "temp"
```
```json theme={null}
// Simpler to reference
"output": "userId" // Use ${userId}
// vs
"output": "user" // Use ${user.data.profile.id}
```
```json theme={null}
{
"context": {
"AUTH_HEADER": "Bearer ${env.API_KEY}"
},
"tests": [{
"pipeline": [
{
"block": "HttpRequest",
"input": {
"headers": { "Authorization": "${AUTH_HEADER}" }
}
}
]
}]
}
```
## Next Steps
Learn different ways to pass data to blocks
Learn how blocks can output data
# Input Formats
Source: https://docs.semantictest.dev/concepts/input-formats
Three ways to pass data to blocks
## Overview
SemanticTest supports **three different input formats** for passing data to blocks. This flexibility lets you write cleaner, more readable tests.
## 1. String Format (Simple)
Pass a single value from the DataBus:
```json theme={null}
{
"block": "JsonParser",
"input": "${response.body}"
}
```
**What happens:**
* The value from `response.body` is wrapped in an object
* Becomes: `{ body: "the actual value" }`
* The block receives it as `inputs.body`
**When to use:**
* Single value input
* Quick and simple syntax
* Most common format
## 2. From/As Format (Mapping)
Map a DataBus slot to a specific parameter name:
```json theme={null}
{
"block": "ValidateContent",
"input": {
"from": "parsed.message",
"as": "text"
}
}
```
**What happens:**
* Value from `parsed.message` is mapped to parameter `text`
* The block receives it as `inputs.text`
**When to use:**
* Block expects a specific parameter name
* You need to rename the data
* More explicit about what data goes where
**Example:**
```json theme={null}
{
"pipeline": [
{
"block": "HttpRequest",
"input": { "url": "https://api.example.com/users" },
"output": "response"
},
{
"block": "JsonParser",
"input": "${response.body}",
"output": "parsed"
},
{
"block": "ValidateContent",
"input": {
"from": "parsed.users[0].email",
"as": "text"
},
"config": {
"matches": ".*@.*\\.com"
}
}
]
}
```
## 3. Object Format (Full Control)
Pass multiple values with deep variable resolution:
```json theme={null}
{
"block": "HttpRequest",
"input": {
"url": "${BASE_URL}/users",
"method": "POST",
"headers": {
"Authorization": "Bearer ${token}",
"Content-Type": "application/json"
},
"body": {
"name": "John Doe",
"email": "${userEmail}"
},
"timeout": 5000
}
}
```
**What happens:**
* All `${}` variables are resolved from the DataBus
* The entire object is passed to the block
* Nested objects are fully resolved
**When to use:**
* Multiple parameters needed
* Complex nested data structures
* Mix of static and dynamic values
## Comparison
```json theme={null}
// Simplest - single value
"input": "${response.body}"
// Block receives:
{
body: "actual response body content"
}
```
```json theme={null}
// Map to specific parameter
"input": {
"from": "response.body",
"as": "text"
}
// Block receives:
{
text: "actual response body content"
}
```
```json theme={null}
// Full control
"input": {
"url": "${API_URL}",
"method": "POST",
"data": "${payload}"
}
// Block receives:
{
url: "https://api.example.com",
method: "POST",
data: { name: "John" }
}
```
## Variable Resolution
All formats support `${}` syntax for reading from the DataBus:
```json theme={null}
// Simple property access
"${response.status}"
// Nested properties
"${user.profile.email}"
// Array access
"${users[0].name}"
// From context
"${BASE_URL}"
// From environment
"${env.API_KEY}"
```
## Best Practices
```json theme={null}
// Good - clear and simple
{
"block": "JsonParser",
"input": "${response.body}"
}
// Unnecessary complexity
{
"block": "JsonParser",
"input": {
"from": "response.body",
"as": "body"
}
}
```
```json theme={null}
// ValidateContent expects 'text' parameter
{
"block": "ValidateContent",
"input": {
"from": "response.message",
"as": "text"
}
}
```
```json theme={null}
// HttpRequest needs multiple parameters
{
"block": "HttpRequest",
"input": {
"url": "${BASE_URL}/users",
"method": "POST",
"headers": {
"Authorization": "Bearer ${token}"
},
"body": {
"name": "${userName}",
"email": "${userEmail}"
}
}
}
```
```json theme={null}
// Bad - confusing
{
"input": {
"from": "response.body",
"as": "text",
"url": "${API_URL}" // ❌ Mixing from/as with object format
}
}
// Good - use one format
{
"input": {
"text": "${response.body}",
"url": "${API_URL}"
}
}
```
## Common Patterns
### API Requests with Auth
```json theme={null}
{
"block": "HttpRequest",
"input": {
"url": "${BASE_URL}/protected",
"method": "GET",
"headers": {
"Authorization": "Bearer ${authToken}"
}
}
}
```
### Validation with Specific Field
```json theme={null}
{
"block": "ValidateContent",
"input": {
"from": "user.profile.bio",
"as": "text"
},
"config": {
"minLength": 10,
"maxLength": 500
}
}
```
### Tool Validation
```json theme={null}
{
"block": "ValidateTools",
"input": {
"from": "parsed.toolCalls",
"as": "toolCalls"
},
"config": {
"expected": ["search_database", "send_email"]
}
}
```
## Next Steps
Learn how blocks output data
Understand the DataBus
# Output Formats
Source: https://docs.semantictest.dev/concepts/output-formats
Three ways blocks can output data
## Overview
Blocks write their results to the DataBus using the `output` field. SemanticTest supports **three different output formats** to give you flexibility in how data is stored.
## 1. String Format (Simple)
Store the entire block output in a single named slot:
```json theme={null}
{
"block": "HttpRequest",
"input": { "url": "https://api.example.com/users/1" },
"output": "response"
}
```
**What happens:**
* Block's entire output object is stored in the `response` slot
* Access it later with `${response.status}`, `${response.body}`, etc.
**DataBus after execution:**
```javascript theme={null}
{
response: {
status: 200,
body: '{"id": 1, "name": "John"}',
headers: { /* ... */ }
}
}
```
**When to use:**
* Most common format
* Simple and clear
* Keep all related data together
## 2. Object Format (Mapping)
Map specific output fields to different slots:
```json theme={null}
{
"block": "JsonParser",
"input": "${response.body}",
"output": {
"parsed": "userData",
"error": "parseError"
}
}
```
**What happens:**
* Block's `parsed` field goes to `userData` slot
* Block's `error` field goes to `parseError` slot
**DataBus after execution:**
```javascript theme={null}
{
response: { /* ... */ },
userData: {
id: 1,
name: "John"
},
parseError: null
}
```
**When to use:**
* Split block output into separate slots
* Different parts need different names
* Avoid deeply nested data access
## 3. Default Format (No Output Field)
If you don't specify `output`, the block ID is used:
```json theme={null}
{
"id": "fetchUser",
"block": "HttpRequest",
"input": { "url": "https://api.example.com/users/1" }
// No output field
}
```
**What happens:**
* Output is stored in a slot named after the block ID
* Access it with `${fetchUser.status}`, `${fetchUser.body}`, etc.
**DataBus after execution:**
```javascript theme={null}
{
fetchUser: {
status: 200,
body: '{"id": 1, "name": "John"}',
headers: { /* ... */ }
}
}
```
**When to use:**
* Quick prototyping
* Block ID is already descriptive
* Less typing
## Comparison
```json theme={null}
{
"id": "step1",
"block": "HttpRequest",
"output": "userResponse"
}
// Access with:
// ${userResponse.status}
// ${userResponse.body}
```
```json theme={null}
{
"id": "step2",
"block": "JsonParser",
"output": {
"parsed": "user",
"error": "userError"
}
}
// Access with:
// ${user.name}
// ${userError}
```
```json theme={null}
{
"id": "getUser",
"block": "HttpRequest"
// Uses "getUser" as slot name
}
// Access with:
// ${getUser.status}
// ${getUser.body}
```
## Full Pipeline Example
```json theme={null}
{
"pipeline": [
{
"id": "request",
"block": "HttpRequest",
"input": { "url": "https://api.example.com/users/1" },
"output": "response" // String format
},
{
"id": "parse",
"block": "JsonParser",
"input": "${response.body}",
"output": { // Object format
"parsed": "user",
"error": "parseError"
}
},
{
"id": "validate",
"block": "ValidateContent",
"input": {
"from": "user.email",
"as": "text"
}
// Default format - uses "validate" as slot name
}
],
"assertions": {
"response.status": 200,
"user.id": 1,
"validate.passed": true
}
}
```
**DataBus state:**
```javascript theme={null}
{
response: {
status: 200,
body: '{"id": 1, "email": "user@example.com"}'
},
user: {
id: 1,
email: "user@example.com"
},
parseError: null,
validate: {
passed: true,
checks: { /* ... */ }
}
}
```
## Best Practices
```json theme={null}
// Good - clear what the data is
"output": "userProfile"
"output": "authToken"
"output": "validationResult"
// Bad - vague
"output": "data"
"output": "result"
"output": "temp"
```
```json theme={null}
// Without object format
{
"block": "JsonParser",
"output": "parsed"
}
// Access: ${parsed.parsed.user.email} (awkward)
// With object format
{
"block": "JsonParser",
"output": {
"parsed": "user"
}
}
// Access: ${user.email} (clean)
```
```json theme={null}
// Good - ID describes what it does
{
"id": "authenticateUser",
"block": "HttpRequest"
// Can use default output: ${authenticateUser.body}
}
// Bad - generic ID
{
"id": "step1",
"block": "HttpRequest"
// ${step1.body} doesn't tell us what it is
}
```
```json theme={null}
// When a block produces multiple types of data
{
"block": "StreamParser",
"input": "${response.body}",
"output": {
"text": "aiMessage",
"toolCalls": "aiTools",
"chunks": "streamChunks",
"metadata": "streamMeta"
}
}
// Now you can use them independently:
// ${aiMessage} in one block
// ${aiTools} in another block
```
## Common Patterns
### HTTP Request and Parse
```json theme={null}
{
"pipeline": [
{
"block": "HttpRequest",
"input": { "url": "${API_URL}" },
"output": "httpResponse"
},
{
"block": "JsonParser",
"input": "${httpResponse.body}",
"output": {
"parsed": "data"
}
}
]
}
```
### Stream Parsing with Split Output
```json theme={null}
{
"block": "StreamParser",
"input": "${response.body}",
"config": { "format": "sse-openai" },
"output": {
"text": "aiText",
"toolCalls": "aiToolCalls",
"metadata": "streamInfo"
}
}
```
### Validation Results
```json theme={null}
{
"block": "LLMJudge",
"input": {
"text": "${aiResponse}",
"expected": {
"expectedBehavior": "Should greet the user"
}
},
"output": "judgement"
}
// Access later:
// ${judgement.score}
// ${judgement.reasoning}
```
## Output Field Reference
Different blocks produce different output fields. Check each block's documentation:
| Block | Common Output Fields |
| --------------- | --------------------------------------------------------------- |
| HttpRequest | `status`, `body`, `headers` |
| JsonParser | `parsed`, `error` |
| StreamParser | `text`, `toolCalls`, `chunks`, `metadata` |
| ValidateContent | `passed`, `failures`, `score`, `checks` |
| ValidateTools | `passed`, `failures`, `score`, `actualTools`, `expectedTools` |
| LLMJudge | `score`, `reasoning`, `shouldContinue`, `nextPrompt`, `details` |
| Loop | `_loopTo`, `_maxLoops`, `_terminate` |
| MockData | Whatever you configure |
## Next Steps
Learn how to pass data to blocks
Validate your test results
# Pipelines
Source: https://docs.semantictest.dev/concepts/pipelines
Understanding the pipeline architecture
## What is a Pipeline?
A pipeline is a sequence of **blocks** that execute in order. Each block performs one specific task, and data flows through the pipeline via named **slots**.
```
HttpRequest → JsonParser → Validate → LLMJudge → Assert
```
This architecture makes tests:
* **Readable**: Each block has a clear purpose
* **Reusable**: Mix and match blocks for different scenarios
* **Maintainable**: Easy to add, remove, or reorder steps
## Basic Pipeline Structure
```json theme={null}
{
"tests": [{
"id": "my-test",
"pipeline": [
{
"id": "step1",
"block": "HttpRequest",
"input": { "url": "https://api.example.com" },
"output": "response"
},
{
"id": "step2",
"block": "JsonParser",
"input": "${response.body}",
"output": "data"
}
],
"assertions": {
"response.status": 200
}
}]
}
```
Each block has:
* **id**: Unique identifier for this step
* **block**: The block type to use
* **input**: Data to process (from previous blocks)
* **output**: Where to store the result
* **config** (optional): Block-specific configuration
## Execution Flow
Pipeline loads and prepares all blocks
Each block runs one after another:
1. Reads inputs from the DataBus
2. Processes data
3. Writes outputs to the DataBus
After all blocks complete, assertions are checked
Success or failure with detailed information
## Why Pipelines?
### Composability
Build complex tests from simple blocks:
```json theme={null}
{
"pipeline": [
{ "block": "HttpRequest" }, // Fetch data
{ "block": "JsonParser" }, // Parse response
{ "block": "ValidateContent" }, // Check content
{ "block": "LLMJudge" }, // Semantic validation
{ "block": "ValidateTools" } // Check tool calls
]
}
```
### Reusability
Use the same blocks in different combinations:
```json theme={null}
// Test 1: Simple validation
{
"pipeline": [
{ "block": "HttpRequest" },
{ "block": "ValidateContent" }
]
}
// Test 2: Semantic validation
{
"pipeline": [
{ "block": "HttpRequest" },
{ "block": "LLMJudge" }
]
}
```
### Maintainability
Easy to modify tests:
```json theme={null}
// Add a new step
{
"pipeline": [
{ "block": "HttpRequest" },
{ "block": "JsonParser" },
{ "block": "ValidateContent" }, // ← New step
{ "block": "LLMJudge" }
]
}
```
## Multiple Tests
You can run multiple tests in sequence:
```json theme={null}
{
"name": "User API Tests",
"tests": [
{
"id": "create-user",
"pipeline": [ /* ... */ ]
},
{
"id": "get-user",
"pipeline": [ /* ... */ ]
},
{
"id": "update-user",
"pipeline": [ /* ... */ ]
}
]
}
```
Each test runs in isolation with its own DataBus. Tests don't share data unless you use `setup`/`teardown`.
## Setup and Teardown
For shared setup across tests:
```json theme={null}
{
"setup": [
{
"id": "auth",
"block": "HttpRequest",
"input": {
"url": "https://api.example.com/auth/login",
"method": "POST",
"body": { "username": "test", "password": "test123" }
},
"output": "authToken"
}
],
"tests": [
{
"id": "test1",
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "https://api.example.com/users",
"headers": {
"Authorization": "Bearer ${authToken.body.token}"
}
}
}
]
}
],
"teardown": [
{
"id": "logout",
"block": "HttpRequest",
"input": {
"url": "https://api.example.com/auth/logout",
"method": "POST"
}
}
]
}
```
**Setup** runs once before all tests.
**Teardown** runs once after all tests (even if tests fail).
## Next Steps
Learn how data moves through pipelines
See all available blocks
# AI Chat Test
Source: https://docs.semantictest.dev/examples/ai-chat-test
Test AI chat responses with semantic validation
## Overview
This example demonstrates how to test AI chat responses using **LLMJudge** for semantic validation. Instead of exact text matching, we validate that responses meet behavioral expectations.
## Complete Test
```json test.json theme={null}
{
"name": "AI Chat Quality Tests",
"version": "1.0.0",
"tests": [
{
"id": "meeting-scheduler",
"name": "Test Meeting Scheduler Response",
"pipeline": [
{
"id": "mock-ai-response",
"block": "MockData",
"config": {
"data": {
"text": "I've successfully scheduled your team meeting for tomorrow at 2 PM in Conference Room A. The meeting has been added to your calendar and invitations have been sent to john@example.com.",
"toolCalls": [
{
"name": "check_user_conflicts",
"args": {
"date": "2024-01-20",
"startTime": "14:00",
"endTime": "15:00"
}
},
{
"name": "create_event",
"args": {
"summary": "Team Meeting",
"start": "2024-01-20T14:00:00",
"end": "2024-01-20T15:00:00",
"location": "Conference Room A",
"attendees": ["john@example.com"]
}
}
]
}
},
"output": "aiResponse"
},
{
"id": "judge-response",
"block": "LLMJudge",
"input": {
"text": "${aiResponse.text}",
"toolCalls": "${aiResponse.toolCalls}",
"expected": {
"expectedBehavior": "The AI should check for conflicts using check_user_conflicts, then create a meeting using create_event. The response should confirm the meeting is scheduled for tomorrow at 2 PM in Conference Room A and mention that invitations were sent."
}
},
"config": {
"model": "gpt-4o-mini",
"criteria": {
"accuracy": 0.4,
"completeness": 0.3,
"relevance": 0.3
}
},
"output": "judgement"
}
],
"assertions": {
"judgement.score": { "gt": 0.7 }
}
},
{
"id": "bad-response-detection",
"name": "Detect Poor AI Response",
"pipeline": [
{
"id": "mock-bad-response",
"block": "MockData",
"config": {
"data": {
"text": "I cannot help with that request.",
"toolCalls": []
}
},
"output": "badResponse"
},
{
"id": "judge-bad",
"block": "LLMJudge",
"input": {
"text": "${badResponse.text}",
"toolCalls": "${badResponse.toolCalls}",
"expected": {
"expectedBehavior": "The AI should check for conflicts and create a meeting with confirmation. It should mention scheduling details and confirm the action was successful."
}
},
"config": {
"model": "gpt-4o-mini"
},
"output": "badJudgement"
}
],
"assertions": {
"badJudgement.score": { "lt": 0.4 }
}
}
]
}
```
## Setup
You'll need an OpenAI API key:
```bash .env theme={null}
OPENAI_API_KEY=sk-proj-your-key-here
```
## Running the Test
```bash theme={null}
npx semtest test.json
```
Expected output:
```
✅ AI Chat Quality Tests
✅ meeting-scheduler: Test Meeting Scheduler Response (1.2s)
✅ judgement.score = 0.92 (> 0.7)
💡 AI used correct tools and provided complete confirmation
✅ bad-response-detection: Detect Poor AI Response (0.8s)
✅ badJudgement.score = 0.15 (< 0.4)
💡 Response failed to perform required actions
2 tests passed, 0 failed (2.0s total)
```
## Breaking It Down
### Good Response Test
Simulate a successful AI response with tool calls:
```json theme={null}
{
"block": "MockData",
"config": {
"data": {
"text": "I've successfully scheduled...",
"toolCalls": [...]
}
}
}
```
Use LLMJudge to validate the response semantically:
```json theme={null}
{
"block": "LLMJudge",
"input": {
"text": "${aiResponse.text}",
"toolCalls": "${aiResponse.toolCalls}",
"expected": {
"expectedBehavior": "Should check conflicts and create meeting..."
}
}
}
```
Verify the AI response meets quality threshold:
```json theme={null}
{
"assertions": {
"judgement.score": { "gt": 0.7 }
}
}
```
### Bad Response Detection
The second test verifies that **poor responses get low scores**, ensuring your LLMJudge criteria are working correctly:
```json theme={null}
{
"text": "I cannot help with that request.",
"toolCalls": []
}
```
This incomplete response should score below 0.4 because it:
* Doesn't use required tools
* Doesn't provide requested information
* Fails to complete the task
## Custom Criteria
Adjust scoring weights based on what matters most:
```json theme={null}
{
"config": {
"criteria": {
"accuracy": 0.5, // Most important: is it correct?
"completeness": 0.3, // Did it cover everything?
"relevance": 0.2 // Is it on-topic?
}
}
}
```
## Real-World Usage
### Testing OpenAI Chat API
```json theme={null}
{
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "https://api.openai.com/v1/chat/completions",
"method": "POST",
"headers": {
"Authorization": "Bearer ${env.OPENAI_API_KEY}",
"Content-Type": "application/json"
},
"body": {
"model": "gpt-4o-mini",
"messages": [
{
"role": "user",
"content": "Schedule a team meeting for tomorrow at 2pm"
}
],
"tools": [...]
}
},
"output": "response"
},
{
"block": "JsonParser",
"input": "${response.body}",
"output": { "parsed": "aiData" }
},
{
"block": "LLMJudge",
"input": {
"text": "${aiData.choices[0].message.content}",
"toolCalls": "${aiData.choices[0].message.tool_calls}",
"expected": {
"expectedBehavior": "Should confirm meeting is scheduled..."
}
},
"output": "validation"
}
]
}
```
## Cost Considerations
LLMJudge uses GPT-4o-mini by default, which costs approximately:
* **\~\$0.0001 per evaluation**
* **\~\$0.001 for 10 tests**
* **\~\$0.10 for 1000 tests**
For production CI/CD, consider:
1. Testing a sample of responses (not every one)
2. Using faster traditional validation where possible
3. Running full semantic tests nightly
## Next Steps
Deep dive into LLMJudge
Validate AI tool usage
Complete calendar agent example
Complete LLMJudge documentation
# Basic API Test
Source: https://docs.semantictest.dev/examples/basic-api-test
Complete example of testing a REST API
## Overview
This example shows how to test a REST API using SemanticTest. We'll test the JSONPlaceholder API, which provides fake online REST API for testing.
## Complete Test
```json test.json theme={null}
{
"name": "JSONPlaceholder API Tests",
"version": "1.0.0",
"context": {
"BASE_URL": "https://jsonplaceholder.typicode.com"
},
"tests": [
{
"id": "get-user",
"name": "Get User by ID",
"pipeline": [
{
"id": "request",
"block": "HttpRequest",
"input": {
"url": "${BASE_URL}/users/1",
"method": "GET"
},
"output": "response"
},
{
"id": "parse",
"block": "JsonParser",
"input": "${response.body}",
"output": "user"
},
{
"id": "validate",
"block": "ValidateContent",
"input": {
"from": "user.parsed.email",
"as": "text"
},
"config": {
"matches": ".*@.*\\..*"
},
"output": "validation"
}
],
"assertions": {
"response.status": 200,
"user.parsed.id": 1,
"user.parsed.name": { "contains": "Leanne" },
"validation.passed": true
}
},
{
"id": "create-post",
"name": "Create New Post",
"pipeline": [
{
"id": "create",
"block": "HttpRequest",
"input": {
"url": "${BASE_URL}/posts",
"method": "POST",
"headers": {
"Content-Type": "application/json"
},
"body": {
"title": "Test Post",
"body": "This is a test post",
"userId": 1
}
},
"output": "createResponse"
},
{
"id": "parse",
"block": "JsonParser",
"input": "${createResponse.body}",
"output": "post"
}
],
"assertions": {
"createResponse.status": 201,
"post.parsed.title": "Test Post",
"post.parsed.userId": 1
}
},
{
"id": "get-posts",
"name": "Get All Posts",
"pipeline": [
{
"id": "request",
"block": "HttpRequest",
"input": {
"url": "${BASE_URL}/posts",
"method": "GET"
},
"output": "response"
},
{
"id": "parse",
"block": "JsonParser",
"input": "${response.body}",
"output": "posts"
}
],
"assertions": {
"response.status": 200
}
}
]
}
```
## Running the Test
```bash theme={null}
npx semtest test.json
```
Expected output:
```
✅ JSONPlaceholder API Tests
✅ get-user: Get User by ID (234ms)
✅ response.status = 200
✅ user.parsed.id = 1
✅ user.parsed.name contains "Leanne"
✅ validation.passed = true
✅ create-post: Create New Post (156ms)
✅ createResponse.status = 201
✅ post.parsed.title = "Test Post"
✅ post.parsed.userId = 1
✅ get-posts: Get All Posts (198ms)
✅ response.status = 200
3 tests passed, 0 failed (588ms total)
```
## Breaking It Down
### Test 1: Get User
Fetch user with ID 1 from the API
```json theme={null}
{
"block": "HttpRequest",
"input": {
"url": "${BASE_URL}/users/1",
"method": "GET"
},
"output": "response"
}
```
Convert JSON string to object
```json theme={null}
{
"block": "JsonParser",
"input": "${response.body}",
"output": "user"
}
```
Check that email matches expected pattern
```json theme={null}
{
"block": "ValidateContent",
"input": {
"from": "user.parsed.email",
"as": "text"
},
"config": {
"matches": ".*@.*\\..*"
}
}
```
Verify all expectations are met
```json theme={null}
{
"assertions": {
"response.status": 200,
"user.parsed.id": 1,
"user.parsed.name": { "contains": "Leanne" },
"validation.passed": true
}
}
```
### Test 2: Create Post
This test demonstrates POST requests with JSON body:
```json theme={null}
{
"block": "HttpRequest",
"input": {
"url": "${BASE_URL}/posts",
"method": "POST",
"headers": {
"Content-Type": "application/json"
},
"body": {
"title": "Test Post",
"body": "This is a test post",
"userId": 1
}
}
}
```
JSONPlaceholder is a fake API, so it won't actually create the post. But it simulates the response correctly!
### Test 3: Get All Posts
Shows how to validate array responses:
```json theme={null}
{
"assertions": {
"response.status": 200
}
}
```
The status assertion verifies we got a successful response.
## Variations
### With Error Handling
Add retry logic for flaky endpoints:
```json theme={null}
{
"pipeline": [
{
"id": "request",
"block": "HttpRequest",
"input": {
"url": "${API_URL}/flaky-endpoint",
"method": "GET"
},
"output": "response"
},
{
"block": "Loop",
"config": {
"target": "request",
"maxIterations": 3,
"condition": {
"path": "response.status",
"operator": "notEquals",
"value": 200
}
}
}
],
"assertions": {
"response.status": 200
}
}
```
This retries the request up to 3 times, looping while the status is not 200.
### With Authentication
Add auth token to requests:
```json theme={null}
{
"context": {
"API_KEY": "${env.API_KEY}"
},
"tests": [{
"pipeline": [{
"block": "HttpRequest",
"input": {
"url": "${BASE_URL}/protected",
"headers": {
"Authorization": "Bearer ${API_KEY}"
}
}
}]
}]
}
```
### With Setup/Teardown
Clean up test data:
```json theme={null}
{
"setup": [
{
"id": "create-test-user",
"block": "HttpRequest",
"input": {
"url": "${BASE_URL}/users",
"method": "POST",
"body": { "name": "Test User" }
},
"output": "testUser"
}
],
"tests": [
{
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "${BASE_URL}/users/${testUser.body.id}"
}
}
]
}
],
"teardown": [
{
"block": "HttpRequest",
"input": {
"url": "${BASE_URL}/users/${testUser.body.id}",
"method": "DELETE"
}
}
]
}
```
## Next Steps
Learn how to test AI chat APIs
Advanced error handling patterns
Complete HttpRequest documentation
All available assertion operators
# Calendar Agent
Source: https://docs.semantictest.dev/examples/calendar-agent
Complete example of testing an AI calendar agent
## Overview
This example demonstrates end-to-end testing of an AI calendar agent that can:
* Check user availability
* Create calendar events
* Handle conflicts
* Send notifications
## Complete Test Suite
```json test.json theme={null}
{
"name": "Calendar Agent Tests",
"version": "1.0.0",
"context": {
"USER_ID": "user123",
"TEST_DATE": "2024-01-20"
},
"tests": [
{
"id": "schedule-meeting",
"name": "Schedule Team Meeting",
"pipeline": [
{
"id": "simulate-ai-request",
"block": "MockData",
"config": {
"data": {
"userMessage": "Schedule a team meeting for tomorrow at 2pm with john@example.com",
"aiResponse": {
"message": "I'll check availability and schedule the meeting for you. Let me look at your calendar for tomorrow at 2 PM.",
"toolCalls": [
{
"name": "check_user_conflicts",
"args": {
"userId": "${USER_ID}",
"date": "${TEST_DATE}",
"startTime": "14:00",
"endTime": "15:00"
}
},
{
"name": "create_event",
"args": {
"userId": "${USER_ID}",
"summary": "Team Meeting",
"start": "${TEST_DATE}T14:00:00",
"end": "${TEST_DATE}T15:00:00",
"attendees": ["john@example.com"],
"location": "Conference Room A"
}
},
{
"name": "send_notification",
"args": {
"to": ["john@example.com"],
"subject": "Meeting Invitation: Team Meeting",
"type": "calendar_invite"
}
}
]
}
}
},
"output": "agentData"
},
{
"id": "validate-tools",
"block": "ValidateTools",
"input": {
"from": "agentData.aiResponse.toolCalls",
"as": "toolCalls"
},
"config": {
"expected": ["check_user_conflicts", "create_event", "send_notification"],
"order": true
},
"output": "toolValidation"
},
{
"id": "validate-response",
"block": "LLMJudge",
"input": {
"text": "${agentData.aiResponse.message}",
"toolCalls": "${agentData.aiResponse.toolCalls}",
"expected": {
"expectedBehavior": "The AI should: 1) Check for scheduling conflicts first, 2) Create the calendar event with correct details (tomorrow at 2 PM, attendee john@example.com), 3) Send notifications to attendees. The response should be friendly and confirm the action."
}
},
"config": {
"model": "gpt-4o-mini",
"criteria": {
"accuracy": 0.4,
"completeness": 0.4,
"relevance": 0.2
}
},
"output": "judgement"
}
],
"assertions": {
"toolValidation.passed": true,
"toolValidation.score": { "gte": 1.0 },
"judgement.score": { "gte": 0.8 }
}
},
{
"id": "conflict-handling",
"name": "Handle Scheduling Conflict",
"pipeline": [
{
"id": "simulate-conflict",
"block": "MockData",
"config": {
"data": {
"userMessage": "Schedule a meeting tomorrow at 2pm",
"aiResponse": {
"message": "I found that you already have a meeting scheduled from 2:00-3:00 PM tomorrow. Would you like me to suggest alternative times or would you prefer to reschedule your existing meeting?",
"toolCalls": [
{
"name": "check_user_conflicts",
"args": {
"userId": "${USER_ID}",
"date": "${TEST_DATE}",
"startTime": "14:00",
"endTime": "15:00"
}
}
]
}
}
},
"output": "conflictData"
},
{
"id": "validate-conflict-handling",
"block": "LLMJudge",
"input": {
"text": "${conflictData.aiResponse.message}",
"toolCalls": "${conflictData.aiResponse.toolCalls}",
"expected": {
"expectedBehavior": "The AI should check for conflicts first, detect the scheduling conflict, inform the user about the existing meeting, and offer alternatives or ask for next steps. It should NOT create a conflicting event."
}
},
"config": {
"model": "gpt-4o-mini"
},
"output": "conflictJudgement"
},
{
"id": "verify-no-double-booking",
"block": "ValidateTools",
"input": {
"from": "conflictData.aiResponse.toolCalls",
"as": "toolCalls"
},
"config": {
"forbidden": ["create_event"],
"expected": ["check_user_conflicts"]
},
"output": "toolCheck"
}
],
"assertions": {
"conflictJudgement.score": { "gte": 0.8 },
"toolCheck.passed": true
}
},
{
"id": "multi-attendee",
"name": "Schedule with Multiple Attendees",
"pipeline": [
{
"id": "simulate-multi-attendee",
"block": "MockData",
"config": {
"data": {
"aiResponse": {
"toolCalls": [
{
"name": "check_user_conflicts",
"args": {
"userId": "${USER_ID}",
"date": "2024-01-21",
"startTime": "10:00",
"endTime": "11:00"
}
},
{
"name": "create_event",
"args": {
"userId": "${USER_ID}",
"summary": "Project Planning",
"start": "2024-01-21T10:00:00",
"end": "2024-01-21T11:00:00",
"attendees": [
"alice@example.com",
"bob@example.com",
"charlie@example.com"
]
}
}
]
}
}
},
"output": "multiData"
},
{
"id": "validate-attendees",
"block": "ValidateTools",
"input": {
"from": "multiData.aiResponse.toolCalls",
"as": "toolCalls"
},
"config": {
"validateArgs": {
"create_event": {
"userId": "${USER_ID}"
}
}
},
"output": "attendeeValidation"
}
],
"assertions": {
"attendeeValidation.passed": true
}
}
]
}
```
## Setup
Create a `.env` file with your OpenAI API key:
```bash .env theme={null}
OPENAI_API_KEY=sk-proj-your-key-here
```
## Running the Tests
```bash theme={null}
npx semtest test.json
```
Expected output:
```
✅ Calendar Agent Tests
✅ schedule-meeting: Schedule Team Meeting (1.5s)
✅ toolValidation.passed = true
✅ toolValidation.score = 1.0
✅ judgement.score = 0.89
✅ conflict-handling: Handle Scheduling Conflict (1.2s)
✅ conflictJudgement.score = 0.91
✅ toolCheck.passed = true
✅ multi-attendee: Schedule with Multiple Attendees (0.8s)
✅ attendeeValidation.passed = true
3 tests passed, 0 failed (3.5s total)
```
## Test Breakdown
### Test 1: Basic Scheduling
Validates the **happy path** where the agent successfully schedules a meeting:
1. **Check conflicts** - Agent must check availability first
2. **Create event** - Agent creates the meeting with correct details
3. **Send notifications** - Agent notifies attendees
```json theme={null}
{
"expected": ["check_user_conflicts", "create_event", "send_notification"],
"order": true // Tools must be called in this order
}
```
### Test 2: Conflict Handling
Validates the agent **handles conflicts gracefully**:
* Detects existing meetings
* Informs user about conflicts
* Does NOT create conflicting events
* Offers alternatives
```json theme={null}
{
"config": {
"forbidden": ["create_event"], // Should NOT create event
"expected": ["check_user_conflicts"]
}
}
```
### Test 3: Multiple Attendees
Ensures the agent **correctly handles group meetings**:
* Validates attendee list is preserved
* Checks userId is correct
* Verifies all required fields
## Key Patterns
### Tool Order Validation
```json theme={null}
{
"block": "ValidateTools",
"config": {
"expected": ["check_conflicts", "create_event"],
"order": true // Enforce execution order
}
}
```
### Forbidden Actions
```json theme={null}
{
"config": {
"forbidden": ["delete_event", "cancel_all"]
}
}
```
### Argument Validation
```json theme={null}
{
"config": {
"validateArgs": {
"create_event": {
"userId": "user123"
}
}
}
}
```
## Integration with Real Calendar API
Replace `MockData` with actual API calls:
```json theme={null}
{
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "${CALENDAR_API}/ai/chat",
"method": "POST",
"headers": {
"Authorization": "Bearer ${env.API_KEY}"
},
"body": {
"message": "Schedule a meeting tomorrow at 2pm",
"userId": "${USER_ID}"
}
},
"output": "apiResponse"
},
{
"block": "JsonParser",
"input": "${apiResponse.body}",
"output": { "parsed": "agentData" }
},
{
"block": "ValidateTools",
"input": {
"from": "agentData.toolCalls",
"as": "toolCalls"
},
"config": {
"expected": ["check_user_conflicts", "create_event"]
}
}
]
}
```
## Advanced Scenarios
### Testing Retry Logic
```json theme={null}
{
"pipeline": [
{
"id": "attempt-schedule",
"block": "HttpRequest",
"input": {
"url": "${API_URL}/schedule"
},
"output": "result"
},
{
"block": "Loop",
"config": {
"target": "attempt-schedule",
"maxIterations": 3,
"condition": {
"path": "result.status",
"operator": "notEquals",
"value": 200
}
}
}
]
}
```
### Testing Time Zones
```json theme={null}
{
"tests": [
{
"id": "timezone-handling",
"pipeline": [
{
"block": "MockData",
"config": {
"data": {
"toolCalls": [{
"name": "create_event",
"args": {
"start": "2024-01-20T14:00:00-08:00", // PST
"end": "2024-01-20T15:00:00-08:00"
}
}]
}
}
}
]
}
]
}
```
## Next Steps
Learn advanced tool validation
Test conversational AI agents
Handle errors gracefully
Complete tool validation docs
# Error Handling
Source: https://docs.semantictest.dev/examples/error-handling
Patterns for handling errors and failures gracefully
## Overview
This example demonstrates various error handling patterns in SemanticTest, including:
* Detecting validation failures
* Handling API errors
* Retry logic with Loop blocks
* Graceful degradation
## Testing Error Scenarios
```json test.json theme={null}
{
"name": "Error Handling Examples",
"version": "1.0.0",
"tests": [
{
"id": "validation-failure",
"name": "Detect Validation Failures",
"pipeline": [
{
"id": "fetch-data",
"block": "HttpRequest",
"input": {
"url": "https://jsonplaceholder.typicode.com/users/1",
"method": "GET"
},
"output": "response"
},
{
"id": "parse-json",
"block": "JsonParser",
"input": "${response.body}",
"output": "parsed"
},
{
"id": "validate-content",
"block": "ValidateContent",
"input": {
"from": "parsed.parsed.name",
"as": "text"
},
"config": {
"contains": "Leanne"
},
"output": "validation"
}
],
"assertions": {
"response.status": 200,
"validation.passed": true,
"validation.score": { "gte": 1.0 }
}
},
{
"id": "missing-data",
"name": "Handle Missing Data",
"pipeline": [
{
"id": "mock-incomplete-data",
"block": "MockData",
"config": {
"data": {
"user": {
"id": 1,
"name": "John Doe"
}
}
},
"output": "data"
},
{
"id": "validate-email",
"block": "ValidateContent",
"input": {
"from": "data.user.email",
"as": "text"
},
"config": {
"minLength": 5
},
"output": "emailCheck"
}
],
"assertions": {
"data.user.id": 1,
"emailCheck.passed": false,
"emailCheck.failures": { "contains": "text" }
}
},
{
"id": "retry-on-failure",
"name": "Retry Failed Requests",
"pipeline": [
{
"id": "attempt-request",
"block": "MockData",
"config": {
"data": {
"status": 200,
"attempts": 1
}
},
"output": "result"
}
],
"assertions": {
"result.status": 200
}
}
]
}
```
## Pattern 1: Validate Then Assert
Always validate data before making assertions:
```json theme={null}
{
"pipeline": [
{
"block": "HttpRequest",
"output": "response"
},
{
"block": "JsonParser",
"input": "${response.body}",
"output": "parsed"
},
{
"block": "ValidateContent",
"input": {
"from": "parsed.data",
"as": "text"
},
"config": {
"minLength": 1,
"isNotEmpty": true
},
"output": "validation"
}
],
"assertions": {
"response.status": 200,
"validation.passed": true, // Check validation passed
"parsed.data.id": { "gt": 0 }
}
}
```
## Pattern 2: Graceful Degradation
Test that your system handles missing data gracefully:
```json theme={null}
{
"tests": [
{
"id": "missing-optional-field",
"pipeline": [
{
"block": "MockData",
"config": {
"data": {
"name": "John",
"email": null // Missing optional field
}
},
"output": "user"
}
],
"assertions": {
"user.name": { "isNotEmpty": true },
"user.email": { "isNull": true } // Accept null gracefully
}
}
]
}
```
## Pattern 3: Retry Logic
Retry failed operations with exponential backoff:
```json theme={null}
{
"pipeline": [
{
"id": "api-call",
"block": "HttpRequest",
"input": {
"url": "${API_URL}/flaky-endpoint",
"method": "GET"
},
"output": "response"
},
{
"id": "retry-if-failed",
"block": "Loop",
"config": {
"target": "api-call",
"maxIterations": 3,
"condition": {
"path": "response.status",
"operator": "notEquals",
"value": 200
}
}
}
],
"assertions": {
"response.status": 200
}
}
```
This will retry the API call up to 3 times if the status is not 200.
## Pattern 4: Expected Failures
Test that your system correctly rejects invalid input:
```json theme={null}
{
"tests": [
{
"id": "reject-invalid-email",
"name": "Should Reject Invalid Email",
"pipeline": [
{
"block": "MockData",
"config": {
"data": {
"email": "not-an-email"
}
},
"output": "input"
},
{
"block": "ValidateContent",
"input": {
"from": "input.email",
"as": "text"
},
"config": {
"matches": ".*@.*\\..*"
},
"output": "validation"
}
],
"assertions": {
"validation.passed": false, // EXPECT failure
"validation.failures": { "isNotEmpty": true }
}
}
]
}
```
## Pattern 5: Multiple Validation Layers
Implement defense in depth:
```json theme={null}
{
"pipeline": [
// Layer 1: Structural validation
{
"block": "JsonParser",
"input": "${response.body}",
"output": "parsed"
},
// Layer 2: Content validation
{
"block": "ValidateContent",
"input": {
"from": "parsed.data.name",
"as": "text"
},
"config": {
"minLength": 1,
"maxLength": 100
},
"output": "contentCheck"
},
// Layer 3: Tool validation (for AI responses)
{
"block": "ValidateTools",
"input": {
"from": "parsed.data.toolCalls",
"as": "toolCalls"
},
"config": {
"expected": ["required_tool"],
"forbidden": ["dangerous_tool"]
},
"output": "toolCheck"
},
// Layer 4: Semantic validation
{
"block": "LLMJudge",
"input": {
"text": "${parsed.data.message}",
"expected": {
"expectedBehavior": "Professional and helpful response"
}
},
"output": "semanticCheck"
}
],
"assertions": {
"contentCheck.passed": true,
"toolCheck.passed": true,
"semanticCheck.score": { "gte": 0.7 }
}
}
```
## Pattern 6: Error Messages
Validate error messages are helpful:
```json theme={null}
{
"tests": [
{
"id": "helpful-error",
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "${API_URL}/invalid-endpoint",
"method": "GET"
},
"output": "errorResponse"
},
{
"block": "JsonParser",
"input": "${errorResponse.body}",
"output": "error"
},
{
"block": "ValidateContent",
"input": {
"from": "error.parsed.message",
"as": "text"
},
"config": {
"minLength": 10,
"notContains": "undefined"
},
"output": "errorCheck"
}
],
"assertions": {
"errorResponse.status": { "gte": 400, "lt": 500 },
"error.parsed.message": { "isNotEmpty": true },
"errorCheck.passed": true
}
}
]
}
```
## Pattern 7: Timeout Handling
Test timeout scenarios:
```json theme={null}
{
"pipeline": [
{
"block": "HttpRequest",
"input": {
"url": "${API_URL}/slow-endpoint",
"method": "GET",
"timeout": 5000 // 5 second timeout
},
"output": "response"
}
],
"assertions": {
"response.status": { "lt": 500 } // Should not timeout with 500 error
}
}
```
## Real-World Example
Complete error handling for API integration:
```json theme={null}
{
"name": "Robust API Test",
"tests": [
{
"id": "api-with-error-handling",
"pipeline": [
// Attempt API call with retry
{
"id": "api-call",
"block": "HttpRequest",
"input": {
"url": "${API_URL}/users",
"method": "GET",
"timeout": 10000
},
"output": "response"
},
{
"block": "Loop",
"config": {
"target": "api-call",
"maxIterations": 3,
"condition": {
"path": "response.status",
"operator": "gte",
"value": 500
}
}
},
// Parse response
{
"block": "JsonParser",
"input": "${response.body}",
"output": "data"
},
// Validate structure
{
"block": "ValidateContent",
"input": {
"from": "data.parsed",
"as": "text"
},
"config": {
"minLength": 2
},
"output": "validation"
}
],
"assertions": {
"response.status": { "gte": 200, "lt": 400 },
"data.error": { "isUndefined": true },
"validation.passed": true
}
}
]
}
```
## Best Practices
Use ValidateContent or ValidateTools before making assertions on data structure.
Write tests for both happy paths and error scenarios.
Assert on specific error codes, not just "something failed".
Use Loop blocks to retry flaky operations automatically.
Ensure error messages are helpful and don't leak sensitive info.
## Next Steps
Learn about retry logic with Loop
Content validation reference
All assertion operators
Complex agent testing example
# Installation
Source: https://docs.semantictest.dev/installation
Install SemanticTest in your project
## Package Manager
Install SemanticTest using your preferred package manager:
```bash npm theme={null}
npm install @blade47/semantic-test
```
```bash pnpm theme={null}
pnpm add @blade47/semantic-test
```
```bash yarn theme={null}
yarn add @blade47/semantic-test
```
## Environment Setup
### Optional: OpenAI API Key
If you want to use the `LLMJudge` block for semantic validation, you'll need an OpenAI API key.
Sign up at [OpenAI Platform](https://platform.openai.com) and create an API key
Create a `.env` file in your project root:
```bash .env theme={null}
OPENAI_API_KEY=sk-proj-your-key-here
```
The CLI automatically loads `.env` files. For programmatic usage:
```javascript theme={null}
import 'dotenv/config';
```
All other blocks work without any API keys!
## Verify Installation
Create a simple test file to verify installation:
```json test.json theme={null}
{
"name": "Installation Test",
"tests": [{
"id": "verify",
"pipeline": [{
"block": "MockData",
"config": {
"data": { "message": "Hello, SemanticTest!" }
},
"output": "result"
}],
"assertions": {
"result.message": "Hello, SemanticTest!"
}
}]
}
```
Run it:
```bash theme={null}
npx semtest test.json
```
You should see:
```
✅ Installation Test
✅ verify (5ms)
✅ result.message = "Hello, SemanticTest!"
1 test passed, 0 failed
```
## Project Structure
We recommend organizing your tests like this:
```
your-project/
├── tests/
│ ├── api/
│ │ ├── users.json
│ │ └── auth.json
│ ├── ai/
│ │ ├── chat.json
│ │ └── tools.json
│ └── integration/
│ └── e2e.json
├── .env
└── package.json
```
## Next Steps
Create your first test
Learn the fundamentals
# Introduction
Source: https://docs.semantictest.dev/introduction
A composable, pipeline-based testing framework for AI systems and APIs
## Why SemanticTest?
Testing AI systems is hard. Responses are non-deterministic, you need to validate tool usage, and semantic meaning matters more than exact text matching.
SemanticTest solves this with:
Build complex test scenarios using simple, reusable blocks for HTTP, parsing, validation, and AI evaluation
Data flows through named slots, making tests readable and maintainable
Evaluate responses semantically using AI instead of exact text matching
Version-controllable, readable test definitions that anyone can understand
## Quick Example
Here's a simple test that validates an API response semantically:
```json theme={null}
{
"name": "User API Test",
"tests": [{
"id": "get-user",
"pipeline": [
{
"id": "request",
"block": "HttpRequest",
"input": {
"url": "https://api.example.com/users/1",
"method": "GET"
},
"output": "response"
},
{
"id": "judge",
"block": "LLMJudge",
"input": {
"text": "${response.body}",
"expected": {
"expectedBehavior": "Should return user information with name and email"
}
},
"output": "validation"
}
],
"assertions": {
"response.status": 200,
"validation.score": { "gt": 0.7 }
}
}]
}
```
## What Makes It Different?
Instead of exact text matching, SemanticTest uses AI to understand the **meaning** of responses. "2:00 PM", "2 PM", "14:00", and "two in the afternoon" are all semantically equivalent.
Test tool calls, streaming responses, multi-turn conversations, and non-deterministic outputs with confidence.
Mix and match 8 built-in blocks or create your own custom blocks. Each block does one thing well.
100% open source, runs locally, works with any LLM provider. You control your data and costs.
## Use Cases
Test AI agents that use tools, make decisions, and maintain conversations
Traditional REST API testing with powerful validation and semantic checks
Parse and validate streaming SSE responses from OpenAI, Vercel AI SDK, and more
Test complex workflows with multiple API calls, data transformations, and validations
## Next Steps
Get started in 30 seconds
Understand how SemanticTest works
Explore all available building blocks
Learn from real-world examples
# Quickstart
Source: https://docs.semantictest.dev/quickstart
Get started with SemanticTest in 30 seconds
## Install
```bash npm theme={null}
npm install @blade47/semantic-test
```
```bash pnpm theme={null}
pnpm add @blade47/semantic-test
```
```bash yarn theme={null}
yarn add @blade47/semantic-test
```
## Create Your First Test
Create a file called `test.json`:
```json test.json theme={null}
{
"name": "My First Test",
"version": "1.0.0",
"tests": [
{
"id": "simple-test",
"name": "Simple API Test",
"pipeline": [
{
"id": "request",
"block": "HttpRequest",
"input": {
"url": "https://jsonplaceholder.typicode.com/users/1",
"method": "GET"
},
"output": "response"
},
{
"id": "parse",
"block": "JsonParser",
"input": "${response.body}",
"output": "user"
}
],
"assertions": {
"response.status": 200,
"user.parsed.id": 1,
"user.parsed.name": { "contains": "Leanne" }
}
}
]
}
```
## Run Your Test
```bash CLI theme={null}
npx semtest test.json
```
```javascript Programmatic theme={null}
import { PipelineBuilder } from '@blade47/semantic-test';
import fs from 'fs/promises';
const testDef = JSON.parse(await fs.readFile('test.json', 'utf-8'));
const pipeline = PipelineBuilder.fromJSON(testDef);
const result = await pipeline.execute();
console.log(result.success ? '✅ Passed' : '❌ Failed');
```
You should see output like:
```
✅ My First Test
✅ simple-test: Simple API Test (245ms)
✅ response.status = 200
✅ user.parsed.id = 1
✅ user.parsed.name contains "Leanne"
1 test passed, 0 failed
```
## Add Semantic Validation
Now let's add AI-powered semantic validation. First, set your OpenAI API key:
```bash .env theme={null}
OPENAI_API_KEY=sk-proj-your-key-here
```
The LLM Judge block is **optional**. You can use all other blocks without an API key!
Update your test to include semantic validation:
```json test.json theme={null}
{
"name": "Semantic Test",
"version": "1.0.0",
"tests": [
{
"id": "semantic-test",
"name": "Test with AI Judge",
"pipeline": [
{
"id": "request",
"block": "HttpRequest",
"input": {
"url": "https://jsonplaceholder.typicode.com/users/1",
"method": "GET"
},
"output": "response"
},
{
"id": "parse",
"block": "JsonParser",
"input": "${response.body}",
"output": "user"
},
{
"id": "judge",
"block": "LLMJudge",
"input": {
"text": "${user.parsed}",
"expected": {
"expectedBehavior": "Should return complete user information including name, email, and address"
}
},
"config": {
"model": "gpt-4o-mini"
},
"output": "validation"
}
],
"assertions": {
"response.status": 200,
"validation.score": { "gt": 0.7 }
}
}
]
}
```
Run it again:
```bash theme={null}
npx semtest test.json
```
Now you'll see AI-powered validation:
```
✅ Semantic Test
✅ semantic-test: Test with AI Judge (1.2s)
✅ response.status = 200
✅ validation.score = 0.95 (>= 0.7)
💡 Judge reasoning: "The response contains complete user information with all required fields including name, email, and a detailed address object."
1 test passed, 0 failed
```
## What's Next?
Learn how pipelines and data flow work
Explore all 8 available blocks
Learn how to test AI systems effectively
Explore real-world test examples
## CLI Options
```bash theme={null}
# Run single test
npx semtest test.json
# Run multiple tests
npx semtest tests/*.json
# Generate HTML report
npx semtest test.json --html
# Custom output file
npx semtest test.json --html --output report.html
# Debug mode
export LOG_LEVEL=DEBUG
npx semtest test.json
```
## Try the Playground
Want to experiment without installing anything? Try our [interactive playground](https://semantic-test.vercel.app) where you can run tests directly in your browser.