Quickstart

Install

npm install @blade47/semantic-test

pnpm add @blade47/semantic-test

yarn add @blade47/semantic-test

Create Your First Test

Create a file called test.json:

test.json

{
  "name": "My First Test",
  "version": "1.0.0",
  "tests": [
    {
      "id": "simple-test",
      "name": "Simple API Test",
      "pipeline": [
        {
          "id": "request",
          "block": "HttpRequest",
          "input": {
            "url": "https://jsonplaceholder.typicode.com/users/1",
            "method": "GET"
          },
          "output": "response"
        },
        {
          "id": "parse",
          "block": "JsonParser",
          "input": "${response.body}",
          "output": "user"
        }
      ],
      "assertions": {
        "response.status": 200,
        "user.parsed.id": 1,
        "user.parsed.name": { "contains": "Leanne" }
      }
    }
  ]
}

Run Your Test

npx semtest test.json

import { PipelineBuilder } from '@blade47/semantic-test';
import fs from 'fs/promises';

const testDef = JSON.parse(await fs.readFile('test.json', 'utf-8'));
const pipeline = PipelineBuilder.fromJSON(testDef);

const result = await pipeline.execute();
console.log(result.success ? '✅ Passed' : '❌ Failed');

You should see output like:

✅ My First Test
  ✅ simple-test: Simple API Test (245ms)
     ✅ response.status = 200
     ✅ user.parsed.id = 1
     ✅ user.parsed.name contains "Leanne"

1 test passed, 0 failed

Add Semantic Validation

Now let’s add AI-powered semantic validation. First, set your OpenAI API key:

.env

OPENAI_API_KEY=sk-proj-your-key-here

The LLM Judge block is optional. You can use all other blocks without an API key!

Update your test to include semantic validation:

test.json

{
  "name": "Semantic Test",
  "version": "1.0.0",
  "tests": [
    {
      "id": "semantic-test",
      "name": "Test with AI Judge",
      "pipeline": [
        {
          "id": "request",
          "block": "HttpRequest",
          "input": {
            "url": "https://jsonplaceholder.typicode.com/users/1",
            "method": "GET"
          },
          "output": "response"
        },
        {
          "id": "parse",
          "block": "JsonParser",
          "input": "${response.body}",
          "output": "user"
        },
        {
          "id": "judge",
          "block": "LLMJudge",
          "input": {
            "text": "${user.parsed}",
            "expected": {
              "expectedBehavior": "Should return complete user information including name, email, and address"
            }
          },
          "config": {
            "model": "gpt-4o-mini"
          },
          "output": "validation"
        }
      ],
      "assertions": {
        "response.status": 200,
        "validation.score": { "gt": 0.7 }
      }
    }
  ]
}

Run it again:

npx semtest test.json

Now you’ll see AI-powered validation:

✅ Semantic Test
  ✅ semantic-test: Test with AI Judge (1.2s)
     ✅ response.status = 200
     ✅ validation.score = 0.95 (>= 0.7)
     💡 Judge reasoning: "The response contains complete user information with all required fields including name, email, and a detailed address object."

1 test passed, 0 failed

What’s Next?

Core Concepts

Learn how pipelines and data flow work

Browse Blocks

Explore all 8 available blocks

AI Testing Guide

Learn how to test AI systems effectively

View Examples

Explore real-world test examples

CLI Options

# Run single test
npx semtest test.json

# Run multiple tests
npx semtest tests/*.json

# Generate HTML report
npx semtest test.json --html

# Custom output file
npx semtest test.json --html --output report.html

# Debug mode
export LOG_LEVEL=DEBUG
npx semtest test.json

Try the Playground

Want to experiment without installing anything? Try our interactive playground where you can run tests directly in your browser.

Get Started

Core Concepts

Blocks

Testing AI Systems

Advanced

Install

Create Your First Test

Run Your Test

Add Semantic Validation

What’s Next?

Core Concepts

Browse Blocks

AI Testing Guide

View Examples

CLI Options

Try the Playground

​Install

​Create Your First Test

​Run Your Test

​Add Semantic Validation

​What’s Next?

Core Concepts

Browse Blocks

AI Testing Guide

View Examples

​CLI Options

​Try the Playground

Install

Create Your First Test

Run Your Test

Add Semantic Validation

What’s Next?

CLI Options

Try the Playground