Skip to main content

​
Install

npm install @blade47/semantic-test

​
Create Your First Test

Create a file called test.json:
test.json
{
  "name": "My First Test",
  "version": "1.0.0",
  "tests": [
    {
      "id": "simple-test",
      "name": "Simple API Test",
      "pipeline": [
        {
          "id": "request",
          "block": "HttpRequest",
          "input": {
            "url": "https://jsonplaceholder.typicode.com/users/1",
            "method": "GET"
          },
          "output": "response"
        },
        {
          "id": "parse",
          "block": "JsonParser",
          "input": "${response.body}",
          "output": "user"
        }
      ],
      "assertions": {
        "response.status": 200,
        "user.parsed.id": 1,
        "user.parsed.name": { "contains": "Leanne" }
      }
    }
  ]
}

​
Run Your Test

npx semtest test.json
You should see output like: 
✅ My First Test
  ✅ simple-test: Simple API Test (245ms)
     ✅ response.status = 200
     ✅ user.parsed.id = 1
     ✅ user.parsed.name contains "Leanne"

1 test passed, 0 failed

​
Add Semantic Validation

Now let’s add AI-powered semantic validation. First, set your OpenAI API key:
.env
OPENAI_API_KEY=sk-proj-your-key-here
The LLM Judge block is optional. You can use all other blocks without an API key!
Update your test to include semantic validation:
test.json
{
  "name": "Semantic Test",
  "version": "1.0.0",
  "tests": [
    {
      "id": "semantic-test",
      "name": "Test with AI Judge",
      "pipeline": [
        {
          "id": "request",
          "block": "HttpRequest",
          "input": {
            "url": "https://jsonplaceholder.typicode.com/users/1",
            "method": "GET"
          },
          "output": "response"
        },
        {
          "id": "parse",
          "block": "JsonParser",
          "input": "${response.body}",
          "output": "user"
        },
        {
          "id": "judge",
          "block": "LLMJudge",
          "input": {
            "text": "${user.parsed}",
            "expected": {
              "expectedBehavior": "Should return complete user information including name, email, and address"
            }
          },
          "config": {
            "model": "gpt-4o-mini"
          },
          "output": "validation"
        }
      ],
      "assertions": {
        "response.status": 200,
        "validation.score": { "gt": 0.7 }
      }
    }
  ]
}
Run it again: 
npx semtest test.json
Now you’ll see AI-powered validation: 
✅ Semantic Test
  ✅ semantic-test: Test with AI Judge (1.2s)
     ✅ response.status = 200
     ✅ validation.score = 0.95 (>= 0.7)
     💡 Judge reasoning: "The response contains complete user information with all required fields including name, email, and a detailed address object."

1 test passed, 0 failed

​
What’s Next?

Core Concepts

Learn how pipelines and data flow work

Browse Blocks

Explore all 8 available blocks

AI Testing Guide

Learn how to test AI systems effectively

View Examples

Explore real-world test examples

​
CLI Options

# Run single test
npx semtest test.json

# Run multiple tests
npx semtest tests/*.json

# Generate HTML report
npx semtest test.json --html

# Custom output file
npx semtest test.json --html --output report.html

# Debug mode
export LOG_LEVEL=DEBUG
npx semtest test.json

​
Try the Playground

Want to experiment without installing anything? Try our interactive playground where you can run tests directly in your browser.