Steadygate

Your eval passed. Your agent got worse.

Before you release an AI feature, you test it. But AI answers vary from run to run, so test scores wobble even when nothing is wrong. Steadygate compares each test run with your own recent history and gives you one of three plain answers.

Install now Try the demo View on GitHub

Other platforms and pip install are in the install section. Windows may say "Windows protected your PC": choose More info → Run anyway. Mac: Unzip it, then open the app. If macOS says it can't be checked, go to System Settings → Privacy & Security and choose Open Anyway.

Who it's for

AI and LLM app teams

You're in the right place if… you run evals with promptfoo, DeepEval or your own script, and a fixed threshold either blocks good releases or lets bad ones through.

What you get: a go/no-go that knows your normal variation.

QA and release owners

You're in the right place if… you sign off releases, and now you're asked to test AI features whose outputs vary.

What you get: one clear verdict per run, an exit code your pipeline can act on, and a report in the pull request.

Teams with flaky tests

You're in the right place if… noisy end-to-end, performance or game tests cry wolf.

What you get: repeated-trial pass rates with confidence intervals, so 7/10 isn't treated as a failure, and crashes are kept apart from real regressions.

If your tests give the same answer every time, a plain pass/fail is all you need.

The example: a support chatbot

You're about to release an update to a customer-support chatbot. Before every release, an automatic test asks it 5 real customer questions, such as "Can I get a refund?" and "Where's my order?". Each question is asked 10 times, and an AI grader marks every answer right or wrong. Your last 3 releases scored about 90%. Is the new version safe to ship?

What's measured

For each question, the score is the share of its 10 answers the grader accepted: 9 right out of 10 is 90%. Response time is tracked too.

Why scores wobble

Ask the same chatbot the same questions twice and you get slightly different answers, so the score moves a few points with nothing broken. A fixed rule like "fail below 90%" would block healthy releases.

What Steadygate does

It compares the new score with your recent releases and only objects to a drop bigger than the normal wobble. If part of the test crashed, it says can't tell instead of pretending all is well.

Try it

Pick what happened in the new version, then see Steadygate's answer and the checks behind it. Every answer comes from the real Steadygate engine.

What happened?

Or try your own version

Loading the example…

How it works in your project

  1. 1 Run your tests as usual

    Keep the tool you already use: promptfoo, pytest, JUnit or a simple JSON file. Steadygate reads its results file; it doesn't replace your tests.

  2. 2 Compare with your history

    Every passing run on your main branch is remembered. A new run is compared with the middle value of recent runs, not a number someone guessed once.

  3. 3 Get one clear answer

    Safe to ship, don't ship, or can't tell. Your CI gets an exit code (0, 1 or 2) to act on, and the pull request gets the report as a comment.

Add it to your repo

Or install with pip, then set up a project:

Terminal
pip install steadygate
steadygate init

Run your evals as usual, then hand the results file to the Steadygate action. Runs on main build the history; pull requests get the verdict and a comment.

.github/workflows/steadygate.yml
name: steadygate
on:
  push:
    branches: [main]
  pull_request:

jobs:
  gate:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write
    steps:
      - uses: actions/checkout@v4
      - run: npx promptfoo eval --repeat 5 --output results.json  # or pytest, Jest, ...
        continue-on-error: true   # failing evals still reach the gate
      - uses: Mesmoraz/steadygate@main
        with:
          results: results.json

Action inputs and outputs Quick start in the README Run the example locally

Why you can trust the answer