Quality review for every change, from engineers or AI coding agents

I’m Arthur Flam. I started QA-Board, the open-source quality tracking platform, in 2018. I set it up for teams on their own machines and connect their projects, so that every change is reviewed with its outputs and metrics next to the reference’s, whoever wrote it.

Write to hello@qamarks.comServices

What tests and averages miss

Six regressions from QA-Board’s public demo projects, which use made-up data.

Test suites

Green CI, half the tests gone

Asked to fix CI, a coding agent deleted the failing tests. The build is green; the comparison with main shows 12 tests fewer and 2 newly skipped.

The Metrics Diff suite by suite against main: the missions suite has 12 tests fewer, in red, and 10 fewer failures, in green; the other suites are unchanged.

Latency

Less CPU, and seconds of tail latency

A shared cache written by a coding agent saves CPU. Under the same load, p99 latency goes to 1.5 seconds, 26 times the reference’s.

Latency percentiles of the new run and the reference under the same load, on log axes: p50 707 ms and p99 1500 ms, in red, against a 12 ms target the reference mostly meets.

LLM agents

The assistant stopped asking before acting

A coding agent shortened the assistant’s tool descriptions and turned off a confirmation check. Now the assistant reboots a robot without asking, even after the operator said no.

Two transcripts of the same task, reference on the left and new on the right: the reference’s reboot is refused because the operator did not confirm; in the new one the robot reboots. Above, the task passed 0%, down 100%.

ML models

Smaller model, people missing

An int8 detector is 72% smaller than the release model and loses 35% of its mAP on indoor scenes, which the overall average hides.

The Metrics Diff of the int8 model against the fp32 release, slice by slice: indoor mAP down 35% and small objects down 34%, model size down 72% on every slice.

Robotics

The robot thinks it’s somewhere else

New loop closures match identical shelf bays in a warehouse, and one false match pulls the robot’s estimated trajectory away from the ground truth.

A figure-8 trajectory in 3D and over time: the ground truth, the release and the new estimate, which drifts away from the other two.

Imaging

The metric improved, the image got worse

A new denoiser wins 1.6 dB of PSNR and draws dark halos around every leaf. Drag the divider to compare with the fix.

Leaves in front of a facade after the new denoiser: a dark rim outlines every leaf against the wall.new denoiser
The same leaves once the sharpening is clamped: the edges stay crisp, with only faint rims.with the fix

Reviewing changes from AI coding agents

In three of these cases, a coding agent wrote the change. With QA-Board, the agent runs your test set and reads the comparison with the reference itself, so it can fix what it broke before anyone reviews it. A quality gate in CI, experimental for now, fails the job and comments on the pull request. Merging stays a person’s decision.

QA-Board and coding agents

Services

A first project, end to end
A server on your machines and one project connected to it, running in CI. You judge it on your own data.
More projects and tools
Runners for LSF, Slurm, Kubernetes or Celery, the quality gate on pull requests, viewers for your file formats.
Running the server
Docker Compose or Helm, single sign-on, backups, upgrades, networks without internet access.
Training, reviews and coding agent setup
For the people who review results, the engineers who integrate projects, and the agents that check their own changes.

What each one includes

How a first project goes

  1. A call about what you measure and where your code runs. I follow up with a written scope.
  2. I set up the server and the first project with someone from your team. From then on it runs in CI on every push.
  3. Your team reviews real changes for a few weeks, and we adjust what it shows to what they look at.
  4. You decide what comes next. The server and the integration are yours.

Who you’ll work with

Arthur Flam

Me, directly. My day job is platform engineering for research teams, which covers GPU clusters, LLM serving and the coding agents they use. More about me

Contact

Write a few lines about what you measure and how you review changes today.

hello@qamarks.com

Or use the form