Anu Kama

Selected work, 02

An AI that gets graded before it ships.

On the platform I build at my day job, AI assistants write messages to real staff. A wrong number or a blank that never got filled in goes straight to a person. So I built a loop around the assistant: record every run, check it, grade it, and never let it change itself.

92test cases guard what the assistant sends

7suites, run before a change can merge

0changes the AI is allowed to make to itself

See it run

This is internal software, so there is no public link. Here is one of its checks at work. Each week the assistant writes a short review of the numbers. The numbers are counted by code first. Then every number in the assistant's text is compared with what was counted.

Counted by code
  calls 41   booked 9   showed 6

The assistant wrote
  "Bookings rose to 12 this week."

Number check
  41  counted
  9   counted
  6   counted
  12  NOT in the counted set

VERDICT  rejected_numbers
The review is thrown away. Nobody sees it.

The numbers in this example are made up. The check is real, and it has 9 test cases of its own.

The loop

The quality loop around the assistant Every run is recorded. Rules check it with no AI involved. A fixed judge grades what was delivered. The AI may draft a fix to its own instructions. A person decides whether that fix is used. Every run is recorded Rules check it. No AI. A fixed judge grades it The AI may draft a fix A person decides nothing applies itself

The outlined step is the only one that can change what the assistant does.

The choices, and why

  1. Rules first. The model second.

    The first check uses no AI at all. Did the run fail? Is the output empty? Is there a blank like [name] that never got filled, or the word "undefined" where a value should be? These are cheap, and nobody can argue with them. A run that passes them is marked "not judged". It is never marked "pass", because not broken is not the same as good.

    No model is called at this step

  2. The judge grades what was sent, not what the AI says it did.

    An assistant will happily report "I sent the summary". That sentence is not the summary. The judge only reads the real message that went out. If the loop could not capture that message, the judge refuses to grade and records that the test setup is broken.

    A fixed model and a fixed version of the grading instructions

  3. The model never checks its own numbers.

    Asking an AI "are you sure?" gets a confident yes. So the counting is done by ordinary code, and the check is a plain comparison: is every number in the text one of the counted numbers? One number that is not gets the whole review thrown away.

    9test cases for this one check

  4. Nothing applies itself.

    The system can draft a better version of its own instructions. It needs at least three recent failures before it tries. Plain code then rejects a draft that quietly cut too much or dropped a fixed detail. What survives is saved as a candidate. A person promotes it, and undoing it is one step.

    3failures before it may propose anything

  5. The guards have tests of their own.

    A guard that is wrong is worse than no guard, because people trust it. So each one has a set of cases, built from the real code, that must pass before a change to it can merge. Several cases exist because the guard was wrong once.

    92cases across 7 suites

What broke

  1. The judge failed every run it graded.

    Five runs, five fails. The work was fine. The judge was reading the assistant's notes about what it did, not the message it delivered. That is where the rule above came from: capture the real output, or do not grade.

    Jul 252026, 5 of 5 failed

  2. A rule saw problems that were not there.

    One rule looked for unfilled blanks. It also matched ordinary web links. Seven of twelve runs were marked as failures, and the system wrote a long fix for a bug that did not exist. I fixed the rule and gave the rules their own test cases, so a false alarm cannot quietly come back.

    Jul 252026, 7 of 12 were false alarms

  3. The loop could not see emails or texts.

    The list of "things that count as delivering" named two tools that did not exist. Every email and text the assistant sent was invisible to the grading. Nothing looked wrong, because nothing was being looked at. The list is now checked against the real tools.

    Jul 272026

What I would do next

  • The loop has not yet made the assistant better in a way I can measure. In its first week it recorded 41 runs, could grade 3 of them, and proposed nothing. It has caught real faults in itself. That is not the same thing.
  • The test cases catch clear breakage. They are too few to detect a small drop in quality.
  • The judge is fixed by name, not by an exact dated version. It should be the dated one.
  • A second system tries to fix code errors by itself, with a size limit and an automatic undo. It passed one supervised test on a bug I planted. It has not yet fixed a real one alone.

The first line is the honest state of this work.