← BACK

CASE STUDY23 OCT 2026 · 6 MIN READ

Rebuilding Recruit41's interview proctoring

About half of its cheating flags were wrong. How we found out why, and why the fix was mostly listening, labelling and keeping the final call out of the model's hands.

A dot-matrix drawing. A face glances up and away to think, above a struck-out "Looking away" tag. An arrow points to an orange sound wave labelled "Listen first".

What proctoring does here

Recruit41 runs AI interviews. A candidate joins, shares their screen, and talks to an AI interviewer for around 40 minutes while it records the camera, the screen and the mic. After the interview, a second AI goes through the recording and looks for signs of outside help, like reading answers off another screen or someone feeding them lines.

The result is a badge on the candidate: no flags, suspicious, or cheating. Each flag points to a timestamp in the recording, so a recruiter can jump straight to it.

one interview, about 40 minutescamerascreenmic040 minflags point at momentsno flagssuspiciouscheatingone badge
Every interview records the camera, the screen and the mic. Flags point to moments in it, and they add up to one badge.

That badge mattered more than it looks. It could move a candidate from "advance" to "watchlist" or "reject" on its own.

About half the flags were wrong

When I joined, the system had been running for about a year, and nobody quite trusted it. Recruiters left notes like "it says she's using an AI chat in another browser, but there's nothing like that in her screen share", or "all 16 events say the same thing, that the candidate is looking off screen".

When we checked, roughly half of the cheating flags were false positives. That's not a small error rate you tune away. Half the time, an honest candidate was being quietly marked down.

What I found when I looked

I started the boring way. I ran real interviews through it by hand and watched every flagged clip myself.

Most of the "cheating" was people looking away to think. Which makes sense if you think about it. The interview screen sits below or beside the camera, so everyone looks off camera all the time. The old prompt treated looking away as a strong signal, so the system kept flagging the most normal thing in the world.

Then I checked the clips themselves, and they had no audio track. The step that cut clips out of the recording was dropping the sound, and the prompt also told the model to judge only what it could see. So the part of the system watching for cheating was deaf.

That turned out to be the key insight. Cheating is usually heard before it's seen. When someone reads an answer, you can hear it. The pause, the sudden polish, the flat reading voice. Eyes alone are a much weaker clue, because people look away for all sorts of reasons.

There were a few smaller problems too:

  • The model was told to use its full budget of clips, so it kept looking until it found something.
  • Each round could see its earlier guesses, so it ended up agreeing with itself.
  • The model gave the final verdict, so the same interview could get a different answer on a second run.

The quick fix that wasn't enough

The first thing I tried was small. Put the sound back into the clips, and tell the model that looking away only counts if something else backs it up.

It helped a lot. On a test set, it cleared about 43% of the "definitely cheating" cases, and every one of those had been flagged on gaze alone. It even caught a real cheater the old system had missed.

But when I ran the same interviews twice, some of them flipped. Cheating on one run, not on the next. A verdict that changes when you press the button again isn't a verdict. So the quick fix showed us the direction, but the design itself had to change.

Perception in the model, decisions in code

The new design came down to one idea: let the model perceive, let code decide, and let a human overrule.

beforerecordingpick clipswatch,no soundmodel'scalldeaf, and it flippedafterlisten toevery sectioncheap audio passledgerwrite onlyjudge picksmomentswatch, nowwith soundloopcode turns itinto a verdicthuman agreesor disagreessame notes in,same verdict out
The old pipeline let the model decide. The new one only lets it look and listen.

In practice, it works like this:

  • Listen first. A cheap audio pass goes through every section of the interview and notes anything that sounds off.
  • Write it down. Everything the model hears or sees goes into a ledger. The tools only write to it, they never read it, so the video check doesn't know what the audio check heard. When they agree, it actually means something.
  • Look where it matters. A judge reads the ledger and picks a few moments to watch, now with sound, or decides it has seen enough. It controls the search, but it never labels anyone.
  • Decide in code. A small piece of plain Python turns the ledger into a verdict. Same notes in, same verdict out, every time. Looking away on its own can never make someone a cheater, and the system can't call someone clean without having seen every section.
  • Let people overrule it. Reviewers can agree or disagree with each flag. A disagreement removes that flag and re-sorts the candidate, and once a human has reviewed an interview, the AI never overwrites it.

We used Gemini Flash for everything. Audio was doing most of the work, Flash caught what our reviewers caught, and one model meant one prompt style and one set of evals. We also skipped agent frameworks. The whole loop is about 40 lines of Python, which is easy to replay and test offline.

Because this runs on thousands of recordings, it also had to be sturdy. A section that fails gets skipped instead of killing the whole run, jobs retry safely, and the final write never stomps on a verdict a human already gave.

The hard part was labelling

The code wasn't the hardest part. Knowing whether it was right was.

So before tuning anything, we labelled. Seven of us, basically the whole team, spent 30 minutes a day going through interviews. For each one, you watch it, judge every flag, judge the final verdict, and write down what gave it away. We did that for 500 interviews.

Those labels became the test. Every prompt change and rule change got scored against what the humans said, and I kept re-running the same interviews to check that the numbers held from one run to the next. I set up caching so that changing a rule re-scored everything for free, and only a prompt change cost a fresh run. That made it cheap to try things, which mattered a lot when most of the work was small tweaks.

Honestly, this was the slow, manual part, and it's the part I'd tell anyone building something similar not to skip.

cheating flags that turned out wrongbeforeabout 50%after3 to 4%85%of verdicts matchedwhat reviewers said500interviews labelledby hand, by 7 of us
Measured against our own labelled interviews.

What I took away

False positives went from about half to 3 to 4%, and 85% of verdicts now match what our human reviewers say. More importantly, recruiters started trusting the badge again, and honest candidates stopped getting marked down for thinking.

If I had to sum it up in three lines, they'd be the ones I ended my internal walkthrough with. Label with humans first. Give the model perception, not the decision. Keep the decision in code you can replay.

← All posts