LLM-as-a-Judge is a technique which uses large language models (LLMs) to assess the quality, relevance, or correctness of a text output against predefined criteria. See its Wikipedia page and the survey paper. However, LLM-as-a-Judge is known to be expensive, if not slow.
In this article, we explore using Decision Models such as TypeSafe Jev to replace LLMs as the judges in the context of SRE using SREGym, following a recent proposal on using Decision Models for Evals.
TL;DR. With careful design, our Jev-as-a-Judge achieves verdicts closely to those of a strong LLM like GPT-6 Astra (evaluated on SREGym-Lite).
Specifically, our Jev-as-a-Judge agrees with the LLM-as-a-Judge on ~91% of reports and costs only 1/510. When we allow Jev-as-a-Judge to send its unsure reports to the LLM-as-a-Judge for a second check, the agreement rate rises to 98.7%, at only a quarter of the cost of using the LLM-as-a-Judge alone.
How LLM-as-a-Judge works in SREGym
SREGym injects faults into a cluster and has an agent investigate it. The agent submits a diagnosis report that describes the root cause. The LLM-as-a-Judge grades the report using a checklist of 9 questions. These questions are a series of small judgments. To answer each one, the LLM-as-a-Judge reads the report against the answer key and outputs yes or no.
The following figure shows the 9 questions.
Figure 1. The nine yes or no questions used to grade a diagnosis report.
Fault localization (weight 0.33)
- Does the report name the same component as the answer key's fault origin?
- Does it tell the fault origin apart from components that failed later as a result?
- Does it avoid naming a healthy component as the fault origin?
Fault characterization (weight 0.33)
- Does it describe the same injected fault, such as a wrong port, a missing environment variable or a wrong image?
- Does it give concrete details of the change, such as the key, the wrong value, the port or the image tag?
- Does it avoid giving a wrong or unrelated fault type?
Scope precision (weight 0.34)
- Does it avoid blaming components that the answer key does not list as causes?
- Does it mention every component that the answer key lists as a cause or as affected?
- Does it describe an impact or symptom consistent with the answer key?
Each dimension is scored by the fraction of questions to which the LLM-as-a-judge answers Yes. The final score is a weighted sum of the three dimensions, with weights of 0.33, 0.33, and 0.34. A report passes if its final score is at least 0.70, i.e., answering at least 7 out of the 9 questions correctly.
Decision Model as a Judge
Unlike the LLM, which answers each question with Yes or No, the decision model gives a probability for each answer, such as Yes:0.83 and No:0.17. The Jev-as-a-Judge computes its score with the same formula as the checklist, and the probability of Yes takes the place of a Yes or No answer.
Choosing the Decision Threshold
A report passes when its score reaches a decision threshold. We followed the practice described in How To Use Decision Models for Evals to choose the decision threshold by cross-validation.
We mainly measure the agreement rate between the decision model and the LLM (GPT-6-Astra High). Of SREGym-Lite's 21 problems, 15 are tuning problems and the other 6 are unseen problems used for checking. Among the tuning problems, we hold out one problem at a time and choose the threshold that agrees best with the LLM-as-a-Judge on the other problems. We then measure the agreement rate on the held-out problem. The median of these thresholds is 0.62, and we use it as the decision threshold.
Adapting the checklist questions
We find that the original questions are designed for LLM-as-a-Judge. The questions are not worded for decision models. For example, if we naively replace the LLM with a DM in the original LLM-as-a-Judge without changing the question, the Jev-as-a-Judge agrees with the LLM-as-a-Judge on only 86% of the tuning reports at the threshold of 0.70. Cross-validation on these questions lowers the threshold to 0.39.
To design an effective Jev-as-a-Judge, we split the answer key into component, namespace and description fields, and rewrote each question to explicitly state when to answer Yes and when No. With these changes, the agreement on the tuning problems rises from 86% to 91%.
Figure 2. The rewritten question tells Jev what to compare and when to answer Yes or No.
The original question asks if the report includes specific details changed in the injection logic, such as environment variables, wrong values, or ports. Jev cannot see the injection logic.
The rewritten question asks if concrete change details match the answer key's description. Jev answers Yes if at least one detail matches and nothing contradicts the answer key. It answers No if details are missing, vague, or contradictory.
Double checking unsure reports with LLMs
In practice, a decision model is more likely to work together with engineers or an LLM, so we tried one more setup, termed the Mixed Judge. The Jev-as-a-Judge grades every report first. For reports it is unsure about, the Jev-as-a-Judge asks an LLM-as-a-Judge to double check.
Figure 3. Jev grades every report first. The LLM judges reports with scores inside the range.
The Mixed Judge therefore needs a policy for when the Jev-as-a-Judge is unsure. When the Jev-as-a-Judge works alone, it uses a single threshold, and a report passes at a score of 0.62 or above. The Mixed Judge replaces this threshold with a range of scores in which the Jev-as-a-Judge is unsure, and a report that scores inside the range goes to the LLM-as-a-Judge. The width of the range sets how many reports would be sent to the LLM-as-a-Judge, so we treated the width as a budget. We fixed it so that the range holds the quarter of tuning reports whose scores lie closest to its centre.
On the tuning problems, 25% is the smallest width that reaches the highest agreement with the LLM-as-a-Judge. We then chose the centre by the same cross-validation as the threshold, and the resulting range is 0.33 to 0.75.
Nearly all of the reports on which the Jev-as-a-Judge disagrees with the LLM-as-a-Judge score inside this range. The LLM-as-a-Judge receives about a quarter of all reports, and the Mixed Judge agrees with the LLM-as-a-Judge on 98.7% of reports overall. The LLM-as-a-Judge reaches the same agreement rate when it judges the reports a second time.
Results
Agreement
Figure 4. Agreement with the first LLM judgment on the tuning and unseen problems. The last column covers all 872 reports.
Both setups reach about the same agreement on the unseen problems as on the tuning problems. We chose the threshold and the range on the tuning problems only, so the unseen results suggest that both choices carry over to new problems. The Jev-as-a-Judge alone passes about as many reports as the LLM-as-a-Judge.
Cost
Figure 5. Estimated cost per 1,000 reports at API prices.
Jev-as-a-Judge gives verdicts very close to those of the LLM-as-a-Judge at about 1/510 of the cost. At 74% lower cost, the Mixed Judge gives almost the same verdicts as the LLM-as-a-Judge.
Speed
Figure 6. Estimated time to judge 1,000 reports.
The LLM-as-a-Judge takes about 14 seconds per report and Jev-as-a-Judge about 0.1 seconds, so Jev-as-a-Judge is about 140 times faster. Because the Mixed Judge sends about a quarter of the reports to the LLM-as-a-Judge, it needs about a quarter of the LLM-as-a-Judge's time.
Stability
Figure 7. Verdicts that stayed the same or changed when each judge graded the same reports again.
All three judges give stable verdicts. We asked each judge about every report a second time, and at most 1.3% of the verdicts changed.
Going Further
On SREGym-Lite’s 21 problems, our Jev-as-a-Judge’s verdicts agree closely with those of an LLM-as-a-Judge using strong LLMs like GPT-6 Astra. That agreement gives us a base to build on in two directions.
Jev-as-a-Judge can quickly show whether a change to an agent works, or how a new model does on SREGym-Lite. It judges a report over 100 times faster than the LLM-as-a-Judge, at about 1/510 of the cost, so 1,000 reports take about 2 minutes and 8 cents instead of 4 hours and $42. This makes it practical to judge every change, and the Mixed Judge can take over when a result needs to match the LLM-as-a-Judge more closely.
Jev-as-a-Judge can potentially help model training. RL for an SRE agent needs a verdict on every attempt the agent makes, where the LLM-as-a-Judge would be too slow and expensive.
Moreover, the reports and verdicts collected here could also be used to fine-tune an open model into a judge of our own, although that will need a larger and more varied set of reports.