Decision Models Lead, LLMs Review: Fast and Accurate SRE Diagnosis

Site reliability engineering (SRE) agents diagnose incidents by inspecting logs, metrics, traces, and cluster state. At each step, they decide what to inspect next and which evidence supports a diagnosis. Decision models can make these repeated decisions quickly, which is especially valuable when responding to critical failures.

In our first study, the LLM agent led the investigation, with TypeSafe AI’s Jev supporting it by ranking tests and reviewing conclusions. In our second experiment, Jev handled diagnosis without an LLM: a programmatic collector gathered cluster evidence, and Jev chose which component to inspect and which observations to submit. That pipeline passed 76.2% of diagnoses on SREGym-Lite, but the collector could miss important evidence and Jev could favor the wrong clue.

In this post, we present a decision-model-led diagnosis agent, bringing the LLM back, but this time in a supporting role. The decision model chooses which tool to run on which target and uses the results to propose a root cause. After the investigation, the LLM reviews this choice against the collected evidence, either confirming it or replacing it with another supplied candidate. It cannot run tools or collect new evidence. Whether the LLM keeps or changes the cause, the decision model then selects observations supporting the final choice from the evidence already collected. The LLM uses those selected observations to write the diagnosis report.

We run two types of experiments to measure the benefit of LLM review. With review, the LLM can confirm or change the proposed root cause. Without review, we keep the decision model's choice. Both setups then select supporting observations and use the LLM to write the report.

Across 21 SREGym-Lite faults, with five attempts per fault in each setup, Jev passes 89.5% (94/105) of diagnoses with LLM review and 78.1% (82/105) without it. GPT-6 Luna Decisions passes 95.2% (100/105) with review and 79.0% (83/105) without it. Mean diagnosis time ranges from 31.9 to 41.5 seconds across these setups.

The improvements from LLM review also show us where the decision model went wrong. For each run where the LLM corrects the decision model's root-cause choice, we save the root-cause question, the evidence and possible answers shown to the decision model, and the LLM's corrected answer. We then test if clearer wording or answer choices help the decision model reach the corrected answer using the saved evidence, without rerunning the investigation.

How the agent works

We built the agent based on the pipeline presented in our last blog post. The main change is in the investigation. In our earlier pipeline, the decision model chose which component to inspect, and the pipeline followed a predefined routine to gather more evidence about it. In this new agent, the decision model also chooses which tool to run on which target at each step. Here is the implementation.

First, the programmatic collector reads the cluster state and scores each component using signals such as ongoing errors and warning events. It then checks the two highest-scoring components by reading their logs or inspecting their workload state.

Expand figure

Blue blocks are programmatic steps, teal blocks are decision-model steps, and orange blocks are LLM steps.

1. Collector: Read the cluster state, looking for changes and signs of failure.
2. Collector: Check logs or workload state for early signs of failure.
3. Orchestration: Build the tool menu by listing the checks the agent can run next.
4. Decision model, one call per step: Choose a check and likely cause. Decide if more evidence is needed.
5. If a check is selected: Run the read-only tool and add its result to the notes. Loop back to build the tool menu for the next step.
6. When a stop rule or budget ends the investigation: The LLM reviews the root cause, but cannot run tools. If it agrees, the agent keeps the decision model's choice. If it disagrees, the agent replaces the cause with the LLM's choice from the supplied candidates. Both paths continue to evidence selection.
7. Decision model: Keep the observations that support the final root cause.
8. LLM: Write and submit the report using the selected evidence.

The investigation then runs one step at a time. First, a list of possible checks is prepared programmatically from the evidence collected so far. One check might read the frontend's logs. Another might test whether it can reach another service. Checks linked to errors or earlier clues come first. By default, the list is built with a limit of 40 checks, plus submit_now to request an end to the investigation.

Then, the list of checks and the evidence collected so far are sent to the decision model together in one request. The questions include:

• Which tool should run next, and on which component or connection? The model chooses a check from the prepared list. This answer determines both the tool and where it will run. The model can instead select submit_now to request an end to the investigation.
• Does the previous tool's output show signs of a problem? The answer is saved alongside that output in the notes, so later steps can distinguish findings that suggest a problem from normal results. If the tool read logs, the model also identifies the kinds of errors reported.
• Is there enough evidence to write a diagnosis? The answer can be yes or no.
• Which root cause best fits the evidence so far? The model can answer "not yet known."

The evidence and root-cause questions are answered independently. Naming a possible cause does not mean there is enough evidence to finish.

The stopping rules then use these answers to decide whether to continue. Selecting submit_now does not submit the report. By default, the first request to stop is accepted if the model assigns it a probability of at least 0.5. If it is below 0.5, that first request is ignored. If the model selects submit_now again at the next step, the investigation can end even when it still says more evidence is needed. Step or time limits, or no remaining checks, can also end it.

If the investigation continues, the tool for the next check is run on the specified component or connection. When a request to stop is ignored, the highest-ranked available check other than submit_now is used instead. The tool's output is added to the investigation notes. A new step then begins with those updated notes and a new list of possible checks.

Once the investigation ends, the LLM reviews the proposed root cause using the collected evidence and the leading candidates. It can confirm the decision model's choice or replace it with another supplied candidate. It cannot run tools or collect new evidence.

Whether the cause was kept or changed, the decision model then uses yes/no questions to select observations from the notes that support the final cause. The LLM uses those selected observations to write the diagnosis report.

Explore the read-only tool menu · Figure 2

Expand figure

Looks at	Tool	What it shows
Workload	describe	Pod states, restarts, events, container settings
Workload	config_and_changes	Who changed its spec and when, and the ConfigMaps and Secrets it reads
Workload	related_objects	Volume claims, Service accounts, policies and autoscalers around it
Logs	logs	Ongoing and stopped error lines, and the components they name
Traffic	service_view	Whether its Services select its pods, with endpoints and ports
Traffic	call_edges	Who calls whom, and how often those calls fail
Traffic	client_check	From a caller's pod, whether the name resolves and the port answers
Metrics	metric_series	15 minutes of CPU, memory, restarts, request rate, errors and latency
Backends	backend_probe	A read-only health command inside a database or queue
Cluster	cluster_objects	Webhooks, quotas, network policies, RBAC, DNS, nodes, storage
Cluster	object_view	The content of one object, such as a ConfigMap
Cluster	events	Recent Kubernetes events, warnings first

The agent in action

Let us look at a Jev run on SREGym-Lite's wrong_dns_policy_astronomy_shop fault in the Astronomy Shop application.

In this fault, the frontend Deployment's DNS settings had been changed. Its Pod sent queries to an external DNS server instead of the cluster's DNS. The frontend could no longer resolve the names of the services it called.

The programmatic collector summarized the 25 components in the cluster and ranked them by signs of failure. These were the top three:

deployment/frontend
  1/1 ready; dnsPolicy and dnsConfig.nameservers changed 134s before the first symptom
  54 ongoing error lines, e.g. Name resolution failed for target dns:product-catalog:8080
daemonset/otel-collector-agent
  3/3 ready; restarted 140s before the first symptom, no setting changed
  8 ongoing error lines, e.g. call to /stats/summary endpoint failed
deployment/postgresql
  1/1 ready; memory 90Mi of its 100Mi limit

Jev first judged that the evidence was insufficient and chose to test whether the frontend could reach product-catalog.

Expand figure

Step 1. One request to Jev, 0.22 S. Which tool should run next? Top 6 of 41 options.

Tool	Target	Probability
client_check	frontend → product-catalog	0.62
describe	product-catalog	0.13
logs	product-catalog	0.07
config_and_changes	frontend	0.05
client_check	frontend → cart	0.03
submit_now		0.03

Is the last result abnormal? No, probability of yes 0.34. Is the evidence enough? No, probability of yes 0.39.

Where did the fault start?

• cluster_object: 0.37
• Deployment frontend: 0.30
• Deployment product-catalog: 0.14

The test showed that the frontend Pod could not resolve the product-catalog Service name. Checks toward cart and checkout failed in the same way, although none of those Services had been changed. After seven tool-use decisions, Jev stopped and selected cluster_object as the root cause. This catch-all option names no specific object. Jev still judged the evidence insufficient; the choice to submit and the sufficiency judgment were separate answers.

Expand figure

Step 7. One request to Jev, 0.16 S. Which tool should run next? Top 6 of 41 options.

Tool	Target	Probability
submit_now		0.17
object_view	Service checkout	0.14
logs	cart	0.13
config_and_changes	frontend	0.10
client_check	frontend → product-reviews	0.09
describe	cart	0.07

Is the last result abnormal? No, probability of yes 0.11. Is the evidence enough? No, probability of yes 0.37.

Where did the fault start?

• cluster_object: 0.62
• Deployment frontend: 0.21

The LLM compared Jev's leading candidates and their evidence. The frontend's DNS settings had changed just before the symptoms, while the Services it called were unchanged. The reviewer used this evidence to replace cluster_object with the frontend Deployment.

Finally, the decision model chose the observations that support this root cause, and the LLM wrote the report from them. The submitted diagnosis began like this:

The root cause is the frontend Deployment, which serves the frontend Service in the astronomy-shop namespace. Its DNS settings changed shortly before symptoms began. The new dnsPolicy or dnsConfig prevents the frontend pod from resolving in-cluster Service names, so requests from the frontend fail before they can reach their targets.

The diagnosis took 44.3 seconds. LLM review and report writing took about 23 seconds; the decision model's API calls took about 1 second. The remaining roughly 20 seconds were spent outside those model calls, including startup, collecting cluster state, running tools, and coordinating the investigation.

Results on SREGym-Lite

We evaluated Jev and GPT-6 Luna Decisions on the same 21 SREGym-Lite faults used in our previous diagnosis post. Each fault had five attempts with review and five without: 105 diagnoses per setup, 420 in total. GPT-6 Sol at high reasoning effort reviews the root cause and writes the report. Without review, it only writes the report.

We scored every diagnosis with GPT-6 Astra at high reasoning effort. A diagnosis passes if its SREGym composite score is at least 0.70. End-to-end diagnosis times below are means over all attempts, excluding the fixed 120-second observation wait and judging.

What does LLM review add?

With review, Jev passes 12 more diagnoses and GPT-6 Luna Decisions passes 17 more. Mean diagnosis time increases by 6.6 seconds for Jev and 4.6 seconds for Luna. Estimated cost increases by about 1.6 cents per diagnosis in both setups.

Most model time and inference cost go to the LLM, not to the decision model. With review, decision-model calls take a total of 1.67 seconds per diagnosis for Jev and 3.13 seconds for Luna, on average.

Where does review help?

Expand figure

FAULT	LUNA REVIEW	LUNA NO REVIEW	JEV REVIEW	JEV NO REVIEW
admission_webhook_outage	5/5	5/5	5/5	5/5
cronjob_sidecar_blocks_completion	4/5	4/5	5/5	4/5
duplicate_pvc_mounts	5/5	1/5	5/5	1/5
edge_request_filter_cpu_saturation	4/5	5/5	4/5	4/5
env_variable_shadowing	5/5	5/5	4/5	5/5
finalizer_deadlock_controller	5/5	5/5	0/5	1/5
internal_traffic_policy_local	5/5	5/5	5/5	3/5
kafka_poison_pill_hol_block	5/5	4/5	5/5	5/5
mutating_webhook_resource_limits	4/5	1/5	5/5	4/5
namespace_memory_limit	5/5	5/5	5/5	5/5
network_policy_block	5/5	5/5	5/5	5/5
readiness_probe_misconfiguration	5/5	5/5	5/5	5/5
rolling_update_misconfigured	5/5	4/5	5/5	5/5
search_rate_retry_collapse	5/5	0/5	4/5	0/5
secret_rotation_stale_env_credentials	5/5	0/5	5/5	5/5
service_dns_resolution_failure	5/5	5/5	5/5	5/5
service_wrong_pod_selection	5/5	5/5	5/5	5/5
unschedulable_incorrect_port_assignment	5/5	5/5	5/5	5/5
valkey_auth_disruption	3/5	4/5	2/5	2/5
wrong_dns_policy	5/5	5/5	5/5	3/5
wrong_service_selector	5/5	5/5	5/5	5/5

On the duplicate PVC mounts fault, both decision models pass 5/5 attempts with review, compared with 1/5 without it. On search-rate retry collapse, neither passes an attempt without review. With review, Luna passes 5/5 and Jev passes 4/5.

Review does not help on every fault. On the cleanup-controller fault, finalizer_deadlock_controller, Jev with review passes 0/5 attempts, while Luna passes 5/5 with or without review. On valkey_auth_disruption, Jev passes 2/5 in both setups. Luna passes 3/5 with review and 4/5 without it.

In one failed cleanup-controller run, the reviewer replaced cluster_object with deployment/cleanup-controller. The report described repeated HTTP 403 denials from the Kubernetes API, but missed the cause: a ClusterRole lacked permission to patch ConfigMaps. Naming the affected Deployment was not enough to explain the failure.

How does this compare with LLM agents?

With review, Luna matches the diagnosis pass rate of Codex with GPT-5.6 Sol (max) at roughly 15× lower estimated cost per diagnosis. Jev comes close to Claude Code with Claude Opus 5 at roughly 28× lower estimated cost.

With review, mean diagnosis time is 41.5 seconds for Luna and 38.4 seconds for Jev. Codex with GPT-5.6 Sol (max) averaged 211 seconds per diagnosis, while Claude Code with Claude Opus 5 averaged 210.1 seconds.

Does the choice of LLM matter?

To test the choice of LLM, we replaced GPT-6 Sol with GPT-5.6 Sol at high reasoning effort and GPT-5.6 Luna at xhigh effort. For both GPT-5.6 models, we kept the recorded investigations and reran report writing with and without root-cause review. The table includes GPT-6 Sol's results from the separate runs in our main experiment. The same judge scored every diagnosis.

Expand figure

LLM review and report writing	GPT-6 Luna Decisions decision model	Jev decision model
WITH REVIEW
GPT-6 Sol, high effort	100/105 (95.2%) mean score 92.1	94/105 (89.5%) mean score 89.7
GPT-5.6 Sol, high effort	91/105 (86.7%) mean score 85.3	84/105 (80.0%) mean score 85.0
GPT-5.6 Luna, xhigh effort	88/105 (83.8%) mean score 83.9	86/105 (81.9%) mean score 83.2
WITHOUT REVIEW
GPT-6 Sol, high effort	83/105 (79.0%) mean score 80.0	82/105 (78.1%) mean score 82.4
GPT-5.6 Sol, high effort	80/105 (76.2%) mean score 79.7	75/105 (71.4%) mean score 76.0
GPT-5.6 Luna, xhigh effort	81/105 (77.1%) mean score 80.1	78/105 (74.3%) mean score 80.0

Review improved the results with every LLM. Each GPT-5.6 model passed 7 to 11 more diagnoses per decision model with review than with report writing alone. With review, GPT-6 Sol passed 8 to 12 more diagnoses than either GPT-5.6 model. Of the three LLMs tested, it produced the most passing diagnoses from the same recorded investigations.

Can clearer questions improve the decisions?

For each run where review changed the root cause, we compared two reports. One was the submitted report using the LLM's reviewed choice. The other was generated afterward using the decision model's original choice. Both used evidence from the same saved investigation, rather than the separate runs with and without review compared above.

Review corrected 14 of Luna's 105 diagnoses and 13 of Jev's, changing them from failing to passing. Only one changed from passing to failing. We scored each report twice and counted a correction only when the original failed both times and the reviewed report passed both times.

For each corrected case, we saved the root-cause question, its evidence and answer choices, and the LLM's corrected answer. We then tested if clearer wording or answer choices would help the decision model select that cause using the same evidence.

We tried two changes:

• Remove a vague answer. We removed the catch-all cluster_object option when the evidence pointed to specific objects, as in the DNS example.
• Compare two candidates directly. We asked a follow-up question containing only the original choice and another candidate supported by the evidence. We asked it twice, reversing the answer order, and changed the choice only if the model selected the alternative both times.

With these changes, the decision model selected the LLM's corrected root cause in about a third of the corrected cases. These tests reused saved questions and evidence without rerunning the investigation. They show that the model's choice can improve, not that the full agent would pass more diagnoses.

Conclusion

With LLM review, the agent passes 89.5% of diagnoses with Jev and 95.2% with GPT-6 Luna Decisions. The decision model guides the investigation, while the LLM checks the proposed cause and writes the report. The LLM can improve a diagnosis without choosing every tool the agent runs.

Next, we plan to build these question changes into the agent and measure whether they improve diagnosis results in live runs. We also want to collect evidence that makes related causes easier to distinguish. The reviewed cases could eventually help train the decision model to make better choices with less need for LLM review.