As immune checkpoint inhibitors become part of treatment across an expanding range of cancers and disease stages, immune-related adverse events (irAEs) have become an increasingly important challenge for oncology care. These toxicities can involve almost any organ system, vary substantially in severity, and may emerge during treatment or after therapy has ended. Yet much of the information needed to recognize and characterize them remains embedded in unstructured clinical notes rather than readily analyzable electronic health record (EHR) fields.
A new study published in Nature Cancer asks whether agentic artificial intelligence could help address this problem. In “An agentic AI system enhances clinical detection of immune-related adverse events in a multi-phase validation study,” Jack Gallifant and colleagues developed a large language model (LLM)-based system capable of extracting not only whether an irAE was documented, but also whether it was current or historical, its severity grade, its attribution to immune checkpoint inhibitor therapy, and the clinician’s certainty about that attribution. The system was evaluated through retrospective testing, prospective silent deployment, and a randomized study examining how AI assistance affected human reviewers.
The findings suggest that agentic AI could make irAE curation faster and considerably more consistent, but they also expose an important limitation: recognizing that a toxicity exists is easier than determining exactly how severe it is. The study therefore raises a clinically relevant question for immuno-oncology: can AI improve immune-toxicity surveillance without replacing the clinical judgment required to interpret complex adverse events?
Why Are irAEs Difficult to Detect Systematically?
Immune checkpoint inhibitors have transformed cancer treatment, but their activity comes with a distinct spectrum of immune-mediated toxicities. The authors note that irAEs affect up to 40% of patients receiving checkpoint inhibitors and can involve virtually any organ system. Timely recognition can be critical, particularly for potentially life-threatening events such as myocarditis and pneumonitis.
The challenge is not simply that these toxicities occur. It is that much of the evidence needed to identify them exists within free-text clinical documentation. Structured EHR fields do not reliably capture irAEs, particularly events that do not result in hospitalization, and even ICD codes have limited sensitivity for hospitalization-associated events. Manual chart review therefore remains important for clinical trials, pharmacovigilance, and real-world evidence generation, but it is time-consuming and can vary between reviewers.
Previous natural language processing and LLM approaches have attempted to automate adverse-event detection, but many focused primarily on a binary question—whether an irAE was present or absent—at the patient or encounter level. Clinical interpretation requires considerably more information. Reviewers need to know whether the event is active or historical, how severe it is according to CTCAE criteria, whether the clinician attributes it to immunotherapy, and how certain that attribution is.
Gallifant and colleagues designed their system around this more complex task.
How Did the Agentic AI System Work?
Rather than asking a single LLM to interpret an entire note in one step, the investigators developed an agentic architecture that decomposed irAE assessment into specialized tasks. The system evaluated six clinically important toxicities: pneumonitis, hepatitis, colitis, myocarditis, thyroiditis, and dermatitis, selected to represent common and potentially severe irAEs.
Clinical notes were first routed to condition-specific extraction pipelines. The system then proceeded through sequential stages for event detection, CTCAE severity grading, attribution to checkpoint inhibitor therapy, and certainty assessment. Detection and grading were each performed through three independent model runs, followed by adjudication from a judge agent when the outputs differed. Attribution and certainty were assessed through single reasoning passes. The final output included structured labels together with supporting text extracted from the clinical note, allowing human reviewers to see the evidence underlying the AI-generated assessment.
This evidence-linking component is particularly important in a clinical setting. Instead of providing only a classification, the system showed reviewers which portions of the note supported its conclusion, allowing the output to function as an auditable suggestion rather than an opaque decision.
The investigators evaluated the system in three phases: retrospective development and testing, prospective silent deployment in a real clinical environment, and a randomized crossover study testing whether AI assistance improved human irAE curation.
Phase 1: Can AI Detect irAEs From Clinical Notes?
The retrospective dataset contained 289 expert-annotated clinical notes, with 26 used for development and 263 forming the held-out test set. Several proprietary and open-source LLMs were evaluated using agentic and non-agentic configurations, with the best overall performance achieved by GPT-4.1-mini within the agentic architecture.
For note-level irAE detection, performance was strong. The system achieved a temporality-agnostic macro-averaged F1 score of 0.93, with an F1 of 0.92 for current events and 0.91 for past events. Performance varied somewhat across toxicities: current-event detection reached an F1 of 0.96 for hepatitis and pneumonitis and 0.86 for myocarditis.
The system also performed well when extracting clinician-documented attribution, reaching a binary F1 of 0.92 for current events and 0.94 for past events. Binary certainty classification similarly reached F1 values of 0.93–0.94.
Severity grading proved substantially more difficult.
Exact multiclass CTCAE grading produced an overall macro F1 of 0.66 for current events. Performance was highest for dermatitis and thyroiditis, with F1 values of 0.73 and 0.72, respectively, and lowest for colitis at 0.51. Most errors occurred in intermediate grades 2 and 3, while performance improved when severity was simplified into clinically relevant thresholds such as grade <2 versus ≥2 or grade <3 versus ≥3.
The difference between detection and grading is clinically informative. Identifying documentation consistent with pneumonitis or colitis may be relatively straightforward, but assigning a CTCAE grade requires understanding how symptoms affect function, whether hospitalization or intervention is attributable to the toxicity, and whether the clinical note contains sufficient quantitative information to apply grading criteria correctly.
Why Did Multiple AI Runs Improve Performance?
One of the study’s more interesting methodological findings involved self-consistency.
For the detection and grading stages, the same note was processed through three independent inference runs before a judge agent reconciled disagreements. When this self-consistency mechanism was removed, detection F1 fell from 0.92 to 0.78, a reduction of 0.14. The trade-off was computational cost: the single-inference version cost approximately US$0.011 per note compared with US$0.021 for the full configuration.
This suggests that some of the value of the agentic architecture comes not only from decomposing the task but also from allowing multiple model outputs to challenge or confirm one another before a final classification is produced.
At the same time, the study produced an important reminder that more complex AI is not automatically necessary for every task. A customized rule-based system containing 779 terms performed reasonably well for simple binary irAE detection and was comparable to or better than some LLM configurations. Its limitation was that keyword matching could not reliably extract the richer clinical information required for severity, attribution, certainty, and supporting evidence.
The potential advantage of agentic AI therefore appears strongest when the task moves beyond identifying a keyword toward reconstructing clinically meaningful context.
Phase 2: What Happened When the System Entered Real Clinical Documentation?
High retrospective performance does not necessarily translate into real-world performance, and the prospective phase provides one of the most important findings of the study.
The investigators deployed the fixed system in silent mode for three months between May and July 2025. New clinical documentation from patients who had received an immune checkpoint inhibitor within the preceding year was processed automatically, including physician and nursing notes, radiology reports, consultations, emergency department records, and discharge summaries. In total, the system evaluated 884 clinical notes and generated 31,824 label adjudications.
Performance declined compared with retrospective testing. Across grade, attribution, and certainty labels, binary F1 was 0.72 for current events and 0.79 for past events. At the note level, model predictions matched human review in 62% of cases for grade, 87% for attribution, and 83% for certainty.
This decline is important because it illustrates the deployment gap that can occur when clinical AI moves from a curated retrospective dataset into changing real-world documentation. The prospective notes came from a broader case mix and were generated years later using evolving EHR templates, clinical terminology, note structures, and documentation practices. The authors therefore argue that clinical AI systems will require continuous monitoring and periodic recalibration rather than one-time validation before deployment.
For immunotherapy toxicity surveillance, this is particularly relevant because both treatment regimens and toxicity patterns continue to evolve.
Phase 3: Does AI Actually Help Human Reviewers?
The third phase moves the study beyond algorithmic performance and asks a more practical question: does AI make people better at the task?
The investigators conducted a randomized 2 × 2 crossover study comparing standard manual annotation with AI-assisted review. Seventeen participants—16 clinical research coordinators and one clinical research nurse—completed the study, producing 316 evaluable timing observations.
AI assistance substantially reduced the time required to review a clinical note. Median annotation time decreased from 428 seconds with manual review to 242 seconds with AI assistance, corresponding to a 43.5% reduction. In the primary analysis, AI assistance saved an estimated 2.71 minutes per note.
The improvement was not limited to speed. Complete-match accuracy, a stringent endpoint requiring all six current irAE grades in a note to match the gold standard, increased from 19.4% with manual annotation to 25.9% with AI assistance. The odds of complete accuracy were 45% higher with AI support (OR 1.45; 95% CI 1.01–2.08; P=0.045).
Perhaps the most striking change involved consistency between reviewers. For current-grade labels, Krippendorff’s α increased from 0.22 to 0.85 in one note set and from 0.51 to 0.82 in the other. Similar improvements were observed for attribution and certainty.
These findings suggest that one of the most useful roles of AI may not be replacing human assessment but standardizing it. When reviewers were presented with structured model suggestions and highlighted supporting evidence, their interpretations became substantially more consistent.
Why Consistency May Matter as Much as Speed
Variability in adverse-event assessment is a significant problem for clinical trials, registries, pharmacovigilance, and real-world evidence. Two reviewers reading the same clinical note may differ in whether they consider an event immune-related, how they grade it, or how they interpret the treating clinician’s certainty.
The randomized study suggests that AI-generated suggestions can provide a common reference point and reduce this variability. Reviewers no longer need to generate every interpretation from the beginning; instead, they can verify, correct, or reject a structured assessment linked to specific evidence in the note. The authors describe this as shifting reviewer effort from generation toward verification.
There is, however, a potential downside. The same mechanism that improves agreement could produce automation complacency if reviewers become overly reliant on model suggestions. A consistently wrong AI output could theoretically make multiple reviewers converge on the same incorrect interpretation. The study was not designed to determine whether the observed improvement in consistency partly reflected inappropriate anchoring on model outputs, and the authors identify this as an important area for future investigation.
Human verification therefore remains central to the proposed workflow.
Why Is CTCAE Grading Still Difficult?
The relatively modest F1 of 0.66 for exact severity grading highlights one of the most clinically relevant limitations of the system.
CTCAE grading often depends on information that extends beyond the diagnosis itself. A patient with immune-related hypothyroidism requiring thyroid replacement may meet criteria for grade 2 toxicity, but if the same patient is hospitalized, the reviewer must determine whether the hypothyroidism caused or prolonged that hospitalization before considering a higher grade. Similarly, grading colitis may depend on quantitative stool frequency, while hepatitis grading may require laboratory values and trends that are incompletely described in a single note.
Some grading errors may therefore reflect incomplete documentation rather than purely incorrect AI reasoning. Human reviewers face the same problem when relevant clinical details are absent.
The authors propose that future systems could incorporate structured information—including laboratory results, corticosteroid initiation, medications, imaging findings, and longitudinal information across multiple notes—to improve grading. Such integration may be particularly important if AI-based irAE systems eventually move from retrospective data curation toward real-time clinical surveillance.
From Retrospective Curation to Real-Time irAE Surveillance?
The potential clinical application of this technology extends beyond accelerating research annotation.
A system capable of continuously processing new oncology documentation could potentially identify candidate irAEs shortly after they are documented, distinguish active events from historical toxicities, estimate severity, and highlight the specific evidence supporting the classification. This could be particularly relevant for high-risk toxicities such as myocarditis or pneumonitis, where delayed recognition may have serious consequences.
However, the current study does not demonstrate that the system improves clinical outcomes.
The prospective phase operated silently and did not change patient management. The investigators did not evaluate whether AI assistance shortened the time to irAE diagnosis, accelerated corticosteroid initiation, reduced hospitalization, prevented treatment interruption, or improved survival. These are essential distinctions between showing that AI can extract toxicity information from documentation and demonstrating that it improves patient care.
The next step is therefore not simply broader technical validation but prospective evaluation of whether AI-supported surveillance changes clinically meaningful outcomes.
Cost and Scalability
The best-performing configuration used GPT-4.1-mini and cost approximately US$0.021 per clinical note. Applied to the 884-note prospective cohort, the estimated inference cost was approximately US$18.50.
The relatively low per-note inference cost suggests that large-scale processing could be technically feasible, but computational price is only one component of implementation. Healthcare systems would also need secure infrastructure, EHR integration, governance, monitoring, human oversight, and mechanisms for recalibrating the system when documentation patterns or treatment practices change.
The best-performing system also relied on a proprietary model accessed through a HIPAA-compliant institutional API. Open-source models could eliminate external API costs and potentially allow local deployment behind institutional firewalls, but the open-source alternatives evaluated in this study performed less well with the tested prompting strategies.
Scalability will therefore depend not only on model performance but also on whether comparable results can be reproduced across institutions with different technical resources.
Limitations
The most important limitation is generalizability. All three study phases were conducted within the same healthcare system, meaning that even the prospective deployment does not constitute external validation. Clinical documentation differs substantially across institutions, EHR platforms, oncology specialties, and local practice patterns, so performance at other centers remains unknown.
The system also evaluated only six irAEs rather than the full spectrum of checkpoint inhibitor toxicities. Full validation relied on one proprietary model, and performance cannot be assumed to transfer directly to other LLMs. Ground-truth labels were themselves dependent on documentation quality and on inherently subjective judgments surrounding severity and attribution.
The randomized human–AI study was also relatively small, with 17 participants, predominantly clinical research coordinators rather than treating oncologists. Its findings therefore establish an effect on a specific annotation workflow rather than demonstrating that the same benefits will occur when physicians use the system for real-time clinical decisions.
Most importantly, no phase of the study evaluated patient outcomes. The evidence supports improved information extraction and human curation, not improved toxicity management or safer immunotherapy delivery.
