Severity stratification changes what hallucination rates mean: An item-matched audit of five large language models in orthodontic decision support

Basit Öğe Kaydı
dc.authorid0009-0005-3767-7415
dc.authorid0009-0008-8331-9493
dc.contributor.authorAllaf, Ferdi
dc.contributor.authorÖzcan, Mustafa
dc.date.accessioned2026-11-13T13:51:35Z
dc.date.available2026-11-13T13:51:35Z
dc.date.issued2026
dc.departmentFakülteler, Diş Hekimliği Fakültesi, Klinik Bilimler Bölümü, Ortodonti Ana Bilim Dalı
dc.description.abstractBackground/Objectives: Large language models (LLMs) are consulted for clinical decision support, and their reliability is judged by hallucination prevalence—a measure treating every unsupported element as equivalent, although a fabricated citation and a fabricated protocol differ in what a clinician acting on them would do. We tested whether prevalence tracks clinical risk. Methods: Five LLMs answered a 100-item orthodontic benchmark validated by three external orthodontists (content validity index 0.923). The reference standard was fixed by construction for the 45 items naming a non-existent entity, so any substantive elaboration is unsupported by design; citations were adjudicated against PubMed and CrossRef. Responses were coded with a seven-category taxonomy and assigned to clinical, operational, or epistemic severity tiers in a post hoc exploratory stratification, pre-specified rather than prospectively registered. Because all models answered the same items, comparisons used Cochran’s Q with pairwise McNemar tests and generalised estimating equations clustered on item. Results: Of 500 responses, 449 (89.8%) contained a hallucination but only 65 (13.0%; 95% CI 10.3–16.2) were clinically consequential: prevalence was roughly sevenfold greater than the rate of clinically consequential output as the authors defined it. Between-model differences were large for undifferentiated prevalence (73–100%; Q = 50.51, p < 0.001) and contracted at the clinical tier (10–16%; Q = 10.00, p = 0.040), where no pairwise contrast survived adjustment. Item-level clustering was far stronger for clinically consequential output than for undifferentiated prevalence (intra-class correlation 0.83 versus 0.07). Conclusions: Prevalence and clinically consequential error are not interchangeable and rank models differently. The stratification is exploratory and author-defined, and the benchmark stress-tests susceptibility to fabricated premises rather than surveying natural use.
dc.identifier.citationÖzcan, M., & Allaf, F. (2026). Severity stratification changes what hallucination rates mean: An item-matched audit of five large language models in orthodontic decision support. Diagnostics, 16(19), pp. 1-20. https://doi.org/10.3390/diagnostics16193180
dc.identifier.doi10.3390/diagnostics16193180
dc.identifier.endpage20
dc.identifier.issn2075-4418
dc.identifier.issue19
dc.identifier.pmidPMID: 42882334
dc.identifier.scopusqualityQ1
dc.identifier.startpage1
dc.identifier.urihttps://doi.org/10.3390/diagnostics16193180
dc.identifier.urihttps://hdl.handle.net/20.500.13055/1722
dc.identifier.volume16
dc.identifier.wosWOS:001904093600001
dc.identifier.wosqualityQ1
dc.indekslendigikaynakWeb of Science
dc.indekslendigikaynakScopus
dc.indekslendigikaynakPubMed
dc.indekslendigikaynak.otherSCI-E - Science Citation Index Expanded
dc.institutionauthorAllaf, Ferdi
dc.institutionauthorÖzcan, Mustafa
dc.institutionauthorid0009-0005-3767-7415
dc.institutionauthorid0009-0008-8331-9493
dc.language.isoen
dc.publisherMDPI Publishing
dc.relation.ispartofDiagnostics
dc.relation.publicationcategoryMakale - Uluslararası Hakemli Dergi - Kurum Öğretim Elemanı
dc.rightsinfo:eu-repo/semantics/openAccess
dc.subjectArtificial Intelligence
dc.subjectLarge Language Models
dc.subjectOrthodontics
dc.subjectPatient Safety
dc.subjectClinical Decision Support
dc.subjectHealth Information Quality
dc.titleSeverity stratification changes what hallucination rates mean: An item-matched audit of five large language models in orthodontic decision support
dc.typeArticle
dspace.entity.typePublication

Dosyalar

Orijinal paket
Listeleniyor 1 - 1 / 1
Küçük Resim
İsim:
Tam Metin / Full Text.pdf
Boyut:
1.22 MB
Biçim:
Adobe Portable Document Format
İndir
Lisans paketi
Listeleniyor 1 - 1 / 1
Kapalı Erişim
İsim:
license.txt
Boyut:
1.17 KB
Biçim:
Item-specific license agreed upon to submission
Açıklama:
İndir

Koleksiyon

Ortodonti Ana Bilim Dalı Koleksiyonu
PubMed İndeksli Yayın Koleksiyonu
WoS İndeksli Yayınlar Koleksiyonu