Abstract Background: Public radiology report corpora are frequently reused in artificial intelligence research, but report-section completeness, redaction artifacts and near-duplicate text may affect the construction and interpretation of generative AI benchmarks. Objective: To quantify data-readiness limitations and selection effects in a public chest radiograph report collection using a reproducible, text-only audit. Methods: We audited 3,955 XML records from the Indiana University/National Library of Medicine chest X-ray collection. Eligibility required nonempty Findings and Impression sections, minimum lengths of 10 and two words, absence of XXXX redaction tokens, and no duplicated Findings after normalized exact matching and within-metadata-stratum character n-gram near-duplicate detection (cosine ≥0.99). We compared retained proportions for records coded normal and non-normal by MeSH metadata, with a retention risk ratio and 95% confidence interval. We also measured source-section lengths, lexical similarity, and simple negation and uncertainty cues; these are descriptive markers, not diagnostic errors. Results: Of 3,955 reports, 1,237 (31.3%) met strict criteria. Exclusions comprised 536 with incomplete sections, 21 below minimum length, 1,606 with redaction placeholders, 553 with repeated Findings and two near-duplicates. The retention proportion was 25.3% among normal-coded and 34.5% among non-normal-coded records (risk ratio 0.73; 95% CI 0.66–0.81). Among retained reports, median Findings and Impression lengths were 27 and five words, respectively; median paired lexical cosine was 0.027. Conclusion: Strict benchmark construction materially reduced usable reports and altered case-metadata composition. Transparent exclusion logs, duplicate audits, and explicit separation of lexical from clinically validated fidelity measures are needed before comparing generative AI reporting systems. No language model or clinical diagnostic performance was evaluated.

REFERENCES

[1]. Demner-Fushman D, Kohli MD, Rosenman MB, Shooshan SE, Rodriguez L, Antani S, Thoma GR, McDonald CJ. Preparing a collection of radiology examinations for distribution and retrieval. J Am Med Inform Assoc. 2016;23(2):304–310. doi:10.1093/jamia/ocv080.

[2]. Demner-Fushman D, Antani S, Simpson M, Thoma GR. Design and development of a multimodal biomedical information retrieval system. J Comput Sci Eng. 2012;6(2):168–177. doi:10.5626/JCSE.2012.6.2.168.

[3]. Weiskopf NG, Weng C. Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research. J Am Med Inform Assoc. 2013;20(1):144–151. doi:10.1136/amiajnl-2011-000681.

[4]. Kapoor S, Narayanan A. Leakage and the reproducibility crisis in machine-learning-based science. Patterns. 2023;4(9):100804. doi:10.1016/j.patter.2023.100804.

[5]. Yu F, Endo M, Krishnan R, et al. Evaluating progress in automatic chest X-ray radiology report generation. Patterns. 2023;4(9):100802. doi:10.1016/j.patter.2023.100802. [6].

[6] Smit A, Jain S, Rajpurkar P, Pareek A, Ng AY, Lungren MP. Combining automatic labelers and expert annotations for accurate radiology report labeling using BERT. Proceedings of EMNLP. 2020:1500–1519. doi:10.18653/v1/2020.emnlp main.117.

[7]. Kottlors J, Iuga AI, Bluethgen C, et al. Guidelines for reporting studies on large language models in radiology: an international Delphi expert survey. Radiology. 2026;318(2):e250913. doi:10.1148/radiol.250913.

[8]. Demner-Fushman D, Antani SK, Simpson M, Rahman MM. Combining Text and Visual Features for Biomedical Information Retrieval. Report to the Board of Scientific Counselors. US National Library of Medicine; 2010.

[9]. Huang J, Wittbrodt MT, Teague CN, et al. Efficiency and quality of generative AI–assisted radiograph reporting. JAMA Netw Open. 2025;8(6):e2513921. doi:10.1001/jamanetworkopen.2025.13921.

[10]. Hwang EJ, Lee JH, Lim WH, et al. Clinical validation of a generative artificial intelligence model for chest radiograph reporting: a multicohort study. Radiology. 2025;316(3):e250568. doi:10.1148/radiol.250568.

[11]. Osma AN. From e-commerce to governance: a triple helix scientometric mapping of ASEAN’s digital trade research evolution (2015–2023). J Scientometr Res. 2026;15(1):133–150. doi:10.5530/jscires.20262074.