From Checkup Reports to Dental Panoramic X-Rays and ECGs: Building a Verifiable Medical AI Benchmark

Summary

As medical AI moves into the multimodal and foundation-model era, the scarce resource is not simply a collection of files. It is a dataset and evaluation protocol that can establish where a model is reliable: for which task and population, on which equipment, and within which clinical workflow. The supplied materials cover three distinct modalities: 11 health-check PDFs totaling 426 pages; 10 DICOM studies whose headers identify digital dental panoramic X-rays (Modality PX, manufacturer field MEYER, primarily 1536 × 3101 pixels); and five one-page ECG PDFs paired with five one-row spreadsheets containing sex, age, and diagnosis fields.

These materials can be used as the starting point for the multi-task medical evaluation platform, but they cannot be directly used as a training set or a benchmark test set. Physical examination reports need to separate free text, structured results, reference intervals, abnormal markers and recommendations; oral imaging requires tooth position systems, lesion instances and image quality labels; electrocardiograms require time and source correspondence between waveforms and diagnostic labels, category definitions and patient-level isolation. The output, truth level and error cost of the three types of tasks are different and should not be mixed into a single accuracy or overall ranking.

This article proposes an evaluation structure of "governance access—task definition—reference standards—layered blind testing—external verification—workflow evaluation—version governance". It is recommended that Langhui start on an independent track and adopt hidden set-out sets, cross-organization and cross-device testing, expert arbitration, and error severity stratification to gradually form an auditable, reproducible, and iterable medical data Benchmark service.

1. Accurately identify the samples first to avoid mistaking file samples for data sets

1.1 Special physical examination: special report and comprehensive report coexist There are 11 PDFs in the

compressed package, with a total of 426 pages. The document covers a variety of special health directions, and there is a significantly longer combined report. This shows that data assets may contain multiple structures such as topic templates, comprehensive reports, cross-section repetitions, and summary/detail pages. Statistics based only on the "number of PDF copies" do not equal the number of patients, examinations or independent task samples; the comprehensive report may contain multiple topic contents, and if duplication is not done at the document level and patient level, the same examination conclusion may be counted repeatedly.

Document pedigree should be established first when constructing: patient anonymous key, examination event, report version, topic category, generated template version, page range, structured field source and parsing confidence. Each extracted field should be able to return to the page number, paragraph or table position, distinguishing original measurements, report conclusions, rule derivation and model generation suggestions. For reference ranges, retain the unit, laboratory/equipment source, applicable gender and age conditions, and the current version of the report. You cannot just save a "high/low" label.

1.2 Oral panoramic image: the nominal name of the attachment is not equal to the real mode

DICOM header information shows 10 independent studies and series, the modality is PX, the image matrix is mainly 1536×3101, and there are also 1200×2326, and the manufacturer field is MEYER. This batch of samples should be understood based on digital dental panoramic radiograph images, rather than designing tasks based on CT volume data. The professional restoration suggestion here is: the project name, interface label, data card and document should all use "dental panoramic radiograph/panoramic film", and only be marked as CT after being confirmed by the imaging department that it is indeed CT. Incorrect modal classification can lead to error preprocessing, error indicators, and procurement acceptance disputes.

In addition, there are non-empty identifying fields such as patient name/ID, date of birth, gender, examination date, and institution in the DICOM header. Even if the sample is only for internal analysis, external dissemination and uploading should be stopped first, and authorization verification, de-identification, and pixel layer text/QR code inspection should be completed; DICOM UID needs to be remapped according to consistent rules, and a controlled mapping table should be retained. Simply changing the file name or removing the name tag is not enough to complete de-identification.

1.3 Electrocardiogram: There are only 5 records in the supporting form, which is a process sample

The attachment contains 5 single-page ECG PDFs and 5 XLSXs. The form has three fields: gender, age, and diagnosis. Each copy has one record. It is suitable for checking "whether the waveform file of a case corresponds to a row of clinical labels." It can also be used to design the data dictionary and deliver the acceptance accordingly. However, this does not constitute a representative arrhythmia cohort and is not sufficient to estimate sensitivity, specificity, subgroup differences, or model generalization. When there are only five cases, adding an erroneous sample changes the proportion significantly; even if a beautiful score is calculated, there is no stable statistical explanation.

The official data should clarify the ECG paper speed, gain, number of leads, sampling rate, equipment, filtering, recording duration, lead dropout and waveform quality; the PDF image needs to confirm whether it can return to the original digital waveform. If the model can only read paper images, pixel reconstruction and time/voltage calibration errors should be entered into the evaluation; if the original waveform is available, the sampling unit, timestamp, lead sequence and signal check value should be saved.

2. The core of medical Benchmark: divided into tracks according to clinical tasks and reference standards

A medical AI benchmark should not start with a single overall-accuracy score, but should set up separate tasks for the three modalities, and then unify the governance, reporting and operating protocols.

Track A: Understanding health-check reports and risk communication

can be divided into: field extraction (check items, results, units, reference intervals, abnormal markers); report structure analysis (topics, abstracts, conclusions, recommendations); rule consistency check (whether results conflict with abnormal markers); risk stratification (according to predefined guidelines or expert rules); summary generation for professionals; explanatory communication for subjects. Each task requires a separate design of truth values: field extraction is based on two-person annotation and original text position linking, medical risk judgment requires guideline versions or expert consensus, and recommendation generation requires an assessment of factual support, applicable conditions, risk omissions, and whether to induce self-diagnosis and treatment.

Track B: Oral Panoramic Image Analysis

can be divided into image quality and readability, tooth/tooth position detection or instance segmentation, periodontal or caries and other clear lesion tasks, anatomical structure localization, and report generation. The tooth position coding system needs to be frozen before the project begins, and missing teeth, impacted teeth, mixed dentition, restorations, overlaps, motion blur, and truncation should all have explicit labels. It is recommended that the detection task report the number of false positives per case and FROC; the segmentation task report Dice, surface distance, and results stratified by tooth position/lesion size; report generation cannot only do text similarity, but requires clinical fact accuracy and omission/overdiagnosis error review.

Track C: ECG waveform interpretation and risk prediction

A distinction should be made between the rhythm/conduction/morphology diagnosis of current ECG and the prospective task of using ECG to predict future disease or outcome. The former reference standard may come from the interpretations of multiple electrocardiologists, clinical reports and necessary examination evidence; the latter must predefine prediction time points, outcome definitions, follow-up windows, competing risks and treatment changes. At a minimum, metrics report macro-average AUROC, AUPRC, sensitivity/specificity, calibration, confidence intervals, and decision thresholds, and interpret positive predictive values ​​by disease prevalence and device source.

Cross-modal comprehensive ability: only controlled sublists

If you want to evaluate whether the multimodal model can combine health-check reports, oral X-rays and ECGs, you need true pairings of the same patient, the same examination event or within a reasonable time window, and authorization for each modality is required. Three unrelated samples cannot be combined into a "multimodal case". Cross-modal tasks require additional detection of time mismatch, information leakage, missing modalities, and whether the model makes inferences on modalities for which there is no evidence. It is recommended to establish three tracks separately first, and then set up a fusion sub-list after meeting the data relationship and sample size requirements.

*Figure 2｜In addition to task performance, the list should also report on safety, generalization, data quality, cost and evidence maturity.*

3. Sample size and evidence level: What can five ECG cases and ten panoramic films show? The value of the

samples is primarily to verify that file reading, field mapping, anonymization, pairing logic, and annotation manuals are executable; they do not demonstrate model clinical performance. Data maturity can be divided into four levels

1. File Availability VerificationThe compressed package is complete, the file can be parsed, DICOM/PDF/XLSX is not damaged, and fields and images can be read correctly.2. marking feasibility pilotA small number of representative samples were independently annotated by two reviewers with adjudication to confirm that there was no ambiguity in terminology, tooth position, ECG label and report field definitions.3. Internal development and testingReach prespecified sample size and isolate by patient, development/validation/blind test sets are mutually exclusive, test set is sealed.4. External and forward-looking verificationValidated across agencies, devices, time, and people to further observe real workflow, security, and decision-making effectiveness.

Each level must indicate the sample unit, inclusion/exclusion, number of patients, number of examinations, number of files and duplicate relationships in the data card. Sample size should be calculated based on the primary endpoint, expected performance, confidence interval precision, class proportions, and cluster structure, rather than working backwards from “how many documents are on hand.”Multiple pages of health-check reports cannot be considered as multiple patients; multiple images and multiple ECGs of the same person cannot be distributed across collections.

4. Data quality engineering: from receipt inspection to auditable ground truth

recommends establishing a six-layer quality pipeline

• file layerZIP list, checksum, file format, number of pages, resolution, parsability, duplicate files and damaged files.
• Identity and Authorization Layersource legality, scope of use, informed consent/ethical approach, privacy impact assessment, revocability/deletion agreement.
• structural layerpatient-examination-report-image-table relationship, timestamp, unit, field dictionary, null value semantics and version.
• clinical content layerLabel body, guide version, inclusion and exclusion standards, reference truth value source, difficult samples and unable to determine status.
• annotation layerExpert background, training calibration, independent annotation, disagreement classification, arbitration, random inspection and version tracking.
• evaluation layerPatient-level segmentation, deduplication audit, blind test access record, running environment, model snapshot, threshold freezing, statistical interval and error analysis.

Special physical examination should connect the "source text position - extracted value - unit - standard coding - model rewriting" of each field to avoid leaving only the final summary. Dental radiographs need to retain available information on DICOM orientation, pixel spacing, exposure, and manufacturer/device, while minimizing identifiable metadata in public versions. The ECG needs to retain the original waveform or clarify the image source, and check the integrity and time/voltage calibration lead by lead. Traceable error codes are used for all exceptions, such as parsing failure, left/right/tooth position uncertainty, mismatch between reports and images, missing device information, and insufficient evidence of ground-truth labels.

5. Ranking statistical design: avoid “high scores but cannot be repeated”

A benchmark should publish protocol and result versions instead of just one ranking list. Each model needs to record the model/weight/API version, prompt words, preprocessing, tool calls, thresholds, random seeds, running date and failed sample processing. Patient-level training, verification, and test isolation should be completed before modeling; multiple reports of the same patient, PDFs and images derived from the same examination, and repeated ECGs must be deduplicated or clustered. Test answers are controlled by an independent custodian, who sets query budgets and repeat submission audits.

Confidence intervals, stratified performance, calibration and failure spectra should be provided in addition to the main indicator. Use patient-level indicators and lesion-level indicators side by side for images to avoid pseudo-large samples caused by a single tooth or pixel number; consider the statistical correlation of repeated records of the same patient for ECG; perform a patient-level cluster bootstrap for health-check reports by patient rather than by field/page. Macro-averaging and micro-averaging are to be interpreted simultaneously, and AUPRC, per-case false positives and threshold performance are provided when classes are unbalanced.

A "score-risk" mapping needs to be established: missed diagnoses that endanger security, incorrect recommendations, unfounded extensions, and privacy leaks should be presented separately and cannot be offset by high scores for format or wording. Expert review can use anonymous random order, double review and arbitration; the LLM scorer is only used for large-scale pre-screening or consistency assistance, and must be calibrated on the manual annotation gold standard, and its preference for language style, answer length and model brand must be monitored.

6. Methodological inspirations from cutting-edge research

In recent years, ECG basic model and benchmark research has begun to emphasize external data sets, cross-device generalization, label quality and clinical diagnosis level. The CODE-II study reported migration evaluations on external benchmarks such as PTB-XL and CPSC, and pointed out that label quality and clinically meaningful diagnostic systems are as critical as data scale; studies such as ECG-CLIP extended the evaluation to disease identification, future risk and outcome prediction, indicating that "interpreting current waveforms" and "predicting future health" must be separate topics. The work of "Nature" on EchoNext in 2025 demonstrated the design value of multi-hospital evaluation and prospective detection studies, suggesting that static retrospective AUC is not enough to answer whether undiagnosed diseases can be found in clinical scenarios.

In terms of oral imaging, the panoramic visual language model research published in "Nature Biomedical Engineering" in 2026 combined instance-level dental information with text reports, and set up expert preference evaluation; this supports the task line of "image structured positioning + report semantics", but it does not mean that any ten panoramic photos can reproduce paper-level conclusions. Benchmark needs to disclose the specific applicable population, imaging equipment, annotation level and external testing center to prevent the model from obtaining shortcut scores on a single device or template background.

For physical examination and preventive health scenarios, evaluation should shift from static medical Q&A to a complete chain: whether the model reads correctly, whether it can be combined with applicable reference intervals, whether it identifies abnormalities that require professional review, whether it provides clear but not excessive communication, and whether it recommends appropriate next steps without replacing the doctor's judgment. Health conversation benchmarks such as HealthBench emphasize evaluation based on real health interactions and refined scoring criteria; when transferring its ideas to health-check reports, original report evidence backlinks, unit/reference range verification, abnormal omission severity, and clinical review referrals should be further added.

7. Langhui’s suggestions for list construction: Use “verifiable evidence” to form differences

Langhuike positions the platform as a professional data, expert production and continuous evaluation service for the medical and life sciences AI team, rather than an unverified "model ranking agency". In the first phase, it is recommended to build three private evaluation tracks: health-check report understanding, dental panoramic image structured analysis, and ECG interpretation and prediction. Each track delivers data cards, labeling SOPs, reference standard descriptions, blind test protocols, baseline results, error stratification and version reports. Sensitive data are evaluated in hospitals or controlled environments as much as possible, and only authorized aggregate statistics and methodological information are released in public reports.

The business leader defines the purpose of each track, clinical/biological experts define ground-truth labels ​​and risks, data engineers maintain pedigrees and verifications, the algorithm team freezes the running protocol, and independent quality control personnel manage blind testing and reproduction. The advantages of Langhui should be reflected in demand clarification, expert matching, labeling consistency, cross-modal data management, blind testing and problem closure; any case size, doctor qualifications, model effect or clinical impact figures must have public evidence before being used for market dissemination.

8. Recommended 90-day implementation route

Week 1-2: Data and authorization inventory. Establish file-level lists and hashes; perform privacy risk assessment and de-identification verification on oral DICOM; confirm the source, template version, duplicate patients and output purpose of the health-check report; verify whether the ECG PDF and XLSX correspond to the same case, and clarify whether it can be used for research.

Weeks 3-5: Tasks and label design. Define inputs, outputs, target populations, exclusions, reference standards, error severity, expert roles, and scorecards for each of the three tracks. Physical examination fields are marked with examples; dental films freeze the tooth position system and image quality labels; ECG determines the diagnostic classification and evidence level.

Weeks 6-9: Small-scale dual-standard pilot. Use approved samples to test standards, measure consistency and arbitration rate; revise rules and freeze versions. Currently, ECG five cases and panoramic ten cases are only used for process test runs, and performance rankings cannot be released.

Weeks 10-13: Capacity expansion and blind test preparation. Calculate the sample size based on the primary endpoint, build multi-center/equipment sources; partition by patient level; seal the test set; complete the baseline model, external testing protocol, privacy audit and data card. The first version of track results will be released only after the sample size and governance thresholds are reached.

Conclusion: Don’t let the sample size create the “illusion of the list”

Each of the three compressed packages provides an entrance worthy of continued construction: the health-check report can be developed into a report understanding and health communication evaluation, the oral panoramic DICOM can be developed into a benchmark for tooth position and lesion identification, and the electrocardiogram pairing table can be used as a sample waveform-label data contract. However, current sample size, metadata, and annotation evidence are not sufficient to support clinical performance claims or public model rankings. A more responsible path is to first verify the source and identity risks, then solidify the tasks, reference standards, segmentation, blind testing and statistical plans, and finally use cross-organization, cross-device and prospective evidence to test whether the model is truly usable.

Langhui’s professional value does not lie in packaging limited samples into a “big list”, but in helping hospitals, research teams and algorithm teams turn every step into an evidence chain that is traceable, reviewable and capable of continuous improvement. This can not only increase the value of training data, but also make the benchmark results fairer, more explainable, and closer to real business decisions.

References

These studies inform the methodological discussion; their results do not establish model performance on the supplied samples.

1. CODE-II: a large-scale dataset for artificial intelligence in ECG analysis. npj Digital Medicine. 2026.
2. Development and external validation of a contrastive learning foundation model for ECG-based prediction of cardiovascular diseases and outcomes. 2026.
3. Detecting structural heart disease from electrocardiograms using AI. Nature. 2025.
4. Bridging the gap from clinical to home ECG: quantifying and overcoming accuracy loss in AI-enabled single-lead ECG models. npj Digital Medicine. 2026.
5. Towards clinical-level interpretation of dental panoramic radiography using an instance-guided vision-language model. Nature Biomedical Engineering. 2026.
6. Towards Better Dental AI: A Multimodal Benchmark. NeurIPS Datasets and Benchmarks. 2025.
7. HealthBench: an evaluation for AI systems and human health. OpenAI. 2025.
8. Checklist for Artificial Intelligence in Medical Imaging (CLAIM): 2024 Update. Radiology: Artificial Intelligence. 2024;6(4):e240300.

Scope and limitationsThis article is a research design and data engineering analysis, not a diagnostic recommendation, medical device evaluation, or legal opinion. Personal information in the attachment should be processed in an authorized environment; data source, authorization, ethics/compliance and de-identification reviews must be completed before public publication.