Abstract: First turn the “case package” into a verifiable evaluation protocol
The difficulty of AI for diabetes complications is not to let the model name a disease from a medical record, but to allow it to stably complete definable tasks in real, cross-time, and complex sources of clinical evidence: read the correct test results, retain the unit and time, distinguish between diagnostic basis and speculation, identify high-risk signals, and clearly stop when the evidence is insufficient. A meaningful benchmark must measure these abilities separately and cannot replace clinical effectiveness with a general "accuracy rate."
This attached compressed package contains 12 cases after structural inventory, covering four categories: diabetic peripheral neuropathy, diabetic nephropathy, diabetic retinopathy, and diabetic foot, with 3 cases in each category; there are 78 PDFs and 251 CSVs, and the CSV totals about 3,413 data records. The number of files and table rows describes the delivery structure and is not equivalent to the number of unique patients or the sample size available for statistical inference. The attachment is suitable as a data dictionary, annotation manual, evaluation interface, arbitration process and feasibility pilot for error classification; it is not sufficient at this stage to provide model rankings, disease prevalence, sensitivity or clinical benefit conclusions.
LumeSage suggested naming the project "LumeSage DiaComp-Bench" as the proposed pilot framework. The name does not imply an existing official list or completed clinical verification. This article provides auditable evaluation instructions, candidate tasks and construction paths; the official list will be released after the governance, sample size, reference standards, multi-center external validation and operating rules have been reviewed.
1. What the attachment tells us and what it has not yet told us
1.1 The four types of diseases are not four labels for one task
The four types of cases point to different clinical evidence chains. The judgment of diabetic nephropathy requires laboratory indicators, duration, comorbidities and possible other causes; retinopathy grading depends on fundus image quality, standardized lesions and grading rules; peripheral neuropathy involves different evidence such as symptoms, sensory examination, protective sensation and neuroelectrophysiology; diabetic foot may include ulcer location, depth, infection, ischemia, neuropathy and healing outcome. They can share data governance and measurement infrastructure, but reference truth values, target populations, error costs, and clinical disposition implications cannot simply be merged. The PDF and CSV description data in the attachment
contain both document description and structured fields. Common examination item codes, names, result values, units, reference ranges and abnormal marks in CSV may also include medical orders, medication, report conclusions or examination findings. PDF can provide contextual, narrative and formatting information. But "there is a certain field in the file" does not automatically mean that the field definition is consistent, the unit is unified, the source is complete, or the label is correct. The source file, page number/column, original value, standardized value, unit, time semantics, conversion rule, missing reason and review status should be recorded for each field during construction.
Currently the attachment does not have a catalog of images delivered as standalone JPEG, PNG, DICOM or fundus/OCT files. Therefore, what can be discussed at this stage is the extraction of evidence from existing reports and forms, document understanding, and limited text tasks when imaging evidence does exist in the report; it cannot be claimed to have the conditions for fundus image grading, OCT interpretation, foot wound image recognition, or cross-modal image fusion evaluation. When images are received in the future, the equipment, imaging protocol, image quality, annotated coordinates, acquisition institution, and scope of authorization also need to be checked.
1.2 The “sample unit” in a small sample must be clearly stated
The independent unit of analysis is the case/patient, not the PDF, CSV, page number, test item, or record row. A case split into dozens of files still only contributes to one independent patient; multiple tests on the same person are also related to each other. Randomly splitting file lines into training and test sets would allow information about the same patient to cross boundaries and introduce serious leakage. If training or development is carried out in the future, patients must be grouped as a whole; in multi-center settings, external validation must be designed by institution or time.
When there are only 3 cases per disease category, a single case represents one-third of the group for that disease category. Even if all 12 cases are correct, it cannot be inferred that clinical sensitivity reaches 100%. At this stage, what can be verified is whether the process is running, whether the fields have sources, whether expert disagreements can be arbitrated, and whether errors are attributable; AUC, disease risk, cross-population generalization, or model pros and cons lists should not be published.
2. Divide the four types of complications into reproducible evaluation tasks
The list must first answer "what is measured", and then "who scores higher". It is recommended to establish a task catalog at two levels: general data understanding tasks and disease-specific tasks. Each item forms a task card with fixed input type, applicable population, target output, evidence source, reference standard, exclusion conditions, scoring function, clinical risk level, version and prohibited uses. The answer must be traceable back to the original evidence; a model that only gives a conclusion but not a chain of evidence should not have the same score as a traceable system.
Track A｜Structured extraction of medical records and examination data
Extracts exam items, results, units, reference intervals, abnormal flags, medications, and diagnoses from PDF/CSV and binds each value to a case, exam event, and time. Indicators not only check whether the fields match, but also need to check the value, unit, sign, interval boundary, date analysis and source positioning respectively."Not seen", "not checked" and "negative" in the report cannot be combined into the same null value; the formula and rounding accuracy must be recorded for unit conversion. It is recommended to report exact matching, field-level precision/recall/F1, combined numerical and unit accuracy, time parsing accuracy, evidence location accuracy, and rejection/missing processing accuracy.
Track B | Diabetic nephropathy: longitudinal risk status and evidence consistency
If the data can be traced back to repeated testing, the measurement model can correctly identify values, units, trends and time intervals around eGFR, urinary albumin/creatinine ratio (UACR), serum creatinine, diabetes duration, blood pressure and medication information. ADA's "2026 Standards for Diagnosis and Treatment of Diabetes" recommends monitoring eGFR and UACR for patients with diabetes, and determining the frequency of reexamination based on CKD risk stratification; this shows that the "time window" and continuous changes themselves are part of the task definition, rather than additional fields[1]. Diagnostic inference based on a single abnormal value cannot replace standardized clinical diagnosis and continuous judgment. If you are doing a prediction task, the prediction starting point, prediction period, outcome, baseline risk, treatment change processing and competing risks must be defined in advance, and this appendix does not have the sample size required to form a reliable risk model verification.
Track C｜Diabetic Retinopathy: Report Comprehension and Fundus Image Testing Separately
It is currently possible to evaluate the extraction of "findings-grading-recommendations" from existing reports, whether different fields are inconsistent with each other, and whether referral recommendations are based on the original text; however, without independent fundus/OCT images, image-level lesion detection, grading, lesion localization, or cross-camera generalization cannot be evaluated. The fundus imaging benchmark should specify the imaging field of view, left and right eyes, image quality, unqualified image processing, grading system and reader arbitration. For example, the multimodal retinal research data published by "Scientific Data" in 2026 contains 24,460 images, which are refined to five levels of severity, seven types of lesions, and OCT macular edema grading, reflecting the requirements of image evaluation for fine-grained labels, modality pairing, and quality control.[2]. A multi-modal large model study observed performance differences between different models and lower severity detection sensitivity in 309 eyes of 188 patients, reminding the leaderboard that the leaderboard cannot only report the overall accuracy, but also needs to show missed diagnoses and grading errors.[3]。
Track D｜Peripheral neuropathy and diabetic foot: evidence stratification, severity and safety
The peripheral neuropathy clerical task checks whether symptoms, examination patterns, protective sensory findings, and their temporal correlation are correctly recorded, and detects whether the model mistakenly writes "not recorded" as "examination normal."The neuropathy and foot care chapters of ADA's "2026 Standards for Diagnosis and Treatment of Diabetes" emphasize neuropathy screening, foot risk assessment and standardized physical examination; therefore, the task card should clarify the examination method, examination frequency and referral threshold to avoid treating missing records as negative[7]. The measurable tasks of diabetic foot can start from the extraction of location, wound size, infection signs, ischemia clues, grading and outcome from medical records; future image tasks should have standard shooting conditions, scale reference, lesion labeling and clinical outcomes. When there are no images, the performance of the written medical record task cannot be used to indicate wound visual ability. Missing a high-risk alert, downgrading an emergency tip to a low-risk one, or making up an ulcer grade out of thin air should all be recorded as serious errors and reported separately rather than offset by correct answers in a large number of simple fields. The
To avoid scores masking clinical differences, the four categories of tracks should not be combined into a single overall ranking. If summary scores must be provided in the future, the original indicators, case composition, confidence intervals, weighting rules and scope of application of each track should be disclosed at the same time; missing modalities and tasks should be displayed as "not evaluated" and cannot be regarded as zero points or passed by default.
3. Reference standards, expert annotations and patient safety
3.1 The reference standard is a traceable evidence system, not a vote
Clinical truth cannot simply copy the last diagnostic field that appears in the medical record. It is necessary to first define the reference evidence level for each task: for example, laboratory tests use qualified laboratory reports and unit/method information; kidney outcomes are constructed according to guideline definitions and specified time windows; fundus grading is based on pre-registered scales, qualified images, and expert readings; foot outcomes clearly define the basis and time for infection, ischemia, healing, or amputation. When a case does not meet the conditions, it should be marked as "unable to determine/insufficient evidence", and labels cannot be forcibly generated. When different evidence conflicts, the cause of the conflict and the final arbiter should be retained instead of overwriting the original data.
recommends that an expert group be established for each disease: the endocrinology department is responsible for the general clinical context of diabetes, and the nephrology, ophthalmology, neurology, and wound/vascular related experts are respectively responsible for the definition of specialized diseases; data engineers, statisticians, and patient safety managers participate in the review of tasks and scoring rules. Develop manuals, positive and negative examples, boundary cases, missing codes and version numbers before labeling; the labelers must be trained and pass the calibration set before entering formal labeling. At least two qualified experts make independent annotations; calculate appropriate consistency indicators and give disagreement categories; and a third senior expert arbitrates. Cohen/Fleiss κ, weighted κ, intraclass correlation or agreement rate should be selected according to the variable type, and one consistency statistic cannot be mechanically applied to all labels.
3.2 Treat “can refuse to answer” as a reliable ability
Healthcare systems should not get high marks just because they have high response rates. The task needs to set up situations such as inability to judge, insufficient image quality, missing key tests, label conflicts, and beyond the applicable population, and evaluate whether the model is appropriate to request supplementary information or transfer it to clinical personnel. Scores should report coverage and error rates jointly: reducing coverage may improve accuracy, but if the system only answers easy cases, it must be presented through a risk-coverage curve or a preset coverage threshold; dangerous misses, false reassurances, and unfounded diagnoses should be listed separately for safety events.
3.3 Authorization, ethics and anonymization must precede model benchmarking
The attached file name indicates "desensitized", which is only a source statement and does not constitute independent proof of anonymization or legal use. Automatic text screening of this batch of materials failed to detect common identity information patterns, but it cannot replace page-by-page inspection of original documents, image/OCR inspection, metadata inspection, rare combination re-identification risk assessment, authorization document verification and ethical judgment. Patient-level data needs to establish legal sources, purposes of use, scope of authorization, access personnel, retention periods, deletion and auditing rules; de-identification still does not necessarily mean anonymization. Only sufficient overall statistics will be released for external display, and no text, screenshots or rare combinations that can be traced back to the cases will be released. The training set and private test set implement decentralized access, and model submission and data access leave traces.
4. Scorecard: From correct answers to credible, robust and usable
The following is a suggested evaluation framework, not a calculated score for the attached case or any model. Governance access serves as a one-vote veto threshold: those who fail ethics/authorization, desensitization, labeling basis, leakage check and version information will not participate in list comparison. After passing the admission, the scores are set according to the target tasks of the specific track; modules such as clinical task correctness, evidence traceability, structured data quality, safety and rejection, robustness/cross-domain performance can be set on a 100-point scale. However, the specific weights should be jointly pre-registered by experts, clinical users and statistics teams before the test set is frozen, and verified through sensitivity analysis. Weights should not be adjusted afterwards to pursue a certain model's lead.
| Dimension | Suggested observations | applicable indicator examples | Main risks |
|---|---|---|---|
| Clinical task correctness | Extraction, classification, trend, classification or preset processing threshold | precision/recall, macro-average F1, MAE, weighted κ | Overall indicators cover up a small number of high-risk missed diagnoses |
| Evidence traceable | Conclusion: Can the original page, fields, check time and rules be located? | Evidence location accuracy, value/unit joint matching rate | The answer is correct but the basis is wrong or cannot be audited |
| Security and Uncertainty | Refusal to answer, upgrade, warning signal recognition, hallucination and key omissions | Serious error rate, risk coverage curve, calibration error | Error suggestion is diluted equally |
| Robust and fair | missing, scan quality, equipment/institution, age and other stratification | Stratified differences, confidence intervals, drift indicators | Insufficient sample results in false subgroup conclusions |
| operation can be reproduced | model version, prompt word, inference parameters, date and data hash | Difference in reruns and completeness rate of environmental records | List results cannot be reviewed The |
indicator must match the task. Field extraction reports exact match with field-level precision/recall/F1; continuous values report errors and unit correctness; ordinal severity uses weighted κ or rank confusion matrix; prediction tasks simultaneously give discrimination, calibration, net benefit at clinical thresholds and confidence intervals; open responses use expert-developed atomic fact lists, evidence support, critical omissions and severity scales, and manual consistency verification of automated reviewers. Calibration cannot be replaced by "model self-reported confidence". Predictive studies should follow transparent reporting recommendations such as TRIPOD+AI; multi-institution/cluster data should also report on institutional heterogeneity and corresponding analysis[4]。
This time n=12 is not enough to stably estimate these overall indicators. Appropriate pilot outputs are case-level error lists (only within authorized environments), task definition gaps, annotation disagreements, systematic formatting failures, manual review times, process breakpoints, and scoring script test results. The real model rankings are not released to the outside world. Before entering the formal verification stage, the sample size demonstration should be based on the expected prevalence, sensitivity/specificity targets, allowed confidence interval width, institutional clustering, proportion of lost follow-up and invalid examinations; patients rather than document behavior statistical units, interval estimation should preferably use methods that can reflect the clustering structure.
5. How to avoid leakage, bias and false accuracy in lists
5.1 Hidden test sets must be able to prevent "swiping the rankings"
If all test cases are disclosed, the model may remember the answers in the training corpus, prompt words, or artificial rules. The list should use a public development set and a controlled hidden set in parallel: public task definitions, field specifications, scoring procedures and representative development samples; retain an authorized private test set, limit the frequency of submission, and retain API/container call logs, input and output summaries, model versions and data hashes. The model team needs to declare whether to use test data for training, tuning or manual review. Test set adaptation caused by repeated submissions must be controlled through new batch set aside, time drift sets and rotating hidden questions.
Segmentation should be performed at least by patient; after sufficient data is available, external testing by hospital, equipment, and time can be added. Check for same patient visits across visits, report copies, homologous templates, near-duplicate PDFs, copied text, and image duplications. Rather than “taking a few random samples from more centers,” cross-center validation retains the complete institution as an independent domain and reports the sample composition, indicator intervals, and failure modes for each center. Multi-modal tasks need to ensure that the matching keys of images, reports, tests, and case outcomes are reliable without leaking patient identification.
5.2 Evaluation should focus on clinical use consequences rather than model performance
case answering only verifies offline capabilities; the value of deployment also depends on whether the model can reduce repeated entries, improve data completeness, shorten expert review time, and not increase wrong referrals, missed reports, or workload. Before entering the real workflow, it should be made clear through silent operation, manual-in-the-loop pilots, prospective usability assessments, and necessary clinical research whether doctors can see the model output, when it can be overwritten, how exceptions are escalated, and who reviews errors. Risk grading needs to cover low-risk auxiliary sorting and high-risk triage/diagnosis recommendations, and small sample offline scores cannot be used to replace regulatory, ethical or clinical evidence.
Recent research shows that the performance and transferability of medical imaging AI are closely related to cohort size, reference standards, imaging scenarios, and external data. DeepDKD used a large-scale development cohort and was validated on 10 multi-ethnic data sets in the retinal image inference diabetic nephropathy study, providing a methodological example of "moving from single-center performance to external validation" rather than model conclusions that can be directly transplanted to these 12 cases.[5]. A systematic review of AI in diabetic foot also noted that underreporting of metrics such as calibration undermines individual risk interpretation and decision-making reliability[6]. Therefore, the innovation points of this project should lie in the evaluation design, expert consensus, evidence chain and cross-domain verification capabilities, and should not be packaged as a verified clinical product.
6. From 12-case process pilot to multi-center evaluation platform
- Stage 0｜Access and InventoryConfirm data provider authorization, ethical path, data usage, retention period, patient-level deduplication method and image/document availability range; form data card, missing report, field dictionary, file pedigree and privacy risk list. Cases are not uploaded to external model services until completed.
- Phase 1｜12 Case Methodology Pilot1–2 adjudicable and reviewable tasks selected by clinical experts for each disease; two-person annotation and arbitration; verification annotation workbench, evidence citation, scoring script, blind test process and error severity label. Only process reports are delivered, no rankings or performance estimates are published.
- Stage 2｜Target population and sample size designNegotiate with multiple hospitals for prospective or retrospective cohorts to clarify inclusion criteria, disease spectrum, equipment, time window, outcome follow-up and data authorization. The statistical team will calculate the sample size based on the precision/confidence interval of the primary endpoint and subgroup requirements; "getting together a batch of files" will not be used as the standard for database construction. Carry out inter-center variable mapping, unit standardization and terminology alignment.
- Stage 3 | Official list and hidden verificationreleases the protocol version, model submission rules, pre-registration indicators, list update time, conflict of interest and appeal mechanism; sets up public development sets and controlled hidden sets, reports sub-center and sub-task results, calibration, uncertainty, security events and applicable groups. It is prohibited to replace unevaluated image modalities with text task scores.
- Phase 4｜Clinical Effectiveness and Continuous MonitoringAssess time, quality, safety, and human factors in actual workflows; monitor equipment/population drift, input quality changes, and error escalations after go-live. Every time data, guidelines, model interfaces or scoring scripts are changed, a new version is generated, rollback records and historical lists are retained, and old results are not silently overwritten.
It is recommended that the data description should follow the idea of a reusable data card and state the collection time, source institution category, patient and case counting caliber, inclusion conditions, each task label definition, reference standards, expert roles, deficiencies and biases, permissions and access methods, prohibited scenarios, version differences and known limitations. For cross-institutional clinical data, internationally common standards such as OMOP CDM, LOINC, SNOMED CT, and ICD can be used as mapping references, but the original codes, mapping table versions, and unmappable items must be recorded, and the original clinical semantics cannot be lost for the sake of "standardization."
7. LumeSage’s construction proposition and business closed loop
For hospitals, the disease-specific case package is not a ready-made AI list, but the raw material for starting data productization and evaluation projects. For the medical AI team, being able to answer "Which group of people, which task, which set of labels, which version does this result correspond to, and what error did this result correspond to?" can better support R&D decisions than a total score that is divorced from application boundaries. LumeSage’s differentiated proposition can be built on three verifiable delivery lines: first, experts define tasks and clinical reference standards; second, the engineering team strings unstructured documents, structured tests, time and sources into traceable data; third, the evaluation team forms a closed loop of continuous iterations through blind testing, severity stratification, external verification and version management.
The value of this business chain goes beyond training models: hospitals can use unified labels and quality rules to carry out scientific research, data governance and multi-center cooperation; model companies can discover document parsing, unit errors, subgroup performance degradation and dangerous underreporting before going online; research teams can obtain experimental protocols that can be reviewed; patients benefit from a clearer evidence chain, more appropriate uncertainty expression and safer human-machine collaboration. All values need to be verified through specific project indicators and cannot be inferred solely based on the amount of data or ranking on the leaderboard.
Therefore, it is recommended that this project be initiated as a "DiaComp-Bench research pilot": first create a data card and four disease task catalogs, limiting 12 cases only for expert annotation process and evaluation tool verification; build a multi-center cohort after completing the compliance review; clinical experts, statisticians, algorithm engineers and patient safety personnel jointly freeze reference standards, indicators and hidden sets; after external verification and real workflow evaluation, decide whether to disclose the rankings and their applicable boundaries. This not only respects the current attachment evidence, but also leaves a clear path for the subsequent formation of a high-quality, reproducible, and clinically valuable AI evaluation system for diabetes complications.
References
- [1] American Diabetes Association Professional Practice Committee. 11. Chronic Kidney Disease and Risk Management: Standards of Care in Diabetes—2026. Diabetes Care. 2026;49(Suppl 1):S246–S260. DOI 10.2337/dc26-S011.
- [2] Tang Z, Wang L, Guo Z, et al. A multimodal retinal image dataset for diabetic retinopathy detection using foundation models. Scientific Data. 2026;13:639. DOI 10.1038/s41597-026-07005-9.
- [3] Can Multimodal Large Language Models Diagnose Diabetic Retinopathy from Fundus Photos? A Quantitative Evaluation. Ophthalmology Science. 2026;6(1):100911. DOI 10.1016/j.xops.2025.100911.
- [4] Collins GS, Moons KGM, Dhiman P, et al. TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ. 2024;385:e078378. DOI 10.1136/bmj-2023-078378.
- [5] Meng et al. Non-invasive biopsy diagnosis of diabetic kidney disease via deep learning applied to retinal images: a population-based study. The Lancet Digital Health. 2025. DOI 10.1016/j.landig.2025.02.008.
- [6] Diabetic Foot Ulcer Classification Models Using Artificial Intelligence and Machine Learning Techniques: Systematic Review. Journal of Medical Internet Research. 2025;27:e69408. DOI 10.2196/69408.
- [7] American Diabetes Association Professional Practice Committee. 12. Retinopathy, Neuropathy, and Foot Care: Standards of Care in Diabetes—2026. Diabetes Care. 2026;49(Suppl 1):S261–S276. DOI 10.2337/dc26-S012.
Methodology DescriptionThe description of the attachments in this article is limited to a file-level structural inventory; the patient-level medical record content is not shown or reproduced. Automated desensitization screening does not constitute privacy compliance certification, nor does the attached data constitute a public training set or clinically validated model rankings. Clinical definitions and use should be judged by a qualified medical professional in conjunction with applicable guidelines, a complete medical history, and local regulations.
