From DICOM to pathology slides: How medical multi-modal AI establishes a verifiable data and evaluation closed loop

Imaging pathology multi-modal closed loop: authorization and de-identification, patient-level association, modality quality control, independent segmentation, model evaluation, and clinical error review. — Figure 1｜Multimodal data is not about piecing together images, reports and labels; it requires a closed loop from source correlation to independent evaluation and failure analysis.

Multimodality is not as simple as “image plus text”

Medical multimodal models are often described as a fusion of images and text, but real tasks usually contain at least four types of information: visual signals, structured measurements, clinical narratives, and temporal context. CT volumes, pathology whole sections, radiology reports, surgical records, laboratory indicators, and diagnostic codes each have different granularities, error mechanisms, and available times. Simply splicing them together may result in greater input, but may not necessarily result in more reliable clinical evidence.

The first step should be to ask clearly about the prediction target and visible modes. For example, the radiology report generation model reads images after the examination is completed; the preoperative risk model can only use information before surgery; the pathology assistant tool may use slice images and limited clinical background. If the task is to predict postoperative outcomes, postoperative pathology as a feature will cause label leakage. If images are not available in clinical deployment, models that rely on images during the training phase will not be able to reproduce. The task definition must also state the target time point, input sources and allowed modalities.

attachment associates surgical cases, images, disease course, pathology and report fields, and requires that the images and texts correspond to the complete sequence. These requirements can be translated into modal contracts: file type, structure, key metadata, quality thresholds, association keys, time windows, annotation sources, and missing status for each modality. Only when the contract is approved can the model engineering know what each sample actually represents.

DICOM volume data integrity and interpretable preprocessing

DICOM is in preserving image pixels as well as acquisition and spatial metadata. CT projects at least check PatientID against project patient key mapping, StudyDate, Modality, BodyPartExamined, SliceThickness, PixelSpacing, Rows, Columns, BitsAllocated and other necessary information while verifying UID references, number of instances, image orientation, slice spacing, duplicate slices and pixel value ranges. Different reconstructed sequences may correspond to different nuclei, slice thicknesses, or contrast agent stages; treating them as homogeneous images will confuse the distribution of the training input.

"Complete sequence" needs to be defined per task. For volume classification, all eligible axial slices may be required; for lesion detection, localization and reconstruction sequences may also be required; for reporting correlation tasks, relevant sequences and identifiers within the same examination should be retained. You can't just pick three keyframes as a default replacement for a full CT. If you do perform keyframe filtering or resampling, statistics of the parent sequence ID, filtering method, algorithm version, spatial transformation, and discarded data should be saved. Preprocessing is not an invisible technical detail, but part of the data spectrum.

De-identification needs to be handled at the image and metadata levels together. The DICOM standard provides attribute profiles, but free text, private tags, names and dates burned into pixels, UID associations, and file metainformation can all pose risks. On the other hand, some tasks require the preservation of relative temporal, age-group or longitudinal connections. What configuration and risk controls are used should be determined by the Privacy and Data Governance team and verified through sample inspections. Research copies and original mapping tables should be managed separately, and permissions, auditing, and retention policies should be integrated into the project process.

Image quality also includes equipment and acquisition differences. Scanner manufacturer, reconstruction kernel, slice thickness, contrast agent protocol, image noise, motion artifacts, field of view, and protocol changes can all contribute to domain shifts. Model evaluation should document site, equipment, year of inspection, and protocol, and avoid inadvertent leakage of near-duplicate samples from the same center in training and test splits. Even if the complete sequence is retained, check whether the scan covers the target site, whether the image is readable, and whether the subject meets the inclusion criteria.

Pathological full sections need to be connected layer by layer from patient, specimen to patch

Pathological whole-slice images (WSI) usually reach billions of pixels. During model processing, they are divided into a large number of local patches and then summarized into slice or patient-level representations. If you only look at the final diagnosis label, it is easy to overlook the specimen type, sampling location, tissue quality, staining, scanning magnification and pathology report version. The correct data level includes at least patients, visits, specimens, tissue blocks, slides, WSI, patches and reports; different levels must have traceability identifiers.

Pathology report text can provide weakly supervised labels, but the report does not always correspond one-to-one with each patch. The entire section may contain tumor, stroma, necrosis, inflammation, and normal tissue simultaneously. Directly copying patient-level malignancy diagnoses to all patches will generate a large number of noisy labels. Multi-instance learning, attention pooling, or region annotation can handle weak labels, but algorithm performance still relies on case-level label quality, slice selection, and specimen coverage. If regional-level lesion detection is studied, a regional labeling and review process by pathologists should be designed.

Immunohistochemical markers, specific stains, histological description, and TNM/AJCC staging should be addressed along with the conclusion. Marking results need to be retained as positive/negative, indeterminate or not tested; the applicable version and source of evidence must be indicated for staging. For malignant tumors, clinical staging, imaging staging, and postoperative pathological staging are not synonymous fields. If the model task uses stage labels, it should be clear which stage is being predicted and based on what time point and standard. Patients who lack a certain test cannot be directly marked as negative.

Three fusion routes for image and text pairing

Early fusion combines text, structured variables and visual features at the input or earlier layers, which can capture fine-grained interactions, but requires fairly accurate time, sample and label pairing. Mid-term fusion allows different encoders to learn independently first, and then align the modalities through cross-attention, contrastive learning or shared latent space to facilitate the retention of modal expertise. Late fusion first produces single-modal results and then merges decisions. The system is easier to explain and more tolerant of the absence of certain modalities, but it may not be able to learn complex cross-modal complementarity.

Before selecting a fusion method, establish a simple baseline: clinical variables only, text only, image only, and simple late fusion. If a complex multimodal model does not consistently outperform the baseline, it may be due to pairing errors, insufficient sample size, label noise, or a modality simply expressing the same information repeatedly. Ablation experiments that “remove a modality” should also be compared to test performance and calibration in the absence of the modality. If a model only works when all modes are complete, the deployment end must confirm that these inputs are stable and available in the target scene.

Nature Medicine's TITAN work in 2025 demonstrates a research line for learning full-slice representation, region description, and cross-task transfer with large-scale de-identification of pathological images and reports; Nature Medicine's PRISM2 in 2026 further conducts research around pathological basic models and clinical dialogue supervision. These papers illustrate that report pairing and multi-scale representation are active research directions, but the training data, tasks, external evaluations and model licenses of the papers have their own boundaries, and their paper indicators should not be directly translated into the data quality or clinical benefit promise of a certain project.

Label, source and time determine whether the review stands up

Imaging reports and pathology reports are clinical evidence, but they may also contain errors, delays, version differences, or diagnostic uncertainty. For supervised learning, labels should retain source type, generation time, review status, rule version, and uncertainty. Labels automatically extracted from the report text should be sampled and checked with the original report; pseudo-labels generated by the model should not be confused with expert labels without review. If the label comes from postoperative pathology, but the input is only preoperative images, its time span and applicable population must be clearly defined.

The training set, validation set and test set need to be isolated by patient to avoid multiple slices, multiple scans or repeated reporting of the same patient across sets. For data from the same institution, there may also be leaks of near-duplicate images, templated documents, image post-processing copies, and equipment markings. Independent external verification should try to come from different sites or different time periods, and check whether the labeling process is consistent. If the target is a multi-country, multi-language or multi-device deployment, data coverage must also reflect these differences; a single site's high score cannot answer the question of out-of-domain reliability.

Ratings should exceed AUROC. The classification task can look at AUPRC, sensitivity, specificity, positive predictive value, negative predictive value, calibration and threshold performance at the same time; the segmentation task needs to report boundary and volume related errors; report generation requires factual consistency, omissions, false positives and clinical expert review. Subgroup analysis is required for device, site, age group, gender, disease subtype, modality loss and case complexity, and confidence intervals are given. Clinical scenarios also evaluate human-machine collaboration, workflow impacts, and safe handling of failures.

A practical multimodal data acceptance checklist

The first layer is the file and metadata: whether the DICOM can be read, whether the sequence is complete, whether the pixel and spatial information is valid; whether the PDF text layer can be extracted, whether the page order and table are complete; whether the pathological image meets the resolution and scanning requirements. The second level is association: whether there is a reviewable relationship between patient keys, visit keys, examination dates, imaging reports, surgical records, pathology numbers and slices.The third level is content: whether clinical fields, labels, and imaging conclusions can be located in the original text, and whether there are explanations for key deficiencies.The fourth layer is governance: whether the scope of authorization, desensitization process, access rights, retention and destruction rules are recorded.

Sampling should be based on the "complete patient package" as a unit, not just a single document. Stratified by modality and complexity, the same patient's course, tests, examinations, surgeries, discharges, and pathology trajectories were rechecked. The attachment proposes that the image report fields include examination date, equipment, site, plain scan/enhanced scan, positive/negative, image findings and conclusion; these fields can be converted into missing rate, value legality and original text consistency check. For serious correlation errors found in a single case, it is necessary to evaluate whether it is a systemic problem and expand the sampling, rather than just correcting one file.

For the algorithm engineering team, the final release package should also have a reproducible manifest: patient-level sample list, modal file hash, data version, filtering rules, segmentation results, transformation pipeline version, label definition and evaluation script. Sensitive identifiers should not appear in public lists, but controlled IDs can be used within the project to complete traceability. Data cards should describe coverage populations, sites, devices, time frames, missing modalities, known biases, and limitations. Such delivery supports subsequent recurrence and model change review.

Set clear boundaries from data preparation to clinical validation

Research baseline performance does not equal clinical usability. Before entering the real workflow, it is also necessary to verify how the system works under target users, target devices, real loads and abnormal inputs; whether the model output is correctly understood; whether it can reject or prompt when input is uncertain or missing; whether new deviations or workload burdens occur. FDA/IMDRF's GMLP principles emphasize data matching the target population, training and testing independence, human-machine teaming, and full life cycle monitoring; DECIDE-AI provides a reporting framework for early clinical evaluation. They support process design and do not automatically endorse a model.

development team can set the evaluation closed loop to: data source and authorization verification, patient-level association, modality quality inspection, label review, version segmentation, baseline and multi-modality comparison, external validation, subgroup and failure analysis, expert review, and revised regression evaluation. Every time the data or model is updated, the scope of the change and the impact on indicators should be recorded.If new equipment, new report templates, or new sites are added, this should be considered a distribution change and key performance should be revalidated rather than assuming that the model will naturally generalize.

Langhui can assist in medical data field sorting, imaging and pathology data quality control, medical annotation, blind testing and algorithm evaluation design based on the confirmed project scope. The specific sample sources, case volume, modal coverage, expert allocation and acceptance results must be subject to the official project documents; this article does not represent spot inventory, clinical trials or commitment to diagnosis and treatment effects. The value of high-quality multi-modal data is that every input, label, conclusion and restriction can be verified, helping the algorithm team make the research problem more reproducible.

Multimodal pairing quality needs to be evaluated independently rather than just checking if the files exist simultaneously. The patient and visit key, time, anatomical site, examination number, pathology number, surgical records and report citations can be used as matching evidence, and clinical reviewers can sample and confirm whether the images and text truly correspond. If the images are from pre-operative but the report is from post-operative review, or the pathological samples are taken from different parts, the files are still not effectively paired in the same patient directory. The matching status should be divided into confirmed, automatic matching pending, suspected mismatch and unmatched, and which status can be used in the model task should be set.

Evaluation data also prevents implicit duplication of slices, sequences or derived images of the same patient. Image hashing and perceptual hashing can find identical or near-duplicate documents; patient keys and exam keys can find duplicate exams; report templates and dates can find duplicate documents. Near duplicate samples may make the test set score falsely high. Deduplication rules should retain the mapping of removed records to the main record and be executed before patient segmentation. During external assessment, it is also necessary to record whether the verification agency appears in the training materials and whether the images are reproduced from the same data supply or public source.

For generative models, evaluation should also focus on outputs that “seem reasonable but have no evidence”. Image question-and-answer and report generation can check for missing key findings, incorrect additions, negative word flipping, incorrect anatomical sites, and expressions of uncertainty; pathology descriptions should differentiate between what is actually visible in the slides and what is inferred from the clinical context. Blind expert reviews, reference evidence-based fact checking and structured error classification need to be used together. If automatic evaluation indicators are used, their relationship with expert judgment should first be verified, and text similarity cannot be directly equated with clinical correctness.

In terms of image preprocessing, normalization, window width and level, resampling, cropping, color normalization, and tissue region filtering can all change the signal seen by the model. Each step should have versions, parameters and failure logs, and try to keep the same rules in training and testing. For CT, window width, window level, and reconstruction kernel selection can affect texture; for pathology, stain normalization may alleviate scan differences, but may also erase meaningful stain intensity cues. This should be validated through ablation and site layering, and avoid treating preprocessing as a harmless, generic step.

If the algorithm is to be used for human-machine collaboration, the data set and evaluation should record the doctor's baseline, the model's performance alone, and the doctor's performance plus the model. Models are improved on an independent test set and there is no guarantee that physician workflow will be safer or more efficient. Actual use also depends on whether recommendations change decisions, when doctors ignore or overwrite models, whether model errors are discovered in time, whether low-confidence samples are converted to manual work, and whether users with different experience levels benefit consistently. Early clinical evaluation should first focus on usability and potential harm, and then gradually evaluate clinical outcomes; it cannot jump directly from offline image-text matching scores to clinical benefit conclusions.

For surgical cases, images must form a verifiable sequential relationship with preoperative medical records, operative records, postoperative observations, and discharge summaries. The procurement specification example in the attachment requires that preoperative/postoperative images correspond to the immediate text description, and stipulates a certain minimum image resolution, number of images per case, and coverage of multiple surgical departments; such numbers are project acceptance terms, not a common scientific standard for all imaging research. The consultation dialogue also has an independent modal structure: dialogue rounds, first consultation/revisit consultation/medication consultation/examination interpretation scenes, patient uploaded images and doctor’s interpretation conclusions all need to be related, and the images cannot be used as labels separately from the corresponding dialogue. Only by explicitly writing these conditions into the sample schema can you check whether the model input is available in real usage scenarios and whether it is consistent with the text.

method boundaries and data sources

This article summarizes the key points of data engineering and cohort design based on the medical data fields and repurchase requirement documents provided by users, and discusses it in conjunction with public peer-reviewed research, technical standards, and regulatory/reporting guidelines. The fields, filter thresholds, and quantities in the attachment are examples of requirement specifications and do not prove that the data has been procured, delivered, or has passed model validation. The algorithms and processes in this article are used for scientific research and engineering design discussions and do not constitute medical diagnosis or treatment recommendations. Specific projects are subject to authorizations, data cards, contracts, and ethics/compliance reviews.