How is medical AI multimodal data organized by patient? From medical record folders to traceable longitudinal data assets

Patient-level multimodal data package: Unified patient index connects visits, documentation, tests, DICOM and pathology while preserving provenance and quality control status. — Figure 1｜The core of the patient package is not a directory tree, but the verifiable relationships between patients, visits, clinical events and original evidence.

Why “aggregating by patient” changes the algorithmic value of data

Medical artificial intelligence training samples are often delivered as files: a PDF, an image, a structured table, which can be opened individually, but when put together, it is not necessarily known whether they come from the same patient, the same hospitalization, or the same clinical stage. For the model, this is not a small finishing flaw, but the sample definition itself does not hold. If the CT in the model input belongs to another patient, or the discharge diagnosis does not correspond to the disease course record, the model may learn the wrong association, and the evaluation indicators will lose the basis for interpretation.

patient-level data package redefines the "sample" from an isolated file to a clinical object with boundaries: the patient's identity uses a stable de-identified key; there are multiple visits or hospitalizations under the patient; each visit is associated with the course documentation, examinations, tests, doctor's orders, imaging sequences and pathology evidence. In this way, the training team can clarify whether the sample is patient-level, visit-level, examination-level, or single-document level, and accordingly prevent the same person's data from leaking across the training set and test set.

attachment proposes to link structured CSV, PDF documents, DICOM and pathology under the same patient ID, and requires each patient package to check cross-modal integrity. This direction corresponds to entity resolution and data kinship in algorithm engineering: a patient key needs to connect multiple clinical entities, but it does not mean that every kind of data must exist. The correct data model should distinguish between "did not occur", "clinically not applicable", "not collected", "collected but not delivered", "association failure" and "unqualified quality", rather than writing all missing values as null values.

Build the patient package into a standardized data model instead of a flattened summary table

A groundable model usually contains five types of core entities. The first category is Patient: uses a project-wide pseudonymized identifier and does not save the real identity field directly. The second category is Encounter: describing outpatient, hospitalization, emergency or follow-up events, recording the start and end time, institution, department and type of visit. The third category is ClinicalEvent: indicating events such as disease course, diagnosis, examination, imaging examination, surgery, medication or discharge. The fourth category is Artifact: points to the source PDF, DICOM Study/Series, pathology report or slide. The fifth category is Assertion: records fields, conclusions, evidence locations, review status, and versions obtained from original text extraction or manual annotation.

Explicit relationships should be used between these entities rather than relying solely on file names. A patient can have multiple medical visits; a medical visit can be associated with multiple documents and multiple examinations; an imaging report can describe an examination, which in turn contains multiple DICOM series; a pathology report can be associated with one or more specimens, sampling sites, and slides. For reports, structured fields, and images that need to be joined, the relationship table should record the associated key, matching method, confidence level, time window, manual review status, and exception reason. When there is no reliable basis, it should be kept as "to be verified", and no forced matching is allowed to increase the completeness rate.

The semantics of the field also need to be standardized. The date must at least distinguish the time of event occurrence, collection time, report issuance time, document recording time, document revision time and the time of entering the data platform. Values need to retain the original value, unit, reference interval, abnormal flag and standardized value; diagnosis should retain the original text, coding system, coding version and mapping basis. Drug orders can retain the original drug name, dosage, frequency, route and execution status; standard coding such as ATC codes are auxiliary mappings and should not cover the original clinical text. The goal of the unified data model is to be both computable and retrievable.

How original text, structured fields and multi-modal files form an evidence chain

If only structured results are left, it will be difficult for subsequent reviewers to determine whether the fields are original records, rule extractions, model predictions, or manual additions. Each key field should be traceable back to the source: file identification, document type, page number or text fragment, original field path, extraction method, annotator role, review time, and data version. For imaging reports, the conclusions in the report can be connected with the examination date, modality, site, and Study/Series identification; for pathology results, the pathology report, specimen information, and corresponding slices need to be connected. The chain of evidence allows downstream training sets to trace errors back instead of just discovering their existence.

attachment puts forward requirements for documents such as the extractable text layer, correspondence between patients and visits, and necessary discharge records. These checks actually cover two different quality axes: one is machine readability and the other is clinical closure. The fact that PDF text can be copied does not mean that the text extraction is error-free; the order of tables, footnotes, units, paging and scanning should be randomly checked. The existence of a discharge record does not mean that the disease course is complete; it should be verified that it is within the same visit range as the admission diagnosis, hospitalization process and discharge outcome. For key indicators, compare structured fields and original text evidence to estimate extraction accuracy and omission rates.

The same "diagnosis" may change gradually during admission, course revision, operative records, pathology reports, and discharge summaries. Suppressing it into a final diagnosis erases important time and uncertainty information from model studies. A more prudent approach is to retain the diagnostic statement with time, author role, original text basis and status, and clearly distinguish between "clinical working diagnosis", "revised diagnosis", "postoperative diagnosis", "pathological diagnosis" and "discharge diagnosis". Models can select features that are available before a certain time cutoff on a per-task basis, while avoiding peeking at later information in prediction tasks.

DICOM and pathology cannot be reduced to a few representative pictures

images are sequence data with acquisition context, not a collection of JPEGs. For CT or MR, the level, orientation, pixel spacing, slice thickness, reconstruction kernel, contrast agent stage, and number of images of the examination record, Study, Series, and Instance all affect the algorithm input. The appendix emphasizes that the complete sequence, multiple examinations of the same patient, and preoperative and postoperative images should be retained. During the project, it is also necessary to check whether the series is complete, duplicate instances, missing layers, orientation and geometric information are consistent, and check the inspection date and location in the report with DICOM metadata. If the training target really only requires the selection of key images, the selection strategy and the relationship between the parent sequence should also be recorded.

Pathology also has multiple levels: pathology number, specimen, tissue block, section, digital whole section image (WSI), local patch and report conclusion cannot be mixed into one file. The scan magnification, pixel size, staining type, scanner, tissue coverage, and artifacts of the slide image all affect usability. The gross, microscopic, immunohistochemical, and stage fields in the report each have semantic meaning; TNM/AJCC staging of cancer cases requires recording of version and source. Postoperative pathology may or may not be consistent with preoperative imaging or clinical diagnosis; the latter is an important label difference and should be retained as a researchable phenomenon rather than directly deleted as a data error.

DICOM de-identification is also not equivalent to deleting a PatientName field. DICOM PS3.15 describes attribute-level confidentiality configuration files, but the actual process also needs to consider UID and reference relationships, private attributes, free text, text burned in pixels, file leading areas, and re-identification risks after combination with other data. When retaining certain necessary age or age characteristics based on the mission, a controlled, minimal, auditable approach should be used and evaluated by data governance and privacy responsible parties. Patient keys can be used for internal correlation, but the mapping table must be isolated from the research data, strictly authorized, and pseudonymization cannot be mistaken for anonymity.

From data acceptance to model usability verification

Patient package acceptance consists of at least four layers. The structure layer checks whether directories, file formats, and required entities exist; the association layer checks whether patients, visits, examinations, reports, and images are connected correctly; the content layer randomly checks fields, text layers, image integrity, and clinical documentary evidence; the governance layer checks authorization scope, de-identification, access control, and retention periods. The idea in the attachment of "checking CSV/PDF/image/pathology under the same ID, checking timestamps and original texts, and checking the proportion of difficult samples" can be rewritten into a repeatable acceptance rule, and the failed entries can be output into a list of differences that can be located.

Technical acceptance does not mean that the model performance is available. The model task must separately define the target population, index time point, prediction window, label source, positive definition, exclusion conditions and clinical usage scenarios. Training, validation, and testing should be divided on a patient basis; if the same patient has multiple visits, multiple image sequences, or multiple slices, they must be isolated as a whole. Sample composition, hospital, time, equipment, department, and case complexity will all affect external generalization, and distribution shifts cannot be eliminated by expanding the file size.

The ultimate goal is to make data reusable, auditable, and reinterpretable during the model life cycle. The fields and association rules defined during procurement delivery should be mapped to data cards, training set versions, evaluation sets, error analysis and subsequent revision records. If a field is upgraded from original text extraction to manual double review, or an image sequence undergoes a new de-identification process, version differences and the impact on existing evaluations must be retained. A patient package is not a permanently fixed ZIP file, but a data product structured around clinical entities, source evidence, permissions, and quality status.

In what aspects can Langhui provide verifiable support?

Medical AI projects often require translation of procurement specifications into data interfaces, sample definitions and acceptance logic. Focusing on the project authorization and agreed scope, Langhui can assist in sorting out the data dictionary, patient-level association rules, correspondence between source files and structured fields, medical annotation specifications, sampling quality control plans and model evaluation design. The specific delivery content, data sources, scale, subject coverage and quality indicators must be subject to the project contract, data card and acceptance plan confirmed by both parties; this article does not represent a commitment to any specific sample spot or delivery results.

An executable way to start is to first select an authorized representative sample to complete a small-scale POC: verify field mapping, patient and visit linkage, time alignment, DICOM/pathology reading, desensitization boundaries, annotation consistency and acceptance reporting. The POC should not only look at "open successfully", but also record the failure category, correction cost and sample boundaries that cannot be covered before deciding whether to expand. For medical data, clearly disclosing deficiencies and uncertainties often has more engineering value than giving a seemingly beautiful total score.

Patient package construction also needs to handle the life cycle of the identity key. The internal ID of the project should be generated by a controlled process, avoiding the direct use of identifiable numbers such as ID numbers, mobile phone numbers, and medical record numbers; cross-source matching should use approved deterministic or probabilistic rules, and save the matching basis and conflict status. Probabilistic matching should not just output a final ID, but should also provide thresholds, number of candidates, and human review. If a patient cannot be reliably matched, it is better to leave it as an unlinked sample than to incorrectly merge it because "the patient package must be complete". Whether the same pseudonym key is allowed to be reused between different projects should be determined by the scope of authorization and governance system.

From an engineering point of view, data quality is best gated separately before collection, processing, delivery and training. The source end checks file integrity, permissions, and format; the processing end checks parsing, field mapping, relationship establishment, and de-identification; the delivery end checks manifest, file verification, and sampling results; and then checks target tasks, labels, patient segmentation, and data versions before training. Quality rules need to have severity levels: for example, patient mismatching and exposure of identifiable information should block delivery; a small number of non-critical field deletions can be accepted after disclosure; modal deletions need to be judged whether they exceed the task contract agreement. Giving just one "pass rate" masks differences in the risk of different errors.

The database design should take into account both the preservation of the original text and the use of algorithms. The original data is read-only saved and checksummed; the parsing results are stored in a derived table, recording the parser version and running time; the revised fields are overwritten with new versions of the logic rather than being rewritten in a traceless manner. Generate a machine-readable list of expected entities, actual entities, source file hashes, correlation results, QC status, and exception codes for each patient package. If a diagnostic mapping is corrected by a clinical expert, the original mapping and the rationale for the revision are retained to facilitate reproducibility of the experiment. Access logs for sensitive raw files and training data access permissions should be managed separately.

Patient packets should also clarify the difference between “minimum available sample” and “extended sample”. For some NLP tasks, a traceable disease course document is sufficient; for longitudinal prediction, patients need to have at least a clear index time point, observation window, and outcome; for imaging-pathology correlation, it may be required that surgery, imaging, specimens, and pathological diagnosis can all be reliably connected. If all cases are screened according to the same set of complete package criteria, it is easy to discard high-quality data that can be used for single-modal tasks; conversely, mixing incomplete packages into multi-modal training will also make the modality missing mechanism of the model opaque. The data catalog can provide three-level rules of "core required", "conditionally required" and "optional extensions" and filter by tasks.

Quality indicators should cover accuracy, completeness, consistency, timeliness, uniqueness and traceability, rather than just expressed in terms of missing rate. For example, the completeness rate of patient IDs may be 100%, but if errors are merged across hospitals, the accuracy is still poor; DICOM files can be read, but patients and reports are incorrectly matched, and the consistency is still unqualified; PDFs have a text layer, but the key tables are extracted in the wrong order, and the content quality is still insufficient. For each indicator, the denominator, sampling frame, passing definition and confidence interval should be stated, and the risk level of critical errors should be presented separately. This way buyers, data suppliers and modeling teams can all discuss improvements around the same facts.

Each data packet also needs a clear life cycle status, such as to be received, parsed, pending review, available for specified tasks, restricted use, withdrawn or expired for destruction. Status changes should retain the time, operating role and basis to prevent old version files from being read by the training system after the new version is approved. If the use of data changes, such as from field extraction to model pre-training or cross-institution evaluation, you should reconfirm whether the original authorization covers this use. Data access policies can distinguish original documents, desensitized images, structured data and derived tags by role, and perform agreed archiving or deletion after the project is completed. Patient-level correlation makes multimodal research more valuable and increases the risk of combinatorial identification, so correlation capabilities and access rights need to be incorporated into governance together, rather than just doing a desensitization at export time.

When handed over across teams, patient package specifications should also provide clear catalog examples, field data types, allowed values, encoding versions, null rules, and exception examples. For example, whether "examination time" refers to appointment, scanning or report issuance should be stated in the field description; "image positive" should indicate whether it is based on report conclusion or manual review. Structured tables should contain stable field IDs to facilitate identification of field name changes or semantic drift during multiple rounds of delivery. Validation scripts and expected outputs are provided for key fields, allowing procurement, annotation, algorithm and quality control teams to all use the same rules. If the receiving party discovers data anomalies, it should report files, fields, original text evidence and impact tasks through case-level problem tickets to form a traceable closed loop of repair.

Author information also belongs to the clinical context. After authorization and necessary de-identification, indirect identity attributes such as author role, professional title, and department can be retained to distinguish resident records, superior physician rounds, specialist consultations, and surgeon documents; names and ID numbers will be removed according to project rules. If all authors are uniformly replaced by "doctor", it will not only reduce the traceability of the source, but also make it impossible for researchers to analyze the semantic differences between different document types and signing roles. For drug and non-drug orders, the patient package should retain the original order name, common name, dosage, frequency, route, execution time and status; standard codes such as ATC can be used as mapping fields, and the basis for mapping should be recorded. Standard codes cannot be used to overwrite the original order.

method boundaries and data sources

This article summarizes the key points of data engineering and cohort design based on the medical data fields and repurchase requirement documents provided by users, and discusses it in conjunction with public peer-reviewed research, technical standards, and regulatory/reporting guidelines. The fields, filter thresholds, and quantities in the attachment are examples of requirement specifications and do not prove that the data has been procured, delivered, or has passed model validation. The algorithms and processes in this article are used for scientific research and engineering design discussions and do not constitute medical diagnosis or treatment recommendations. Specific projects are subject to authorizations, data cards, contracts, and ethics/compliance reviews.