Define the data unit first and then discuss the purchase quantity
A physical examination report may be a comprehensive report for a complete year, or it may be just a single report form for a certain examination. Both are PDFs or pictures on the surface, but the training tasks are completely different: the former can be used for report understanding, risk induction and cross-year change analysis; the latter usually serves OCR, field extraction, anomaly identification or image report understanding. If the sample unit, minimum deliverable content, and expected tasks are not determined before the inquiry, suppliers may interpret them individually in terms of "pages," "sheets," and "parts." The final quantity may seem sufficient, but the model lacks complete context.
Therefore, procurement requirements should be deduced from the data definition from "what the model wants to learn". If the goal is to parse inspection indicators, the smallest unit can be "an indicator and its report context"; if the goal is to understand a physical examination conclusion, the sample unit should include the report text, conclusion suggestions and corresponding examination materials; if the goal is to learn longitudinal health changes, multiple reports from the same anonymous individual must be stably related. The quantitative caliber needs to simultaneously describe the number of patients, number of medical visits or physical examination events, number of files, number of pages, and number of image sequences. These units cannot be mixed into a total amount. The demand types in the
attachment cover different processing objects such as comprehensive physical examination reports, report sheets, report sheets plus case context, complex form OCR, text corpus, medical books and knowledge materials. They cannot share an acceptance form. A reasonable procurement package should first be split according to purpose, and then define data units, required fields, original files, associated keys, quality thresholds and sampling methods for each category. This can reduce later rework and make it easier for the model team to determine which samples are suitable for training, which are suitable for verification, and which are only suitable for retrieval knowledge.
Refine “reported” into executable acceptance fields
The availability of medical reports is not simply "can be opened". It is necessary to check the report type, source institution type, inspection date, sample anonymous ID, report text, inspection items, results, units, reference intervals, abnormal flags, conclusion recommendations, and the mapping between files and structured records. For image inspections, the relationship between modality, inspection site, sequence or image identification, inspection date, image report and corresponding original image must also be recorded. The field list should distinguish between required, conditionally required and optional, and explain how to code null, unknown, unchecked and unidentified values respectively. If the data package also contains a medical institution or doctor information directory, the source of the information, the affiliation of the institution and department, the field update time and the scope of application should be recorded; dynamic fields such as shift scheduling, house calls or number sources must have a clear refresh cycle, and the current availability cannot be judged solely on the completion rate of static fields.
It is recommended to retain three levels of content. The first layer is the original document to ensure subsequent traceability; the second layer is faithful transcription or OCR text, marking unconfirmed characters and uncertain areas; the third layer is standardized fields, which saves the mapping results of standard units, terms or encodings. There must be sample ID and location mapping between the three layers, and you cannot only retain the "clean results" that have been processed by the model or manually. Original values and standardized values are saved in separate columns, and standardized conversion retains rules and versions so that original medical information and engineering processing results can be distinguished.
In particular, avoid taking OCR confidence as medical accuracy. Layout recognition may misalign table columns, numerical recognition may miss decimal points, reference intervals may be interchanged with adjacent indicators, and abnormal arrows may also be missed. Quality control needs to design checks based on error types: character recognition, cell positioning, field attribution, numerical values and units, reference ranges, exception identification and report-level conclusions are verified separately. For low-resolution samples, the correct behavior should be to mark "unreliable reading" and retain the reason, rather than autocomplete seemingly reasonable values. The requirement for low-quality report images in the attachment also illustrates this point: systems need to learn to recognize gaps in information rather than guess at ambiguities.
Medical quality should check the semantic chain instead of just looking at the field completion rate
The structured fields are complete, which does not mean that the clinical semantics of the sample are correct. The medical chain of a report at least includes original examination results, units and reference ranges, abnormality judgments, doctor's descriptions, and general examination conclusions and recommendations. Quality audits should ask: Does the abnormality identification match the numerical value and reference range? Are the conclusions supported by the text of the report? Do the findings in the images correspond to the conclusions? Does desensitization cover important medical content? Are patient ages, dates, or test items conflicting on different pages of the report? These problems usually cannot be caught by a simple non-null check.
For the comprehensive physical examination report, it is recommended to adopt a two-tier approach of “admission red line + sub-score”. The red line is used to determine unacceptable issues such as authorization, privacy leaks, severe page missing, missing images, mismatches, and medical logic errors; after passing the red line, the report is scored on dimensions such as completeness of the report, image quality, field accuracy, item richness, anomaly explanation, and longitudinal comparison. The missing rate of basic fields should be calculated separately, and medical logic errors should be counted separately and should not be diluted by a set of average scores. All indicators must write down the denominator, sampling unit, missing value processing and duplicate sample deduplication rules. The
procurement standard itself is also subject to consistency review. For example, if an acceptance document adopts different thresholds for the "number of valid medical pages" in different chapters, the supplier and the accepting party may each implement different thresholds. Similarly, the applicable scope, image attachment requirements and scoring levels of "physical examination report", "report sheet" and "case information" must also be clearly defined. It is recommended to implement each requirement into a unique field or acceptance item, and provide the definition, detection method, passing conditions, responsible party and dispute handling process. In the POC stage, a batch of representative samples are used to verify the rules. If threshold conflicts are found, the standards will be revised first before entering batch delivery.
Image pairing and privacy processing need to be part of the delivery chain
When the report mentions ultrasound, CT, MRI, endoscopy or other imaging examinations, the image files should be able to correspond to the same anonymous sample, the same examination and the corresponding text description. Acceptance is not only based on "there are pictures in the folder", but also confirms that the inspection time, modality, location and report conclusions match each other; duplicate images, wrong lesion maps, and irrelevant sequences may cause model learning bias. If the delivery contains DICOM, the image sequence integrity, key metadata and identification information processing should be checked; if the delivery is a screenshot or report image, the resolution, direction, text readability and image-text correspondence should be ensured.
De-identification cannot only cover names in the PDF body. Personal information may appear in headers and footers, QR codes, barcodes, image burning characters, DICOM header fields, scanned file names, document free text or export directories. On the other hand, date, age, examination parameters and longitudinal associations may be variables required for the research task and should be handled in a manner premised on risk assessment and scope of authorization. Both parties to the delivery should record the processing scope, processing tools or rules, review methods, exception handling results and responsibility attribution; conduct sampling reviews of images and documents, and keep auditable processing logs. The DICOM standard provides a reference framework for image de-identification configuration, but the actual implementation still needs to be reviewed based on the text embedded in the image, information outside the file and specific usage scenarios.[1]
Data sources and authorized materials should be managed separately from training data, but can be linked at the governance level. Procurement acceptance should not only rely on the supplier's "desensitized" statement, but should confirm that the source description, scope of authorization, usage restrictions and transfer conditions meet the project requirements; for data whose source, authorization or purpose is unclear, use should be frozen and clarified first. Before the sample enters the data pool, source records, processing versions, and content summaries should also be generated to avoid being unable to answer "where does this data come from, what processing it has undergone, and what tasks it can be used for."
Use POC and stratified sampling to reduce batch risks
Before bulk purchasing, the purpose of POC is not to show the best sample selected by the supplier, but to verify whether the entire delivery process is repeatable. Sampling should cover report types, different formats, image quality, normal vs. abnormal results, complex forms, records spanning multiple years, and different equipment sources. The sample list should be determined with the participation of the demand side to avoid seeing only easily processed data pre-screened by the supplier. The POC stage separately checks the format, fields, image-text correspondence, medical semantics, desensitization and file mapping, outputs item-by-item defects and fixes the closed loop, and then decides whether to scale up. After
enters batch acceptance, in addition to the overall pass rate, attention should also be paid to the performance of key subgroups. Overall accuracy may mask obvious flaws in a certain type of report, a certain device, a certain format, or a low-definition sample. Stratified sampling can be performed by reporting category, image quality, field complexity, and source category, reporting error rates and confidence intervals for each stratum. Adopt whole-case return or batch suspension rules for one-vote veto issues; set review ratios, rework time limits, and re-inspection standards for fixable issues. The original version, modified version and retest results should be retained for each revision to prevent problems from recurring in different batches.
From an algorithmic perspective, sample segmentation also belongs to procurement acceptance design. If multiple reports from the same person are scattered into the training set and the test set, the model may "remember the answer" through repeated content or personal characteristics, resulting in higher validation results. For longitudinal tasks, the data set should be segmented by patient, and the time segmentation should be clear; for layout recognition, the distribution deviation caused by the same template across collections should also be considered. The test set should ideally retain difficult examples, low-quality samples, and true missing cases, and the test labels should be managed by an independent process. Through the linkage of procurement, governance and model evaluation, it can be determined that model improvement comes from true generalization rather than data duplication or test leakage.
Quality indicators must be mapped to model risks
Different errors have different effects on the model. Misplaced subtitle identification, incorrect reading of abnormal test values, and merging of reports from different patients cannot all be recorded as a common field error. It is recommended that defects be graded according to severity: errors that will change the clinical meaning or patient attribution are set as critical defects; errors that affect model judgment but can still be found in the manual process are set as major defects; format issues that do not change the core semantics are regarded as general defects. Each level should explain how to count, how to review, and what corrective or rejection actions are triggered. The acceptance report should present a distribution of defects, not just an overall pass rate.
It is also necessary to distinguish between "field level accuracy rate" and "sample level pass rate". A field extraction task can report precision, recall, F1, numerical difference tolerance range, and field positioning accuracy; the qualification of a report also depends on required fields, completeness, associated files, and medical consistency. If millions of fields are simply averaged, a few serious errors may be diluted by a large number of easy fields. A better practice is to report micro-averages, macro-averages, key field metrics, and sample-level pass proportions simultaneously, stratified by reporting category and difficulty. The sampling method and estimated uncertainty should also be given for the sampling results to avoid promoting the accidental high scores of a small sample as full quality assurance.
The data distribution itself also requires quality control. Proportions of gender, age group, department, examination type, abnormal status, institution, and report format may be relevant to the model goals. Enrichment in order to highlight difficult examples during data collection can improve training efficiency, but changes the overall sample structure; if the same enrichment method is used for the evaluation set, the results cannot represent general usage scenarios. It is recommended to maintain both a "challenge set for proficiency testing" and a "representative evaluation set close to the target usage scenario", and clarify the selection method of the two. For rare categories, the true sample size and confidence interval should be reported to avoid making deterministic conclusions based on a small number of samples.
Close the loop on data defects into the supply and annotation process
The acceptance result cannot just stop at the purchaser’s deduction or return record. Every flaw should be traced back to its origin: whether source screening does not meet the definition, whether scan acquisition causes image distortion, whether OCR gets column relationships wrong, whether de-identification rules obscure medical values, whether manual review is missed, whether delivery transformation destroys fields or paths. After associating the cause of the defect with the batch, processing version, and responsible link, the project team can determine whether to change the collection specifications, processing rules, labeling training, or file packaging process.
It is recommended that each batch of delivery have a machine-readable manifest recording the file hash or checksum value, anonymous sample key, file type, generation time, processing version, correlation check event and review status. For revisions, the original file should not be directly overwritten, but the relationship between the old and new versions and the reasons for the modification should be recorded. When requirements change, the standard version number, effective date, and affected batches are retained to avoid using a set of implicit rules for different batches of the same project. For long-term cooperation, repurchase does not mean simply copying the format of the previous batch, but continuously updating field definitions and difficult case coverage using the problem list as input.
Langhui’s delivery idea should start with verifiable rules
The professionalism of medical data procurement does not depend on how long the requirements document is written, but on whether the goals can be converted into executable sample definitions, field specifications, quality indicators, POC processes and batch acceptance evidence. Langhui's external expression of medical data projects can specifically fall into the steps of requirement disassembly, data structure design, medical quality control, de-identification review and acceptance closed loop; each project then determines the actual delivery scope based on its authorization, samples, tasks and acceptance rules. Such a statement can not only reflect engineering capabilities, but also prevent the threshold of a certain procurement project from being mistakenly written as a universal commitment for all customers.
A mature procurement solution will ultimately allow algorithm engineers to obtain directly readable files and fields, allow medical experts to review context and clinical logic, and allow procurement and compliance personnel to trace sources, authorizations and acceptance conclusions. The number of reports is just the entrance; the relationship between reports, between reports and images, and between fields and original text determines whether it is a reliable data asset.
FAQ
Is it enough to only look at the field accuracy for acceptance of medical AI training data? It is not enough. Things such as sample definition, source authorization, documentation completeness, image quality, picture-to-text correspondence, medical semantics, de-identification, duplication rate, and patient-level segmentation should also be checked, and separate rules should be established for high-risk defects.
How should OCR of medical reports handle fuzzy areas? Mark content that cannot be reliably identified as uncertain, retain the original image location and confidence, and enter the manual review or elimination process. Don’t guess at check values, exception flags, and diagnostic conclusions based solely on context. How to select
How should representative POC samples be selected? Cover common and rare layouts, different report types, abnormal and normal, clear and low definition, complex tables and image correlations according to expected tasks, and retain random samples to avoid using only selected samples that are easy to pass.
References
[1] DICOM PS3.15 2025e, Security and System Management Profiles, Appendix E: Attribute Confidentiality Profiles.https://dicom.nema.org/medical/dicom/2025e/output/html/part15.html
[2] Mongan J, et al. Checklist for Artificial Intelligence in Medical Imaging (CLAIM): 2024 Update. Radiology: Artificial Intelligence. https://pmc.ncbi.nlm.nih.gov/articles/PMC11304031/
[3] STARD-AI Steering Group. The STARD-AI reporting guideline for diagnostic accuracy studies using artificial intelligence. Nature Medicine, 2025. https://doi.org/10.1038/s41591-025-03953-8
