First define the labeling object and judgment boundary
“Medical annotation” is not a uniform task. Determining whether answers are accurate, extracting entities from medical records, identifying lesions in images, determining whether reports match images, and marking doctor-patient conversations into intent categories each require different label definitions and reference bases. Before starting, the task should be broken down into a problem that can be judged independently: what input the annotator sees, what labels or modifications are output, what data are allowed to be relied on, what situations must be marked as uncertain, and how to deal with missing or contradictory information. If these boundaries are not clearly written, the annotator will fill in the definitions based on personal experience, and differences between teams will be mixed into the training labels.
An effective medical annotation guide must at least include the purpose of the task, inclusion and exclusion criteria, label definition, principles of positive and negative examples, boundary cases, evidence priority, missing value processing and upgrading rules. The guide should not just give a list of label names, but also explain whether the labels are mutually exclusive, whether they can be multiple-selected, whether the label applies to sentences or the entire medical record, and how to handle negative and historical information. For medical Q&A, "inaccurate facts", "insufficient evidence", "answers that do not address the question", "unreadable structure" and "suggestions lacking applicable conditions" must be distinguished; for medical record extraction, it should also be stated that the patient's chief complaint, past history, current diagnosis, pending diagnoses and family history cannot be mixed into one disease entity.
In medical scenarios, the annotator’s professional background and task matching are also important. Tasks such as medical terminology proofreading, guideline evidence evaluation, clinical decision-making helpfulness scoring, and image lesion localization should not be completed by annotators with the same qualifications by default. The project should explain the relationship between doctors' professional direction, working years or qualification requirements and tasks, provide necessary training, and use task assessment to confirm that the annotators understand the specifications. For content that may involve high-risk diagnosis or treatment recommendations, a clinical expert review and dispute escalation mechanism should be set up to avoid allowing non-professional annotators to independently judge medical safety.
Split subjective judgment into multiple observable dimensions
The quality of medical AI answers is often multi-dimensional. The annotation process in the attachment separately scores answer accuracy, content relevance, structural clarity, evidence support, decision-making helpfulness and clinical adoption rate, and requires annotators to explain the reasons for deductions and provide suggestions for improving medical logic. This design has important value: an answer may be factually correct but not answer the patient's actual question; it may have a complete structure but lack evidence; or it may have a correct citation but miss a key contraindication. A single “good/bad” label cannot support model teams in locating defects.
An anchor description must be provided for each dimension. For example, accuracy focuses on whether the medical facts are correct and whether there are key omissions or contradictions; relevance focuses on whether the answer responds to the input question and patient context; evidence support focuses on whether the key conclusions have verifiable sources or reasonable reasoning; decision-making assistance focuses on whether the information helps professionals complete the current task; structural clarity focuses on whether the content is easy to read and review; clinical adoption-related dimensions must limit the judgment objects and evidence, and cannot allow annotators to speculate out of thin air "whether the doctor will adopt it."The scoring scale needs to explain the difference between adjacent scores, as well as how to handle full scores, low scores, and undecidable scores.
More importantly, scores must carry defect labels and justifications. It is recommended to set error types such as factual errors, omissions, over-inference, evidence mismatch, omission of patient conditions, insufficient risk warning, chronological confusion, inconsistent graphics and text, formatting or readability issues, etc. Each comment should identify where the problem occurred, explain why it's important, and give directions for improvement if possible. Annotators cannot just copy answers, nor can they replace medical judgment with low-value opinions such as punctuation and formatting. After error labels are associated with specific segments, model training can be used to construct preference pairs, correction samples, or safe test sets.
Use double review, blind testing and arbitration to control judgment deviations
The core of blind testing is to reduce annotators’ preconceptions about model identity, supplier, version or expected results. When comparing two or more responses, the order of presentation should be randomized, the source information should be hidden, the same question and patient context should be used, and the annotators should be rated separately on the same dimensions before comparison. If AI and human answers are being evaluated, the prompts and data conditions should be as consistent as possible. Just showing two responses side by side, but letting the judges know which one comes from the model, cannot be called sufficient blindness. The
project can adopt stratified double standards based on risk and budget: all high-risk questions, difficult questions, and safety-related questions will be independently marked by two qualified reviewers; a portion of common questions will be selected for double standards to continuously monitor consistency; samples with high differences will be arbitrated by a third party or a senior doctor. Each annotator should have a stable anonymous code, recording task version, guide version, reviewer, arbitration conclusion and reason for change. Rather than simply using a majority vote to override a minority opinion, arbitration should retain categories of disagreement: whether the definition is unclear, conflicting sources of evidence, incomplete context, or legitimate clinical differences.
The consistency index must match the tag type. Classification labels can use chance correction indicators such as Cohen’s kappa or Fleiss’ kappa; continuous scoring can consider intra-group correlation coefficients or correlation analysis, combined with divergence ranges; entity extraction can report precision, recall, F1, and boundary matching rules. The sample size, category distribution, and number of annotators will affect the interpretation of the index, and you cannot claim that the annotation is reliable just by posting a consistency score. Low consistency sometimes indicates that the task definition is vague, sometimes it indicates that the case itself is clinically ambiguous, and it may also reflect insufficient review training, and the analysis of specific disagreement samples should be returned. When interpreting consistency, the original consistency rate, frequency of each tag, number of reviewers, double-standard sample extraction method and confidence interval should also be reported simultaneously. When categories are extremely imbalanced, chance-corrected metrics may be inconsistent with intuitive agreement; therefore, statistical methods should be chosen based on label type and study purpose, and numbers should be interpreted using stratified results and disagreement cases. If the sample is drawn from multiple specialties or risk strata, the overall index may mask a lack of consistency within a subcategory, and key subgroups should be reported separately. The primary endpoint, secondary endpoint and analysis plan should be written in advance before the project is launched to reduce the space for selecting favorable indicators afterwards. The final label after arbitration cannot replace the consistency result before arbitration: the former describes the final delivery, and the latter reflects whether the rules can be independently and stably implemented.
The medical LLM manual evaluation framework published in "npj Digital Medicine" in 2024 divides the evaluation process into stages such as planning, execution and arbitration, scoring and review, and emphasizes that goals, reviewers, sample selection, evaluation indicators and statistical methods need to be designed in advance.[1] Another study of visual language models in radiology employed a blind preference comparison and error revision task and reported significant disagreements between reviewers, suggesting that expert opinions do not automatically form a single truth value.[2] Therefore, the goal of the annotation project is not to cover up disagreements, but to make them measurable, traceable, and resolvable.
evaluation set should prevent data leakage and question contamination
Clinical AI evaluation sets should set management boundaries before training and development. Patient-level samples cannot be repeated between training and testing; leaks can also occur from approximate rewrites of the same medical record, different slices of the same image, or split segments of the same conversation. For longitudinal data, it is also necessary to check whether future information enters the current question input; for medical knowledge question and answer, the possibility of public questions being absorbed by the model training corpus should be monitored. The person in charge of the evaluation needs to manage access rights, versions, extraction records and question exposure. After major revisions, he should decide whether to continue to use the original test set.
In recent years, multimodal medical benchmarks have begun to shift from single-round question and answer to real task structures. NeurIPS 2025's SMMILE is a multi-modal contextual learning evaluation constructed by medical experts. The study found that the capabilities of most tested models in open multi-modal tasks are still limited, and irrelevant examples may lower performance.[3] The MTBBench of the same session focused on multi-modal and longitudinal tumor decision-making problems, pointing out that the model still exhibits unreliable performance when dealing with time changes, conflicting evidence and cross-modal information.[4] These studies support a practical judgment: assessment questions must retain sufficient clinical context and systematically test the model's ability to handle conflicts, deletions, and time changes, and clinical capabilities cannot be inferred solely by short questions and multiple-choice questions.
test set also defines task coverage and difficulty. It can cover common problems, rare situations, critical and severe risks, special groups, incomplete data, evidence conflicts, cross-modal correlations and long-term changes in layers. The number of samples and source distribution are reported for each stratum. In addition to the average score, high-risk error rates, critical defect recall rates, subgroup performance, confidence intervals, and uncertainty in model outputs should also be examined. For “don’t know” or “insufficient information” scenarios, appropriate refusal to answer or request for additional information must be regarded as evaluable behavior, and each question cannot be forced to be mapped to a medical conclusion.
The annotation process needs to manage reviewer drift
Experts will also change the judgment scale as the project progresses. In long projects, guideline updates, staff turnover, fatigue, or the recurrence of common errors can cause the meaning of the same label to drift from month to month. A set of anchor questions that are not notified in advance can be retained, and annotation tasks can be inserted periodically to compare the consistency of the old and new reviews; annotators who continue to deviate from the norms can be retrained and their recent results can be reviewed. Anchor questions should also be updated regularly to prevent reviewers from just memorizing answers rather than understanding the principles of judgment. The scope of impact should be recorded for each guideline change, and it should be determined whether the old batches need to be relabeled or only versioned.
Data auditing of medical annotations should focus on both samples and reviewers. The sample side records the source, desensitization status, context integrity, task version and evidence location; the personnel side only retains the anonymous coding, qualification type, training status and review results required to complete the quality audit. Evaluators should reduce unnecessary exposure of patient information, present the content necessary to complete the judgment on the task interface, and control export, download and access permissions. If the sample involves speech, images or free text, direct identification information and combinable identification clues should be included in the desensitization check, rather than just searching the name field.
Also distinguish between "correct answer" and "safe answer". The model may give factually correct content without specifying applicable conditions, contraindications, or emergency medical advice; it may also make risky and incorrect suggestions, but the text is very fluent. Safety annotations require independent severity stratification and clear harm paths, error discoverability, and manual interception capabilities around real usage scenarios. For situations that may lead to delays in diagnosis and treatment or improper treatment, the passing condition cannot be just based on the average accuracy reaching the threshold. A separate failure threshold and escalation action should be set.
Turn blind test results into executable quality closed loop
A blind test is not the end of the project. The results need to be returned to the data, specifications, models and product processes: if the error comes from missing fields, complete the data dictionary or input information; if the error comes from medical concept confusion, revise the labeling system and add control samples; if the error comes from evidence citation, check the search source and reference correspondence; if the error comes from multi-modal mismatch, review the file mapping and patient association; if different experts continue to disagree, supplement the boundary definition or mark the situation as requiring clinical judgment. Each round of optimization should retain the task version, sample version, model version, review conditions and result differences to avoid directly splicing incomparable scores into a progress curve.
Project management can set three stages: startup threshold, process quality and delivery acceptance. The startup threshold checks whether the guide passes the trial bid, whether medical experts confirm the rules, and whether the evaluation samples are kept independently; process quality monitors the sampling error rate, annotator drift and arbitration ratio; delivery acceptance checks the completeness of fields, medical accuracy, evidence traceability, privacy compliance and file versions. Set different severity levels for error classification: defects involving major diagnostic errors or privacy leaks should trigger suspension and comprehensive investigation; general format errors can be directed to repair. Settlement and repurchase terms must be consistent with the rule version to avoid using different scoring calibers for different batches.
Medical labeling should not use “many opinions” as evidence of high quality. High-quality annotation can tell the model team: where the answer is wrong, why the error is important, what context the evidence comes from, whether there is a reasonable disagreement among doctors, and how to verify it in the next round of training or product changes. For clinical applications, it must also be made clear that AI output is auxiliary information, which needs to be reviewed by professionals based on the actual patient situation, the latest evidence, and institutional processes. Annotation reviews can help discover system boundaries but are not a substitute for clinical accountability or real-world safety evaluation.
Langhui’s medical labeling capabilities should be reflected in the traceability process
introduces Langhui's medical data services to the outside world and recommends focusing on labeling specification design, medical expert collaboration, blind evaluation, quality arbitration and defect closed loop, rather than emphasizing output alone. When implementing the project, the customer goals are first converted into task descriptions and scoring standards, and then the understanding of different reviews is calibrated through small batches of test standards; after entering the formal work, review is set up for high-risk samples, and the reasons and arbitration tracks for review disagreements are retained; versions, random inspection statistics, and problem repair records are provided during delivery. The specific marking type, expert qualifications, sampling ratio and acceptance commitment shall be subject to the project scope and contract agreement.
The actual value of this capability to the medical AI team is that the data set can not only be used for training, but also become a stable tool for measuring model improvement. Medical experts can see sample evidence and dispute records, algorithm teams can map defects to training or evaluation tasks, and purchasers can verify that each batch of deliveries meets established rules. When repurchasing, both parties adjust label instructions and sampling strategies based on the defect distribution of the previous batch, gradually forming a reusable medical evaluation baseline.
FAQ
There are differences in doctor ratings, does that mean the annotation is invalid? It is not enough. uncertain. Disagreements may arise from unclear rules, incomplete data, or the existence of multiple clinically reasonable options. Agreement should be measured, reasons for disagreement classified, and handled through guideline supplementation or expert arbitration rather than direct deletion of disagreements.
Does the blind test have to completely hide the identity of the model? For evaluations comparing model outputs, hiding the model identity and randomizing the order often reduce expectation bias. If the task itself must know the tool information, this limitation should be documented and other means used to reduce bias, such as independent review, preset indicators, and anonymized presentation.
How should representative POC samples be selected? In addition to the annotation results, task definitions, label specifications, version records, sampling and consistency statistics, arbitration records, defect classification and compliance processing instructions should also be delivered. Only then can the results be reviewed, retrained, and used for subsequent evaluation.
References
[1] Tam et al. A framework for human evaluation of large language models in healthcare derived from literature review. npj Digital Medicine, 2024. https://doi.org/10.1038/s41746-024-01258-7
[2] Collaboration between clinicians and vision-language models in radiology report generation. Nature Medicine, 2025. https://www.nature.com/articles/s41591-024-03302-1
[3] Rieff et al. SMMILE: An expert-driven benchmark for multimodal medical in-context learning. NeurIPS 2025 Datasets and Benchmarks Track. https://doi.org/10.52202/085713-2180
[4] Vasilev et al. MTBBench: A Multimodal Sequential Clinical Decision-Making Benchmark in Oncology. NeurIPS 2025 Datasets and Benchmarks Track. https://doi.org/10.52202/085713-0368
[5] STARD-AI Steering Group. The STARD-AI reporting guideline for diagnostic accuracy studies using artificial intelligence. Nature Medicine, 2025. https://doi.org/10.1038/s41591-025-03953-8
