Abstract: Turning “difficulty” into verifiable measurements
Medical artificial intelligence is moving from knowledge question and answer to paper understanding, cross-modal evidence integration, clinical text processing and agent workflow. The model's correct answer to a difficult problem may come from memory, retrieval, tool invocation, accidental guessing, or true multi-step reasoning; if the list cannot distinguish these mechanisms, it will be difficult to answer the questions that purchasers and R&D teams are most concerned about: On what types of tasks is the model reliable, where did the errors occur, and can the results be reproduced?
HLE (Humanity’s Last Exam) provides an important methodological reference: using difficult academic questions raised by experts to evaluate the model’s ability on closed, verifiable tasks. The HLE benchmark introduced in its Nature paper contains 2,500 interdisciplinary questions, emphasizing that the questions are original, the conditions are clear, the answers are verifiable, and private questions are reserved to observe public set overfitting. It is a general academic competency benchmark, not a medical clinical validation tool, and should not be mistaken for clinical certification.[1] The draft requirements provided by
users plan 500 questions each for Biology/Medicine, Other/Trivia, and Engineering, totaling 1,500 questions. Medicine and life sciences cover molecular and cell biology, genomics, immunity, neuroscience, bioinformatics, translational medicine, clinical diagnosis, and pathology. This scale and classification is a requirement of the draft project and is not part of the HLE official dataset composition or official expansion plan. To transform it into professional delivery, you need to first define the purpose of the assessment, and then control the question quality, medical source, difficulty screening and statistical uncertainty.
This article proposes a set of independent "medical HLE-style scientific research reasoning evaluation" solutions. It evaluates the model's ability to handle expert-level medical research questions with traceable sources and determinable answers; it does not extrapolate test performance to patient diagnosis and treatment results. LumeSage can provide project-based support around medical data governance, question design collaboration, medical expert annotation, quality control, model operation and evaluation reports, but the specific personnel, resources, question volume, cycle and capability boundaries need to be confirmed based on the contract, authorization and expert resources before the project is started.
1. Why does the medical field need HLE-style high-difficulty evaluation?
1. Regular exam scores are no longer enough to answer R&D questions
The multiple-choice test has the advantages of low cost, fast automatic scoring, and intuitive results, but it can easily test knowledge familiarity and option recognition ability. Medical research questions often require understanding experimental designs, identifying confounding factors, interpreting applicable populations, tracking levels of evidence, or combining conditions from multiple sources. Just because a model answers a single knowledge point correctly does not mean that it can explain where the conclusion comes from and what conditions are indispensable, nor does it mean that it performs stably under different institutions and data distributions.
HLE-style questions can focus the evaluation on expert-level limited-answer tasks: such as deriving a standard term, variation category, experimental explanation, or explicit numerical value based on given research conditions, rather than asking the model to write a long essay that cannot be stably scored. Medical questions must still limit the questions to academic research or explicit hypothetical situations, and do not provide diagnosis, treatment or medication suggestions for real individuals. This is consistent with the appendix’s requirements for non-choice questions, unique final answers, conditional completeness and multi-step reasoning.
2. The difficulty must come from the reasoning structure, not the stacking of uncommon words
“Doctoral level” cannot be defined solely by the names of unpopular diseases, complex abbreviations, or details in a single paper. High-quality questions should have a clear reasoning load: combining at least two or more relevant knowledge links, requiring the model to identify question conditions, apply research methods or professional concepts, and obtain a verifiable final answer. If the question requires guessing the author's intention, relies on unprovided diagrams, has multiple reasonable answers, or if key facts change as the guide is updated but the version is not stated, then it may measure ambiguity rather than ability.
Therefore, the question blueprint should first specify what each type of task hopes to measure: knowledge integration, research design analysis, mechanism inference, result interpretation, statistical and methodological judgment, molecular or clinical terminology normalization, etc.; then use representative sample questions to calibrate the question difficulty and scoring rules. Each question must have a traceable chain of "question stem - reasoning basis - final answer - source", and if necessary, acceptable answers, units, synonyms and scoring boundaries.
3. The important asset of medical rankings is credibility, not championship ranking
HLE emphasized closed and verifiable questions in the design, and improved the quality of the questions through multi-stage review; the paper also discussed the reality that experts disagree on some questions. This reminds the medical evaluation team: the participation of experts in the field does not automatically mean that the answers are natural and uncontroversial. The questions and reference answers still require quality control and arbitration.[1] The subsequent HLE-Verified preprint reported the systematic verification, revision and uncertainty labeling of the original question, and observed changes in model accuracy after the revision; this is a preprint research result, and it cannot be directly inferred that the medical question bank will also produce changes of the same magnitude, but it highlights the important risk that "the benchmark itself may be noisy".[2]
2. Convert the draft requirements into a medical evaluation blueprint
1. Reserve the planned amount of 500 medical questions and do a hierarchical quota design first
It is recommended to use 500 questions as the project planning amount of the medical and life sciences question bank, rather than mechanically dividing them equally among the eight secondary subjects from the beginning. The project party, medical experts and algorithm team jointly produced a blueprint to clarify the proposed testing tasks in each direction, sample size basis, question type restrictions, source levels, required professional reviewers, and whether it involves time-sensitive guide content. The eight stable directions listed in the appendix can serve as a starting point: Molecular and Cellular Biology/Biochemistry, Genetics and Genomics, Microbiology and Immunology, Neuroscience and Biophysics, Ecological Evolution and Marine Biology, Computational Biology and Bioinformatics, Basic and Translational Medicine, and Clinical Medicine/Diagnosis/Pathology.
The allocation of question volume should reflect the coverage of the plan and should not imply that each subdivision direction already has the same number of question sources or equivalent expert resources. The hierarchical report must also display the "design quota, actual number of available questions, number of eliminations due to quality issues and areas not covered". For subdivision directions with too few samples, only exploratory results are reported and seemingly precise independent rankings are not given. Before starting the project, the target users should also be clearly defined: scientific research model teams, medical data purchasers, medical AI product teams, or internal algorithm research and development. The evaluation question types and report granularity required by different users are different.
2. The question should be original, the facts can be quoted, and the question stem and reasoning chain cannot be copied.
You can cite facts from reliable papers, guides, authoritative databases or standards as basis, but the wording of the question stem, situation combinations and reasoning paths should be original and cannot directly copy public question banks, essay exercises, textbook exercises or online content. The DOI, version, publication date, access date and specific supporting arguments must be registered for conclusions in public sources; if the source has errata, retraction or guideline update, it must be reviewed for updates.
Source management should be further refined: it is best to indicate the main source and independent cross-check source for a question; record the paragraphs, tables or chapters that support the answer; retain the first entry of the question into the database, the latest review, the current status and the version number. If the question requires inferring the results of the paper, the question must provide necessary conditions or include quoted materials in the question stem to prevent the model from "accidentally answering" based on only a certain version of training knowledge. Biomedical domain database versions, statistical definitions, reference genome versions, drug nomenclature, and disease classifications may all affect the answer and must be frozen as conditions or written out clearly.
3. The question bank should take into account both research issues and medical safety boundaries.
recommends the priority development of four categories of tasks: first, research methodological analysis based on clear materials, such as identifying what conclusions can be supported by the research design; second, biological mechanism and cross-level reasoning, such as linking molecular changes to experimental observations; third, structured explanations in genetics, bioinformatics, and pathology, but with clear reference to versions, thresholds, and output terms; fourth, academic situational reasoning in clinical research or diagnostics, providing complete hypothesis conditions and requiring the answer to a limited professional question.
All cases should be synthetic scenarios, original adaptations of publicly available research data, or data that has been subject to compliance authorization, de-identification and use approval. Privacy, copyright, sensitive information, and re-identification risks should be reviewed before publicly releasing questions and model responses. Any personal health description must not be packaged as real consultation; on the subject of disease severity and treatment decision-making, its educational or research purpose should be made clear, and individualized treatment plans that can be directly implemented should be avoided.
3. Manage question quality and model scores separately
1. 98% is the quality target, not the model pass rate The
requirement draft states that the overall solvability rate and answer accuracy rate of each first-level field shall not be less than 98%. This should be converted into a question quality threshold: for sampling or full review, at least 98% of the questions must meet sufficient conditions, the reference answer can be reviewed, the reasoning chain is consistent with the answer, the source is valid, and the scoring rules are clear. The project should report the numerator, denominator, sampling method and confidence interval at the same time, and cannot just report a percentage; if there are unqualified questions among the 500 questions, they need to be repaired, replaced or removed from the main evaluation set according to pre-set rules.
model pass rate ≤62.5% is the difficulty filtering condition. The answer is "How many correct answers can a specified model get under certain settings?" and whether the question itself is correct are two different things. Do not take "wrong answers to the model" as evidence that the quality of the question is up to standard, nor should you mistake high difficulty for more scientific questions. Difficulty screening must be done after reviewing answers for validity; otherwise, ambiguities, missing conditions, outdated knowledge, or missed decisions by the grader will be rewarded.
2. First resolve the field conflicts between 8 and 24 runs
Attachment field one requirement model_response_counts summarizes 24 model responses, another requirement model_responses has exactly 8 items, which also indicates that each of the three models has been run 8 times. These three cannot be true at the same time. It is recommended to unify as follows: pre-freeze 3 model versions for each question, and run each model independently 8 times, so the original response array should have exactly 24 items; at the same time, increase model_id、model_version、run_id、attempt、prompt_hash、tool_policy、decoding_config、timestamp、raw_response、normalized_answer、is_correct、grader_version and other fields. Count correct, incorrect, rejected and invalid outputs respectively according to the model. The sum of all sub-item counts should be equal to 8; the sum of the total questions should be equal to 24. If the procurement actually requires only 8 times for one model, the requirements for 8 times for each of the three models and the "24 times summary" should be deleted and cannot be temporarily explained during delivery.
It is recommended to record each run as a line of JSONL and use stable question_id Aggregates questions, reference answers and sources, or nests 24 responses in a question object and writes it in the schema minItems=maxItems=24. An executable structure includes: question object storage question_id/domain/sub_domain/question/reasoning_steps/answer/reference_sources/rubric/versionThe running object saves the model and calls parameters, responses, normalized answers and scores. should be clear reasoning_steps is a reference solution chain for reviewers, not a hidden chain of thinking that must be exposed by the model; evaluations should be scored based on final answers and observable explanation evidence to avoid relying on unverifiable reasoning text within the model.
3. Eight repetitions can only be used for preliminary screening and cannot claim fine ranking.
When each model has only 8 runs per question, the single question pass rate can only take discrete values such as 0, 12.5%, 25%, etc. If the threshold is greater than 0 and no more than 0.625, the actual retention range is 1 to 5 times correct. This setting can be used as a pre-screening rule, but cannot reliably differentiate between adjacent models or accurately estimate overall capabilities. It is recommended to report the numerator/denominator and binomial ratio confidence interval of each model, and to stratify and summarize by topic, field and task type; formal comparison should be based on a sufficient number of topics and a preset statistical plan, rather than based on a small number of repeated sampling to generate the overall ranking order.
needs to freeze the model name and exact version, access date, system prompt words, user prompt word template, context and search materials, tool permissions, temperature and sampling parameters, maximum token, timeout and retry rules before running. Separate tracks should be set up for models with retrieval/tools and purely closed-book models because they use different information boundaries. Record the original request and response, failure status, cost and operation log, and disclose the evaluation conditions in public reports to facilitate recurrence.
4. Scoring and reporting: not just giving an accuracy rate
For single standard answer questions, deterministic scoring is preferred: standardized capitalization, spaces, units, synonymous terms and agreed formats, but all acceptable variations must be listed before evaluation and verified by professional reviewers. Numerical questions specify tolerances and significant digits; names of genes, proteins, drugs, and diseases specify databases or naming standard versions. Medical correctness cannot be left to an uncalibrated generic LLM judge alone. If necessary for open-ended or complex explanation questions, a fine-grained rubric based on source evidence should be developed, blindly and independently scored by at least two domain reviewers, arbitrating disagreements, and reporting agreement and arbitration ratios.
It is recommended that the report include at least: overall accuracy and confidence intervals, scores at each domain/task level, question difficulty distribution, rejection and formatting error rates, answer calibration and selective response performance, source evidence correctness, out-of-time or private set performance, repeated run fluctuations, error types and severity, as well as inference latency, tool calls and costs. In particular, medical research tasks must distinguish between different failure modes such as “the conclusion is correct but the basis is wrong”, “the answer is correct but the premise is not met”, “insufficient information for correct identification”, and “overconfidence in giving the wrong answer”. The public page of the
list can use comprehensive scores, but it must provide transparent weights, scope of application, missing items processing methods and confidence intervals; it is more reliable to provide a multi-dimensional ability portrait to avoid a single total score from covering up safety, reproducibility or specific subspecialty shortcomings. Methods, sample questions and statistical codes can be made public, but controlled private questions, out-of-time questions or rolling update questions should be retained to reduce pollution and targeted optimization. There should be clear data management rules for public and hidden question banks, and scientific research transparency should not be sacrificed for the sake of "preventing list fraud."
5. Scientific boundaries: HLE-type evaluation cannot replace clinical evaluation
HLE-style evaluations are best suited to answer a limited question: Can the model solve a difficult, closed, and verifiable academic task under given conditions? It cannot alone demonstrate a model's diagnostic accuracy, clinical benefit, physician workflow fit, medical device compliance, or long-term safety in real patients.
The verification of medical AI needs to be expanded step by step along uses and risks. The five-stage framework, published in npj Digital Medicine in 2026, proposes a staged evaluation from technical validation, operational robustness, controlled interactions, clinical evidence to real-world integration, emphasizing that multiple layers of validation are still needed between model performance and actual clinical value.[3] Medical large model evaluation studies such as CLEVER demonstrate the path for blind, structured expert evaluation of newly constructed cases and responses by practicing doctors, which can be used as a reference for open clinical text evaluation, rather than the only standard.[4]
Therefore, the medical HLE-style question bank should be positioned as a special evaluation layer for scientific reasoning ability, which can complement medical knowledge Q&A, open safety evaluation, simulation workflow, external data verification and prospective research. If patient-level data, real medical records or clinical recommendation tasks are used, additional ethical review, authorization, privacy protection, statistical analysis plan and corresponding clinical evaluation need to be completed. Each indicator in the list should have a clear level of evidence, and laboratory scores cannot be marketed as a guarantee of clinical results.
6. Langhui’s implementable solution: from data quality to evaluation operation
In response to the multi-field question bank and running record requirements proposed in the attachment, Langhui can organize a project-based workflow by stage
1. Demand and Risk Blueprint Clarify the purpose, tested model, subject boundaries, case sources, medical safety boundaries, sample quotas and acceptance criteria with customers and domain experts to form a signable question blueprint and data dictionary.2. Original question and evidence package organizes the question, reference solution basis, final answer, source link/DOI, version date, acceptable answers, scoring rules and copyright status into structured records. Facts that cannot be confirmed are first marked as pending and will not enter the formal collection.3. Expert review and disagreement arbitration Medical or life science experts with appropriate professional orientation will review the question meaning, answers and sources respectively, record the reasons for the disagreement, and the designated arbitrator will complete the final review. Different qualifications and specialty coverage should be disclosed truthfully, and general “expert review” should not be used to replace staffing and review records.4. Data and Question Bank Quality Control performs field verification, unique ID, version freezing, duplicate and near-duplicate screening, source accessibility check, question leakage check, authorization and de-identification verification, and outputs a problem list and revision records.5. controlled model operation runs the model according to the frozen configuration, saving requests, outputs, model meta-information, tool trajectories, times and costs; closed-book and tool-assisted tasks are divided into tracks, and are scored deterministically according to the question rules, and exceptions are reviewed manually in a blind manner.6. evaluation report and retest plan outputs reviewable stratification indicators, interval estimates, model error cases, task coverage boundaries and suggested supplementary question types; set up hidden set management, periodic retesting, version changes and pollution monitoring mechanisms.
This solution is connected with Langhui’s open business positioning in the direction of medical data collection, data standardization, medical annotation, quality control and evaluation services. The truly deliverable capability should not only be reflected in "how many questions there are", but rather whether the source, authorization, judgment caliber, expert process, model operation and acceptance evidence can be connected into a chain. Based on its own data governance, annotation process and evaluation engineering capabilities, Langhui can help customers break down abstract question bank requirements into implementable work packages; each project should still be based on actual expert resources, data authorization, delivery conditions and written scope. Understand LumeSage Medical Data Service 与 Dataset Service。
7. Recommended project milestones and acceptance criteria
recommends making a small-scale, interdisciplinary feasibility sample first, rather than rolling out 500 questions at once. The first stage is to complete the question blueprint, field specifications and a small number of sample question verifications to check whether different experts can answer and judge according to the same rules; the second stage is rolling production and blind review by field to observe the reasons for elimination and expert consistency; the third stage is to freeze the data set, model operation configuration and statistical plan, and complete the formal evaluation; the fourth stage is to release a report with method description and version update rules. Each stage sets conditions for continuation, rework or suspension.
Acceptance suggestions are divided into six categories: field and format completeness, question solvability and answer source validity, expert review and arbitration record completeness, model operation record traceability, scoring script and spot check consistency, hierarchical report and statistical explanation completeness.98% can be used as the project target for the question bank quality threshold, but it must be agreed upon first whether sampling or full review, error definition, number of allowable repairs, denominator caliber, and treatment methods for questions that cannot be covered. The model difficulty threshold is listed separately as a screening condition for question selection and cannot be mixed with the question bank quality acceptance to form a total score.
In terms of commercial execution, the work package can cover question design and source verification, expert review, structured data sorting, operating platform/script, statistical analysis, reporting and continuous maintenance. The purchaser should specify in the contract the number of questions, field ratio, expert qualifications and investment methods, delivery format, intellectual property ownership, data usage scope, confidentiality and security responsibilities, acceptance samples and change management of each package. This type of split makes workload and quality responsibilities transparent, and also makes it easier for projects to start with small-scale pilots and determine subsequent investment based on verification results.
Conclusion: Promote responsible iteration of medical AI with reviewable problems
The medical field needs more difficult evaluation questions, but it also needs questions that can withstand questioning, clear scoring, reproducible operating environments and restrained interpretation of results. The inspiration of HLE is not just to make the questions difficult, but to transform expert knowledge into measurement tools with boundaries, answers, and audit records. For medical scenarios, source validity, medical security, privacy and clinical evidence stratification should also be added.
The HLE-style medical scientific research reasoning evaluation proposed by Lang Hui is a construction path that can be discussed, piloted, and gradually expanded based on evidence. The next step should be to start with a demand blueprint and a small number of interdisciplinary sample questions, use dual expert review and freeze running verification methods, and then decide on the question bank size, disclosure strategy and list operation method. The professionalism of the list is ultimately reflected in the fact that each score clearly explains: what is measured, what is the basis for it, where the error may come from, and what cannot be inferred. Is
FAQ
References
[1] HLE Contributors Consortium. A benchmark of expert-level academic questions to assess AI capabilities. *Nature*. 2026;649:1139–1146. doi:10.1038/s41586-025-09962-4；HLE official project page。
[2] Zhai W, et al. HLE-Verified: A Systematic Verification and Structured Revision of Humanity’s Last Exam. arXiv:2602.13964, 2026. The preprint should be interpreted according to the preprint evidence level.论文。
[3] A five-phase evaluation framework for diagnostic and predictive medical artificial intelligence. *npj Digital Medicine*. 2026; published September 3. doi:10.1038/s41746-026-03155-7。
[4] Clinical Large Language Model Evaluation by Expert Review (CLEVER): Framework Development and Validation. *JMIR AI*. 2025;4: e72153. doi:10.2196/72153。
[5] Center for AI Safety, Scale AI. Introducing HLE-Diamond. 2026-09-22. official project description. What this project announced is a refined subset of HLE, which is not the medical benchmark mentioned in this article, nor does it mean official endorsement of this plan.
