Psychiatry AI training data: longitudinal records, expert labels and evaluation

Data structure abstracted from the sample

The workbook organizes several types of clinical forms, illustrating how psychiatric records connect across documents, time and data modalities. This public page describes field categories only; it does not publish patient-level content.

Care-document sequence

An index, admission assessment, initial and ongoing progress notes, interim summaries, consultations and discharge records can support links between an encounter, its documents and their timing.

Clinical evidence links

Laboratory and imaging records, medication orders and prescriptions can be linked to narrative notes by time and source. Missing or conflicting entries should be retained and flagged.

Safety-assessment fields

The schema includes safety-risk assessment forms. These tasks require clinician-defined labels and escalation workflows; extracting a documented assessment is not a diagnosis or an automated intervention.

A working definition of a high-quality psychiatry dataset

Quality is not a synonym for “many records” or “cleaned data.” It is a versioned data product with lawful provenance, a defined task, reviewable reference labels, traceable annotation, leakage-resistant evaluation and explicit limits.

• Task validityDefine the research or model-evaluation objective, intended users, inputs and outputs, exclusions, and clinical decisions the dataset must not support.
• Verifiable provenance and governanceDocument source, authorization, applicable ethics pathway, purpose, access roles, retention, de-identification method and deletion. Health data is sensitive personal information under Chinese law; masking alone does not establish a lawful basis.
• Psychiatry-aware semanticsPreserve symptom presence, timing, source, negation and uncertainty, patient report versus clinician observation, comorbidity and context. Record the version and mapping rules for diagnostic terms and codes.
• Transparent reference standardDistinguish a diagnosis recorded in the chart from an expert-adjudicated reference label for a research task. Link labels to evidence, guideline version, review status and uncertainty.
• Trustworthy longitudinal splitsSplit training, tuning and test sets by patient. For prognosis, define a feature cut-off and isolate future information. External site or time-based validation tests transportability.
• Reproducible quality evidenceKeep a data dictionary, annotation manual, versioning, deduplication, missingness definitions, adjudication, audit sampling, data card and change log. Report metrics with denominators and scope.

Research and evaluation tasks the design can support

These are candidate tasks, not claims that the sample has been annotated or that any model has achieved a result. Each task needs its own rights check, label feasibility review and clinical boundary.

Clinical narrative structuring

Extract symptoms, time expressions, negation, uncertainty, source attribution and document location; evaluate entities, relations and timelines, not classification accuracy alone.

Longitudinal course reconstruction

Organize admission, progress notes, consultations, tests and discharge plans into evidence-linked event sequences; check order, time and cross-document consistency.

Document consistency and summaries

Compare documented diagnoses, tests, orders and plans across notes. Require summaries to cite supporting evidence and assess omissions, contradictions and unsupported additions.

Safety-related information extraction

Measure whether documented safety assessments are identified accurately, uncertainty is preserved and human review is triggered. This must not replace clinical judgement, crisis services or local care pathways.

Psychiatry model training: match the objective to the data form

A psychiatric record is not automatically a training example. Each modeling route needs its own data structure, supervision and isolation controls. These are project-scoped options, not claims that the sample already contains labels or production-scale data.

Continued pretraining / domain adaptation

Use only a corpus whose authorization, minimization and de-identification have been reviewed. Preserve document type, source and time granularity, deduplicate text and assess memorization risk; one workbook cannot establish training scale.

Supervised fine-tuning (SFT)

Raw notes are not instruction examples. Build task-specific inputs and target outputs, evidence locations, negation and uncertainty labels, and unanswerable cases; obtain expert review and define acceptance separately for each task.

Information extraction and classification

Define structured labels for symptoms, onset, course, medication events, tests and document source. Report precision, recall, F1 and class distribution by label unit; a diagnosis recorded in a chart is not automatically a gold standard.

Longitudinal outcome or risk research

Specify index time, observation and outcome windows, and censoring rules. Use only information available before the prediction point, split by patient, and validate across time or sites. This is research evaluation, not a clinical risk-management capability.

Retrieval and evidence-grounded summaries

For RAG or summarization, preserve source passages, document location and time. Assess citation support, omissions, contradictions, unsupported additions and abstention. Keep retrieval corpora, training data and sealed test sets separately governed.

Product pathways for different project teams

Start with the operational question, then combine data preparation, expert annotation and model evaluation into stage-gated deliverables. Each project first verifies permissions, label feasibility and available scope.

Hospital research and specialty teams

Translate a protocol into inclusion rules, field mappings, longitudinal events and outcome variables. Deliver a reviewable dictionary, data card and pilot findings. The authorized institution determines the applicable ethics and data-use pathway.

Psychiatry model developers

Build clinical-text structuring, SFT or task-tuning data around the model objective, while keeping tuning data separate from a sealed test set. Deliver an annotation guide, review records and task-level quality report.

Digital mental health and therapeutics research

Potential text tasks include symptom change, treatment-course documentation, adherence and adverse-event extraction. Study endpoints, definitions and permissions require separate agreement; the data product is not treatment advice or proof of efficacy.

Model quality and safety teams

Create a private benchmark, clinician rubric, error taxonomy and version-regression evaluation. Inspect missed signals, over-inference, uncertainty expression and human-review triggers in higher-risk tasks. Evaluation is not clinical certification.

A practical build and evaluation pathway

1. 1
   Lock the task first

   Psychiatrists, psychology professionals, data-governance staff and ML teams define the objective, label unit, inclusion rules, time window, risk tier and prohibited uses.

2. 2
   Verify data rights

   Check provider authorization, applicability of ethics review or exemption, consent requirements, minimum necessary fields, access controls and retention/deletion. Ethics determinations belong to the authorized committee.

3. 3
   Create expert reference labels

   Train annotators, run independent double annotation and use blinded adjudication for disagreement. Report raw agreement and an appropriate statistic (for example, a kappa-family measure for categorical labels or ICC for continuous ratings), with confidence intervals and distributions.

4. 4
   Benchmark without leakage

   Split by patient, freeze test items and protocol, and prevent documents, time points or templates from the same patient leaking across sets. Record model version, prompts, tools and run conditions.

5. 5
   Validate externally and monitor

   Where authorized, test across sites, time periods and subgroups. Evaluate calibration, abstention, error types, workflow impact and drift. Clinical use requires the relevant clinical studies and regulatory assessment.

How LumeSage can turn the specification into a product

A single sample becomes a procurable, reviewable and continuously improvable task product only when data, experts and evaluation form one controlled loop:

Data card

Specify source category, dictionary, inclusion rules, time granularity, modality links, de-identification/access method, use limits, quality metrics and known biases.

Expert production

Assign psychiatry and psychology expertise plus annotation and QC roles. Define the manual, training, double review, adjudication, sampling and rework logs for each task. Verify credentials and scope per project.

Evaluation benchmark

Separate subtasks such as field extraction, temporal relations, note consistency, summary faithfulness and safety-related information recognition; use blinded testing, fixed versions, error taxonomy and retesting.

Controlled delivery

Record samples, license scope, counting unit, acceptance criteria, storage/access environment, retention and deletion in the data card and contract. A catalogue entry is not available inventory.

Quality gates and acceptance evidence

No single accuracy score establishes a high-quality dataset. Agree on metric definitions, denominators, sampling and stop conditions before a pilot, then set acceptance criteria for the task and risk level.

Provenance and permissions

Verify source, authorization chain, applicable ethics pathway, purpose, de-identification, access roles, retention and deletion. Preserve version history and audit evidence.

Semantic and temporal integrity

Measure document coverage, key-field missingness, usable timestamps, patient report versus clinician observation, polarity and uncertainty, cross-note conflicts and duplicates. Mark missing and conflicting evidence explicitly.

Labels and expert agreement

Train reviewers, run independent double annotation, then adjudicate disagreements blind. Report raw agreement and suitable statistics with confidence intervals, distributions and adjudication rate; thresholds are set from the pilot and contract.

Leakage and transportability

Split by patient and test for near-duplicate, template and future-information leakage. Where authorized, add temporal, site and key-subgroup validation, and disclose sample composition and limits.

Task and safety metrics

Use precision/recall/F1 for extraction, evidence support and omission measures for summaries, and calibration for prediction research. High-risk information tasks also examine misses, false alarms, abstention and human-escalation burden.

Methods and governance references

These sources inform evaluation and governance design. Reporting guidance, ethics guidance and empirical findings serve different purposes; none certifies this product or endorses its performance.

• WHO (2024): ethics and governance guidance for large multimodal models in health. World Health Organization (2024). Ethics and governance of artificial intelligence for health: large multi-modal models.
• BMJ (2024): TRIPOD+AI reporting guidance for clinical prediction models. It improves reporting transparency; it is not a dataset-quality certification. Collins et al. (2024). TRIPOD+AI statement. BMJ 385:e078378.
• Nature Medicine (2025): MedHELM organizes medical-model evaluation around practical task categories. Nature Medicine (2025). Holistic evaluation of large language models for medical tasks with MedHELM.
• BMC Medical Informatics and Decision Making (2025): an example of NLP research on symptom-related information in psychiatric clinical notes. BMC Medical Informatics and Decision Making (2025). NLP to identify suicidal ideation and anhedonia in major depressive disorder.
• China National Health Commission (2023): ethics review measures for life-science and medical research involving humans. National Health Commission of China (2023). Ethics review measures for life-science and medical research involving humans.
• China Personal Information Protection Law, Articles 28–30: rules for processing sensitive personal information, including health data. Personal Information Protection Law of the PRC, Articles 28–30.
• npj Digital Medicine (2026): PsychiatryBench proposes a multi-task evaluation framework for psychiatry. Its task decomposition is a design reference, not evidence of performance on this project. Lenz et al. (2026). Benchmarking large language models against practicing clinicians on psychopathological assessment. npj Digital Medicine 9, 518.
• npj Digital Medicine (2026): a structured psychopathology-assessment study compared models, clinicians and expert consensus. It used a small set of simulated interviews, underscoring the need to score “not assessable” responses and error types without generalizing to real-world cohorts. PsychiatryBench: a multi-task benchmark for LLMs in psychiatry. npj Digital Medicine (2026).
• Nature Machine Intelligence (2026): a psychiatry-adapted model study illustrates specialty-corpus and multi-task adaptation directions. Its training data, permissions and findings do not apply to this sample. A domain-adapted large language model to support clinicians in psychiatric clinical practice. Nature Machine Intelligence (2026).