Enhanced Contrastive Learning with Multi-view Longitudinal Data for Chest X-ray Report Generation

           

• 2026-02-08 Compute more metrics, including BertScore, SemScore, 1/RadCliQ-V1, and RATEScore
• 2025-05-09   Upload poster
• 2025-03-16   Release checkpoints for MIMIC-ABN and Two-view CXR
• 2025-03-01   Upload official code and checkpoints for MIMIC-CXR
• 2025-03-01   Release generated-radiology-reports — labels = reference reports, report = generated reports

# create virtual environment
conda create -n mlrg python=3.9.0

# install dependencies
pip install -r requirements.txt

• torch==2.3.1+cu118
• transformers==4.43.3 (As stated in Issue #4, the transformers version should be maintained to prevent potential problems. Credit goes to @Andy for this clarification)
• torchvision==0.18.1+cu118
• radgraph==0.0.9

Please refer to requirements.txt for more details.

• MIMIC-CXR / MIMIC-ABN — PhysioNet, with data systematically organized under root directories labeled p10 through p19, maintaining consistency with MIMIC-CXR's default configuration.
• IU X-ray — NIH, its root directory is the NLMCXR_png.
• Two-View CXR — aggregated studies with two views from MIMIC-CXR + IU X-ray (arXiv   )

files/
├── p10
    └── p10000032
            └── s50414267
               ├── 02aa804e-bde0afdd-112c0b34-7bc16630-4e384014.jpg
               └── 174413ec-4ec4c1f7-34ea26b7-c5f994f8-79ef1962.jpg
├── p11
├── p12
├── p13
├── p14
├── p15
├── p16
├── p17
├── p18
├── p19
└── NLMCXR_png
   ├── CXR1_1_IM-0001-3001.png
   ├── CXR1_1_IM-0001-4001.png
   └── CXR2_IM-0652-1001.png

• MIMIC-CXR and MIMIC-ABN: PhysioNet.
• Two-view CXR: HuggingFace 🤗.

• To simplify usage, we have organized multi-view longitudinal data using the study_id. The processed datasets—MIMIC-CXR, MIMIC-ABN, and Two-view CXR—are available on HuggingFace 🤗 (PhysioNet authorization required). Note that the IU X-ray dataset (NLMCXR_png) does not include previous visit data due to the absence of study_id.
• MIMIC-CXR: five_work_mimic_cxr_annotation_v1.1.json
• MIMIC-ABN: mlrg_mimic_abn_annotation_v1.1.json
• Two-view CXR: mlrg_multiview_cxr_annotation_v1.1.json
• View Position for all datasets: five_work_mimic_cxr_view_position_v1.1.json

def compute_performance_using_generated_reports():
    from tools.metrics.metrics import compute_all_scores, compute_chexbert_details_scores
    mimic_cxr_generated_path = 'generated-radiology-reports/MIMIC-CXR/test_reports_epoch-1_20-10-2024_16-28-28.csv'
    mimic_abn_generated_path = 'generated-radiology-reports/MIMIC-ABN/test_reports_epoch-1_23-10-2024_10-25-20.csv'
    twoview_cxr_generated_path = 'generated-radiology-reports/Two-view CXR/test_reports_epoch-0_25-10-2024_11-38-35.csv'
    args = {
        'chexbert_path': "/home/miao/data/dataset/checkpoints/chexbert.pth",
        'bert_path': "/home/miao/data/dataset/checkpoints/bert-base-uncased",
        'radgraph_path': "/home/miao/data/dataset/checkpoints/radgraph",
    }
    for generated_path in [mimic_cxr_generated_path, mimic_abn_generated_path, twoview_cxr_generated_path]:
        data = pd.read_csv(generated_path)
        gts, gens = data['labels'].tolist(), data['report'].tolist()
        scores = compute_all_scores(gts, gens, args)
        print(scores)

{
    'BertScore': 0.5716221332550049,
    'SemScore': 0.4368664622306824,
    '1/RadCliQ-V1': 1.0102079556023098,
    'RATEScore': 0.5668122046732644,
    'green': 0.353027779,
    'chexbert_5_micro_f1': 0.5503549017590783,
    'chexbert_5_macro_f1': 0.4862237881570195,
    'chexbert_all_micro_p': 0.5489597467209407,
    'chexbert_all_micro_r': 0.467591254935953,
    'chexbert_all_micro_f1': 0.5050189837208093,
    'chexbert_all_macro_p': 0.4399492462801775,
    'chexbert_all_macro_r': 0.354060820803069,
    'chexbert_all_macro_f1': 0.3641635446370755,
    'BLEU_1': 0.41114996799739173, 'BLEU_2': 0.2769778918508422,
    'BLEU_3': 0.20362264525354418, 'BLEU_4': 0.1582088781713785,
    'METEOR': 0.17633810974007486, 'ROUGE_L': 0.3195399064699496, 'CIDer': 0.3599887171235284
}

1. Download checkpoints for architecture and metrics.

• For CE metrics calculation: chexbert.pth, radgraph, and bert-base-uncased.
• For model initialization: microsoft/rad-dino (image encoder), microsoft/BiomedVLP-CXR-BERT-specialized (text encoder), distilbert/distilgpt2 (define text generator), and cvt2distilgpt2 (initialize text generator).
• Checkpoint directory: Place all checkpoints in a local directory (e.g., "/home/data/checkpoints"), and configure the --ckpt_zoo_dir /home/data/checkpoints argument in the corresponding script/**/**.sh file.

2. Conducting Stages 1 and 2

# Stage 1: Multi-view Longitudinal Contrastive Learning
cd script/MIMIC-CXR
bash run_cxr_pt_v0906_fs.sh

# Stage 2: Chest X-ray Report Generation based on Patient-specific Prior Knowledge
cd script/MIMIC-CXR
bash run_cxr_ft_mlrg_v1011.sh

If you use or extend our work, please cite our paper at CVPR 2025.

@InProceedings{Liu-2025-CVPR,
  author={Liu, Kang and Ma, Zhuoqi and Kang, Xiaolu and Li, Yunan and Xie, Kun and Jiao, Zhicheng and Miao, Qiguang},
  booktitle={2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, 
  title={Enhanced Contrastive Learning with Multi-view Longitudinal Data for Chest X-ray Report Generation}, 
  year={2025},
  volume={},
  number={},
  pages={10348-10359},
  doi={10.1109/CVPR52734.2025.00968}
}

• cvt2distilgpt2 — adapted from R2Gen
• EVOKE — dataset & implementation references