A Unified Framework for Comprehensive Cardiac CT Segmentation and Phenotyping: Human-in-the-Loop Data Annotation, Vision Foundation Model Development, Multicenter Evaluation and Clinical Validation

Preprint

Link to Source: Preprint, Open-Access Dataset, code, CTAug library, released nnUZoo models, nnUZoo code

Authors: Pooya Mohammadi Kazaj, Leo Fridolin Weber, Wen Xie, Seyed Amir Ahmad Safavi-Naini, Anselm Stark, Giovanni Baj, Ali Mokhtari, Toshiya Yoshida, Christoph Ryffel, Taishi Okuno, Yoshihiro Akashi, Ronny R Buechel, Thomas Pilgrim, Waldo Valenzuela, George CM Siontis, Xiaowei Xu, Moritz Hundertmark, Stephan Windecker, Christoph Gräni, Isaac Shiri

Summary: A new open-source AI tool can automatically measure 14 distinct heart structures from routine CT scans with greater accuracy than existing tools. This development makes it practical to assess heart health on a larger scale using scans that people already receive. Following #OpenScience standards, the model, code, libraries, and dataset have been released to promote reproducibility and encourage innovative approaches to cardiac CT research.

CCT-FM development and evaluation pipeline. (a) Dataset and task overview. The left columns map each dataset to its geographic source: the labeled training set, the self-supervised pre training corpus (SSL Corpus), five external test sets (ExtTest-1 to -5), five clinical correlation cohorts (ExtClin-1 to -5), and three downstream task datasets (DownStr-1 to -3), each annotated with case count and modality. The right column maps these to tasks spanning the translational spectrum from benchside to real-world: architecture benchmarking, open access model benchmarking, generalization, clinical correlation, and population phenotyping. (b) Development pipeline. A human-in-the-loop process trains and refines segmentation through iterative training, inference, and expert correction; self-supervised pre training of a masked autoencoder, followed by supervised fine-tuning on 1K labeled CCT with CTAug within a U-Net, yields CCT-FM. The three-dimensional rendering shows the 14 segmented cardiac structures color-coded by anatomical category. Models and datasets feed an evaluation pipeline, after which the datasets, libraries, and models are publicly released and an in-house model is retrained to support clinical correlation and population phenotyping. Datasets released as open access and open-sourced models are marked with an open-source icon. Structure abbreviations: Aor, aorta; CA/Coronary Art, coronary arteries; EF, epicardial fat; IVC, inferior vena cava; LA, left atrium; LAA/LA App, left atrial appendage; LV, left ventricle; LVM, left ventricular myocardium; PA, pulmonary arteries; PF, pericardial fat; PV, pulmonary vein; RA, right atrium; RV, right ventricle; SVC, superior vena cava.

Comprehensive quantification of cardiac structures from computed tomography (CT) remains limited not by data availability but by the scalability of measurements, which makes routine use impractical. Here we present a unified framework for comprehensive cardiac CT segmentation and phenotyping that combines a human-in-the-loop annotation pipeline, a cardiac CT augmentation technique, and a self-supervised foundation model pre-trained on 60,000 unlabeled cardiac CT scans. Using this approach, we assembled the largest and most comprehensive expert-annotated cardiac CT segmentation dataset to date, comprising 1598 cases and 14 distinct cardiac structures (1000 for training, 598 for the external test set). Across five external datasets, the framework segmented all structures more accurately and comprehensively than existing open-source tools. Self-supervised pre-training improved labeling efficiency, with the most significant gains observed during external evaluation in the low-data regime. Benchmarking across convolutional, transformer, and state-space architectures showed comparable performance, indicating that data quality and pre-training, rather than architecture, drove accuracy. The framework was scaled to population-level phenotyping, with segmented anatomy that carries functionally relevant information about ventricular function and disease severity beyond demographic variables. By openly releasing the largest dataset with human labels, code, model weights, a CT augmentation library, and software, this work provides a reproducible foundation for opportunistic cardiac phenotyping from routinely acquired CT scans.

Whole-heart segmentation benchmark across in-house models and open-source tools. (a) Showcase of a representative external test case, showing the expert ground truth (GT) alongside CCT-FM and three open-source tools (Atlas, TotalSegmentator, and MOOSE). (b) Per-structure Dice coefficients for the seven in-house models (CCTFM, nnU-Net, nnU-Net-CTAug, SwinUMamba, Swin-UNETR, U²-Net, and U²-Net-Small) across all fourteen cardiac structures, where All-Organs denotes the mean across structures. (c) Per-structure Dice coefficients comparing CCTFM with the three open-source tools, restricted to the structures each tool produces.