Publications |

Advancing Radiograph Representation Learning with Masked Record Modeling

Sun, 01 Jan 2023 00:00:00 +0000

Abstract

Modern studies in radiograph representation learning (R2L) rely on either selfsupervision to encode invariant semantics or associated radiology reports to incorporate medical expertise, while the complementarity between them is barely noticed. To explore this, we formulate the self- and report-completion as two complementary objectives and present a unified framework based on masked record modeling (MRM). In practice, MRM reconstructs masked image patches and masked report tokens following a multi-task scheme to learn knowledge-enhanced semantic representations. With MRM pre-training, we obtain pre-trained models that can be well transferred to various radiography tasks. Specifically, we find that MRM offers superior performance in label-efficient fine-tuning. For instance, MRM achieves 88.5% mean AUC on CheXpert using 1% labeled data, outperforming previous R2L methods with 100% labels. On NIH ChestX-ray, MRM outperforms the best performing counterpart by about 3% under small labeling ratios. Besides, MRM surpasses self- and report-supervised pre-training in identifying the pneumonia type and the pneumothorax area, sometimes by large margins. Code and models are available at https://github.com/RL4M/MRM-pytorch.

CoCycleReg: Collaborative Cycle-consistency Method for Multi-modal Medical Image Registration. Neurocomputing

Mon, 01 Aug 2022 00:00:00 +0000

Abstract

Multi-modal image registration is an essential step for many medical image analysis applications. Recent advances in multi-modal image registration rely on image-to-image translation to achieve good performance. However, the performance is still limited owing to the poor use of complementary regularization between image registration and translation, which is able to simultaneously enhance both parts’ accuracy. To this end, we propose CoCycleReg, a novel method that formulates image registration and translation in a Collaborative Cycle-consistency manner. Instead of dividing into two discrete stages, we unify the image registration and translation via cycle-consistency in an end-to-end training process, such that each part can benefit from the other one. To ensure the deformation fields’ reversibility in the cycle, we extensively introduce a novel dual-head registration network, consisting of one single backbone to extract the features and two heads to respectively predict the deformation fields. The experiments on T1-T2(MRI) and CT-MRI datasets validate that the proposed CoCycleReg surpasses the other state-ofthe-art conventional and deep learning approaches comprehensively considering the speed, accuracy, and regularity of deformation fields. In the ablation analysis, a method that sets the cycle-consistency Corresponding authors at: Department of Computer Science at School of Informatics, Xiamen University, Xiamen 361005, Chinaconstraints of registration and image-to-image translation separately is compared, and the results demonstrate the effectiveness of collaborative cycle-consistency. In addition, the improvement of image-to-image translation is also verified in further analysis. The code is publicly available at https://github.com/DopamineLcy/cocycle-reg/.

Breaking the Dilemma of Medical Image-to-image Translation

Fri, 01 Oct 2021 00:00:00 +0000

Abstract

Supervised Pix2Pix and unsupervised Cycle-consistency are two modes that dominate the field of medical image-to-image translation. However, neither modes are ideal. The Pix2Pix mode has excellent performance. But it requires paired and well pixel-wise aligned images, which may not always be achievable due to respiratory motion or anatomy change between times that paired images are acquired. The Cycle-consistency mode is less stringent with training data and works well on unpaired or misaligned images. But its performance may not be optimal. In order to break the dilemma of the existing modes, we propose a new unsupervised mode called RegGAN for medical image-to-image translation. It is based on the theory of “loss-correction”. In RegGAN, the misaligned target images are considered as noisy labelsaand the generator is trained with an additional registration network to fit the misaligned noise distribution adaptively.The goal is to search for the common optimal solution to both image-to-image translation and registration tasks. We incorporated RegGAN into a few state-of-the-art image-to-image translation methods and demonstrated that RegGAN could be easily combined with these methods to improve their performances. Such as a simple CycleGAN in our mode surpasses latest NICEGAN even though using less network parameters. Based on our results, RegGAN outperformed both Pix2Pix on aligned data and Cycle-consistency on misaligned or unpaired data. RegGAN is insensitive to noises which makes it a better choice for a wide range of scenarios, especially for medical image-to-image translation tasks in which well pixel-wise aligned data are not available. Code and data used in this study can be found at https://github.com/Kid-Liet/Reg-GAN.

Endoscopy Artefact Detection and Segmentation Using Deep Convolutional Neural Network.

Wed, 01 Apr 2020 00:00:00 +0000