CROSS-MODALITY IMAGE SYNTHESIS

• US 20190311228A1
• Filed: 01/28/2019
• Published: 10/10/2019
• Est. Priority Date: 04/05/2018
• Status: Active Grant

• Current Assignee: Siemens Healthineers AG (Siemens AG)
• Original Assignee: Siemens Healthcare GMBH (Siemens AG)
• Inventors: Zhao, Yu, Gao, Yimo, Liao, Shu, Zhao, Liang, Yan, Zhennan, Zhan, Yiqiang, Zhou, Xiang Sean
• Granted Patent: US 10,803,354 B2
• Time in Patent Office: Days
• CPC Class Codes: A61B 6/03 Computed tomography [CT] echo-tomography A61B8/14
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06T 11/008 Specific post-processing after tomographic reconstruction, e.g. voxelisation, metal artifact correction
• CPC Class Codes: G06T 2207/10081 Computed x-ray tomography [CT]
• CPC Class Codes: G06T 2207/10088 Magnetic resonance imaging [MRI]
• CPC Class Codes: G06T 2207/10104 Positron emission tomography [PET]
• CPC Class Codes: G06T 2207/10108 Single photon emission computed tomography [SPECT]
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 7/0014 using an image reference approach

Abstract

A framework for cross-modality image synthesis. A first and second model may be trained using respective first and second pairs of complementary images and corresponding first and second ground truth images that represent first and second views of a region of interest. The first and second pairs of complementary images may be acquired by a first modality and the first and second ground truth images may be acquired by a second modality. A combinational network may further be trained to combine features from the first and second models. At least one synthetic second modality image may then be generated by passing current images through the trained first or second model and the combinational network, wherein the current images are acquired by the first modality and represent the first or second view of the region of interest.

Claims

1. A system for image synthesis, comprising:

a non-transitory memory device for storing computer readable program code; and

a processor device in communication with the memory device, the processor being operative with the computer readable program code to perform steps including

training a first model using first pairs of complementary images and corresponding first ground truth images, wherein the first pairs of complementary images are acquired by magnetic resonance (MR) imaging and the first ground truth images are acquired by computed tomography (CT), wherein the first pairs of complementary images and the first ground truth images represent a first view of a region of interest,

training a second model using second pairs of complementary images and corresponding second ground truth images, wherein the second pairs of complementary images are acquired by MR imaging and the second ground truth images are acquired by CT, wherein the second pairs of complementary images and the second ground truth images represent a second view of the region of interest

training a combinational network to combine features from the first and second models, and

generating at least one synthetic CT image by passing current MR images through the trained first or second model and the combinational network, wherein the current MR images represent the first or second view of the region of interest.

2. The system of claim 1 wherein the first view comprises an axial view.

3. The system of claim 2 wherein the second view comprises a sagittal view.

4. The system of claim 1 wherein the first and second pairs of complementary images comprise in-phase and out-of-phase MR images.

5. A method for image synthesis, comprising:

training a first model using first pairs of complementary images and corresponding first ground truth images, wherein the first pairs of complementary images are acquired by a first modality and the first ground truth images are acquired by a second modality, wherein the first pairs of complementary images and the first ground truth images represent a first view of a region of interest;

training a second model using second pairs of complementary images and corresponding second ground truth images, wherein the second pairs of complementary images are acquired by the first modality and the second ground truth images are acquired by the second modality, wherein the second pairs of complementary images and the second ground truth images represent a second view of the region of interest;

training a combinational network to combine features from the first and second models; and

generating at least one synthetic second modality image by passing current images through the trained first or second model and the combinational network, wherein the current images are acquired by the first modality and represent the first or second view of the region of interest.

6. The method of claim 5 further comprises acquiring the first and second pairs of complementary images by magnetic resonance (MR) imaging.

7. The method of claim 6 further comprises acquiring the corresponding first and second ground truth images by computed tomography (CT).

8. The method of claim 6 further comprises acquiring the corresponding first and second ground truth images by magnetic resonance (MR) imaging.

9. The method of claim 5 further comprises acquiring the first and second pairs of complementary images by magnetic resonance (MR) in-phase and out-of-phase imaging.

10. The method of claim 5 further comprises acquiring the first and second pairs of complementary images by high-resolution computed tomography (HRCT), helical CT, X-ray, angiography, positron emission tomography (PET), fluoroscopy, ultrasound, single photon emission computed tomography (SPECT), or a combination thereof.

11. The method of claim 5 further comprises acquiring the corresponding first and second ground truth images by high-resolution computed tomography (HRCT), helical CT, X-ray, angiography, positron emission tomography (PET), fluoroscopy, ultrasound, single photon emission computed tomography (SPECT), or a combination thereof.

12. The method of claim 5 wherein the first pairs of complementary images and the first ground truth images represent an axial view of the region of interest.

13. The method of claim 5 wherein the second pairs of complementary images and the second ground truth images represent a sagittal view of the region of interest

14. The method of claim 5 wherein the first model comprises a first convolutional neural network classifier.

15. The method of claim 5 wherein the second model comprises a second convolutional neural network classifier.

16. The method of claim 5 wherein the combinational network comprises a simple one-layer network.

17. The method of claim 5 further comprises stacking multiple synthetic second modality images to generate a three-dimensional synthetic image volume.

18. One or more non-transitory computer-readable media embodying instructions executable by machine to perform operations, comprising:

training a first model using first pairs of complementary images and corresponding first ground truth images, wherein the first pairs of complementary images are acquired by a first modality and the first ground truth images are acquired by a second modality, wherein the first pairs of complementary images and the first ground truth images represent a first view of a region of interest;

training a second model using second pairs of complementary images and corresponding second ground truth images, wherein the second pairs of complementary images are acquired by the first modality and the second ground truth images are acquired by the second modality, wherein the second pairs of complementary images and the second ground truth images represent a second view of the region of interest;

training a combinational network to combine features from the first and second models; and

generating at least one synthetic second modality image by passing current images through the trained first or second model and the combinational network, wherein the current images are acquired by the first modality and represent the first or second view of the region of interest.

19. The one or more non-transitory computer-readable media of claim 18 wherein the first and second modalities are different.

20. The one or more non-transitory computer-readable media of claim 18 wherein the first and second modalities are same.