IMAGE SYNTHESIS USING ADVERSARIAL NETWORKS SUCH AS FOR RADIATION THERAPY

• US 20190318474A1
• Filed: 04/13/2018
• Published: 10/17/2019
• Est. Priority Date: 04/13/2018
• Status: Active Grant

• Current Assignee: Elekta Incorporated (Elekta AB)
• Original Assignee: Elekta Incorporated (Elekta AB)
• Inventors: Han, Xiao
• Granted Patent: US 11,100,632 B2
• Time in Patent Office: Days
• CPC Class Codes: A61N 5/103 Treatment planning systems
• CPC Class Codes: G06T 2207/10081 Computed x-ray tomography [CT]
• CPC Class Codes: G06T 2207/10088 Magnetic resonance imaging [MRI]
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks

Abstract

A statistical learning technique that does not rely upon paired imaging information is described herein. The technique may be computer-implemented and may be used in order to train a statistical learning model to perform image synthesis, such as in support of radiation therapy treatment planning. In an example, a trained statistical learning model may include a convolutional neural network established as a generator convolutional network, and the generator may be trained at least in part using a separate convolutional neural network established as a discriminator convolutional network. The generator convolutional network and the discriminator convolutional network may form an adversarial network architecture for use during training. After training, the generator convolutional network may be provided for use in synthesis of images, such as to receive imaging data corresponding to a first imaging modality type, and to synthesize imaging data corresponding to a different, second imaging modality type.

Claims

1. A computer-implemented method for synthesizing a medical image using a trained statistical learning model, the method comprising:

receiving medical imaging data obtained using a first imaging modality type;

applying the trained statistical learning model to the received medical imaging data to synthesize a medical image corresponding to a different second imaging modality type; and

providing the synthesized medical image for presentation or for use in further processing;

wherein the trained statistical learning model is established at least in part using a similarity determination between training imaging data provided at the model input, the training imaging data corresponding to the first imaging modality type and synthesized imaging data at the model output corresponding to the second imaging modality type; and

wherein the trained statistical learning model is established at least in part using a separate statistical learning model, the separate statistical learning model established to discriminate between actual imaging data corresponding to the second imaging modality and the synthesized imaging data.

2. The computer-implemented method of claim 1, wherein the trained statistical learning model comprises a first generator convolutional network and the separate statistical learning model comprises a first discriminator convolutional network.

3. The computer-implemented method of claim 2, wherein the trained statistical learning model is established without requiring use of a second generator convolutional network and a second discriminator convolutional network, the second generator convolutional network trained to synthesize a medical image corresponding to the first imaging modality type from an input image corresponding to the second imaging modality type, and the second discriminator network established to discriminate between synthesized imaging data corresponding to the first imaging modality type and actual imaging data corresponding to the first imaging modality type.

4. The computer-implemented method of claim 2, wherein the first generator convolutional network and the first discriminator convolutional network form an adversarial network architecture, the adversarial network architecture structured to:

adjust the first discriminator convolutional network to enhance or maximize a value of an objective over a distribution of inputs to the first discriminator convolutional network; and

adjust the first generator convolutional network to reduce or minimize the value of the objective over a distribution of inputs to the first generator convolutional network.

5. The computer-implemented method of claim 4, wherein the objective is represented as a sum of contributions from an adversarial objective and the similarity determination.

6. The computer-implemented method of claim 4, wherein the objective comprises terms representing (1) an expectation value that an input to the first discriminator including actual imaging data is identified as such by the first discriminator and (2) an expectation value that an input to the discriminator including a synthesized image from the first generator is not identified as actual imaging data by the first discriminator.

7. The computer-implemented method of claim 1, wherein the similarity determination comprises at least one of:

a mutual information determination;

or a cross-correlation determination.

8. The computer-implemented method of claim 1, wherein the first imaging modality type comprises magnetic resonance (MR) imaging;

and

wherein the second imaging modality type comprises computed tomography (CT) imaging.

9. The computer-implemented method of claim 1, wherein presenting the synthesized image includes displaying the image on an MR-LINAC workstation for use in radiation therapy treatment planning or administration.

10. The computer-implemented method of claim 1, comprising updating, or receiving an update, of a radiation therapy treatment protocol based at least in part upon the synthesized medical image.

11. A computer-implemented method for establishing a trained statistical learning model for synthesizing a medical image without requiring paired imaging for training, the method comprising:

receiving training medical imaging data corresponding to a first imaging modality type;

applying a statistical learning model to the received medical imaging data to synthesize imaging data corresponding to a different second imaging modality type;

adjusting the statistical learning model at least in part using a similarity determination between the training imaging data and synthesized imaging data at the model output; and

adjusting the statistical learning model at least in part using a separate statistical learning model, the separate statistical learning model established to discriminate between the synthesized imaging data and actual imaging data corresponding to the second imaging modality type.

12. The computer-implemented method of claim 11, wherein the statistical learning model comprises a first generator convolutional network and the separate statistical learning model comprises a first discriminator convolutional network.

13. The computer-implemented method of claim 12, wherein adjusting the statistical learning model includes using a stochastic gradient descent technique.

14. The computer-implemented method of claim 12, wherein the first generator convolutional network and the first discriminator convolutional network form an adversarial network architecture, the adversarial network architecture structured to:

adjust the first discriminator convolutional network to enhance or maximize a value of an objective over a distribution of inputs to the first discriminator convolutional network; and

adjust the first generator convolutional network to reduce or minimize the value of the objective over a distribution of inputs to the first generator convolutional network.

15. The computer-implemented method of claim 14, wherein the objective is represented as a sum of contributions from an adversarial objective and the similarity determination.

16. The computer-implemented method of claim 14, wherein the objective comprises terms representing (1) an expectation value that an input to the first discriminator including actual imaging data is identified as such by the first discriminator and (2) an expectation value that an input to the discriminator including a synthesized image from the first generator is not identified as actual imaging data by the first discriminator.

17. The computer-implemented method of claim 11, wherein the similarity determination comprises at least one of:

a mutual information determination;

or a cross-correlation determination.

18. The computer-implemented method of claim 11, wherein the first imaging modality type comprises magnetic resonance (MR) imaging;

and

wherein the second imaging modality type comprises computed tomography (CT) imaging.

19. A system, comprising:

processing circuitry comprising at least one processor; and

a storage medium comprising instructions, which when executed by the at least one processor, cause the processor to;

receive medical imaging data obtained using a first imaging modality type;

apply the trained statistical learning model to the received medical imaging data to synthesize a medical image corresponding to a different second imaging modality type; and

provide the synthesized medical image for presentation or for use in further processing;

wherein the trained statistical learning model is established at least in part using a similarity determination between training imaging data provided at the model input, the training imaging data corresponding to the first imaging modality type and synthesized imaging data at the model output corresponding to the second imaging modality type; and

wherein the trained statistical learning model is established at least in part using a separate statistical learning model, the separate statistical learning model established to discriminate between the synthesized imaging data and actual imaging data corresponding to the second imaging modality type.

20. The system of claim 19, wherein the instructions cause the processor to update, or receive an update, of a radiation therapy treatment protocol based at least in part upon the synthesized medical image.