GENERATING SIMULATED BODY PARTS FOR IMAGES

• US 20200134876A1
• Filed: 10/30/2018
• Published: 04/30/2020
• Est. Priority Date: 10/30/2018
• Status: Active Grant

• Current Assignee: Guerbet
• Original Assignee: International Business Machines Corporation
• Inventors: Park, Sun Young, Sargent, Dusty
• Granted Patent: US 11,080,895 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/2431 Multiple classes
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/047 Probabilistic or stochastic networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/088 Non-supervised learning, e.g. competitive learning
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 11/008 Specific post-processing after tomographic reconstruction, e.g. voxelisation, metal artifact correction
• CPC Class Codes: G06T 2207/10081 Computed x-ray tomography [CT]
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30004 Biomedical image processing
• CPC Class Codes: G06T 2210/41 Medical
• CPC Class Codes: G06T 5/60 using machine learning, e.g. neural networks
• CPC Class Codes: G06T 5/92 based on global image properties
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 2201/03 : Recognition of patterns in medical or anatomical images
• CPC Class Codes: G06V 2201/031 : of internal organs
• CPC Class Codes: G06V 40/10 : Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands

Abstract

A system for generating simulated body parts for images may include a body part recognition convolutional neural network (CNN) to recognize a body part in an input image. The body part recognition CNN may be trained using first training data including training images including body parts contained in the input image being identified. The system may also include a body part generative adversarial network (GAN) to complete an image of the body part in the input image based on a body part identification output by the body part recognition CNN. The body part GAN may be trained using second training data including at least partial training images.

Claims

1. A system for generating simulated body parts for images, comprising:

a body part recognition convolutional neural network (CNN) to recognize a body part in an input image, the body part recognition CNN being trained using first training data comprising training images comprising body parts contained in the input image being identified; and

a body part generative adversarial network (GAN) to complete an image of the body part in the input image based on a body part identification output by the body part recognition CNN, the body part GAN being trained using second training data comprising at least partial training images.

2. The system of claim 1, comprising a contrast GAN to simulate a contrast level in the input image based on contrast images.

3. The system of claim 1, wherein the input image is taken without a radiocontrast agent present.

4. The system of claim 1, comprising a body part sequence recurrent neural network (RNN) to refine identification of the body part output by the body part recognition CNN by processing a series of images output by the body part recognition CNN in sequence.

5. The system of claim 1, comprising at least a second GAN to simulate a contrast level in an image based on image images taken.

6. The system of claim 5, wherein the at least second GAN comprises:

a contrast GAN trained to simulate a contrast level in an image based on images taken without a radiocontrast agent present; and

a non-contrast GAN trained to simulate non-contrast images based on images taken with a radiocontrast agent present.

7. The system of claim 1, comprising:

a first database comprising the first training data used by the body part recognition CNN; and

a second database comprising the second training data used by the body part GAN.

8. The system of claim 6, comprising:

a third database of the images taken without a radiocontrast agent present; and

a fourth database of the images taken with a radiocontrast agent present.

9. The system of claim 6, wherein the body part GAN, the contrast GAN, and the non-contrast GAN each comprise:

a generator to produce an artificial image that includes the input image; and

a discriminator to distinguish a real image from the artificial image.

18. The computer program product of claim 5, wherein recognizing the at least one body part in the input CT image comprises:

recognizing body parts contained in the input CT image with a body part recognition convolutional neural network (CNN) trained using training data comprising two-dimensional CT images with body parts contained in the input CT image being identified; and

refining identification of the body part output by the body part recognition CNN with a body part sequence recurrent neural network (RNN) by processing a series of images output by the body part recognition CNN in sequence.

10. A method of augmenting a computed tomography (CT) image comprising:

training a body part recognition convolutional neural network (CNN) to identify a body part in an input CT image using first training data comprising CT images comprising body parts contained in the input CT image being identified;

with a body part sequence recurrent neural network (RNN), processing a series of images output by the body part recognition CNN in sequence to refine the identification of the body part output by the body part recognition CNN;

training a first generative adversarial network (GAN) to complete an image of a body part in the input CT image based on body part identification output by the trained body part recognition CNN using training data comprising partial and complete CT images.

11. The method of claim 10, wherein:

the body part recognition CNN is trained using a first database comprising the first training data; and

the first GAN is trained using a second database comprising the second training data.

12. The method of claim 11, further comprising training at least a second GAN to simulate a contrast level in a CT image based on CT image images taken.

13. The method of claim 12, wherein the at least second GAN comprises:

a contrast GAN trained with a third database to simulate a contrast level in a CT image based on CT images taken without a radiocontrast agent present, the third database comprising the CT images taken without a radiocontrast agent present; and

a non-contrast GAN trained with a fourth database to simulate a contrast level in a CT image based on CT images taken with a radiocontrast agent present, the fourth database comprising the CT images taken with a radiocontrast agent present.

14. The method of claim 13, wherein training the body part GAN, the contrast GAN, and the non-contrast GAN comprises:

with a generator, producing an artificial CT image that includes the CT image; and

with a discriminator, distinguishing a real CT image from the artificial CT image.

15. The method of claim 12, further comprising, in a partial CT image:

recognizing body parts contained in the partial CT image with the CNN and body part sequence RNN;

completing the partial CT image by filling in missing areas using the first GAN; and

adjusting the contrast level of the completed CT image to an adjusted contrast level using the at least second GAN.

16. A computer program product for augmenting a computed tomography (CT) image, the computer program product comprising:

a computer readable storage medium comprising computer usable program code embodied therewith, the computer usable program code to, when executed by a processor;

recognize at least one body part in an input CT image, the input CT image comprising a partially captured image of a portion of the body comprising the at least one body part; and

complete the input CT image based on the recognition of body parts shown to generate a complete CT image.

17. The computer program product of claim 16, comprising computer usable program code to, when executed by the processor, simulating a desired contrast level in the completed CT image irrespective of a level of radiocontrast agent present when the partial CT image was taken.

19. The computer program product of claim 16, wherein completing the input CT image comprises executing a first generative adversarial network (GAN) trained using training data comprising partial and complete CT images

20. The computer program product of claim 16, comprising computer usable program code to, when executed by the processor, simulate a contrast level in a CT image based on previously captured CT images taken with and without a radiocontrast agent present.