TWO-DIMENSIONAL CINEMATIC MEDICAL IMAGING IN COLOR BASED ON DEEP LEARNING

• US 20180082487A1
• Filed: 08/15/2017
• Published: 03/22/2018
• Est. Priority Date: 09/20/2016
• Status: Active Grant

• Current Assignee: Siemens Healthineers AG (Siemens AG)
• Original Assignee: Siemens Healthcare GMBH (Siemens AG)
• Inventors: Kiraly, Atilla Peter, Petkov, Kaloian, Park, Jin-hyeong
• Granted Patent: US 10,282,918 B2
• Time in Patent Office: Days
• CPC Class Codes: A61B 5/7425 Displaying combinations of multiple images regardless of image source, e.g. displaying a reference anatomical image with a live image geometric transforms for image registration G06T3/14; enhancement using image fusion G06T5/50; algorithms for registration of images G06T7/30
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10072 Tomographic images
• CPC Class Codes: G06T 2207/10081 Computed x-ray tomography [CT]
• CPC Class Codes: G06T 2207/10088 Magnetic resonance imaging [MRI]
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2210/41 Medical
• CPC Class Codes: G06T 2219/2012 Colour editing, changing, or manipulating; Use of colour codes
• CPC Class Codes: H04N 1/6027 Correction or control of colour gradation or colour contrast H04N1/6058 takes precedence
• CPC Class Codes: Y04S 10/50 Systems or methods supporting the power network operation or management, involving a certain degree of interaction with the load-side end user applications

Abstract

A 2D medical image is colorized. In one approach, a deep-learnt classifier is trained to colorize from color 2D medical images. The color 2D medical images for training are cinematically rendered from slabs to add color. In another approach, a deep machine-learnt generator creates slices as if adjacent to the 2D medical image. The slices and 2D medical image form a slab, which is cinematically rendered to add color. The result is a colorized 2D medical image.

Claims

1. A method for training a deep machine-learnt classifier for colorizing a two-dimensional medical image, the method comprising:

loading to a memory training data representing volumes of multiple patients, the volumes comprising scalar values distributed in three dimensions from medical scanning of the patients;

isolating slabs each representing multiple slices in each of the volumes;

three-dimensional color rendering the scalar values of the slabs to two-dimensional color images with cinematic rendering;

training, from the two-dimensional color images, a deep machine-learnt classifier with deep machine learning to colorize the two-dimensional medical image.

2. The method of claim 1 wherein isolating comprises selecting first slices representing only first planes in the volumes and adding two to four adjacent slices representing only adjacent planes, the first planes and respective adjacent planes comprising the slabs.

3. The method of claim 1 wherein three-dimensional color rendering comprises rendering with unbiased path tracing.

4. The method of claim 1 wherein three-dimensional color rendering comprises rendering with a look-up table relating the scalar values to colors.

5. The method of claim 1 wherein three-dimensional color rendering comprises rendering with a camera oriented substantially orthogonally to the slabs.

6. The method of claim 1 wherein three-dimensional color rendering comprises collapsing from the slabs, which represent multiple planes, to single planes for the two-dimensional color images.

7. The method of claim 1 wherein training comprises training a convolutional neural network.

8. The method of claim 1 wherein training comprises training with the scalar values of one of the slices of each slab as input images and the two-dimensional color images as ground truth.

9. The method of claim 1 further comprising applying the deep machine-learnt classifier to individual slices representing only planes in a new volume, assembling the individual colorized slices into a color volume, and rendering from the color volume.

10. The method of claim 1 further comprising generating multi-label masks, and wherein three-dimensional color rendering comprises rendering based on the multi-label masks.

11. The method of claim 1 further comprising training a second machine-learnt classifier to classify a diagnostic or therapeutic characteristic based on an input of the colorized version of the two-dimensional medical image.

12. A method for colorizing a two-dimensional medical image of a patient, the method comprising:

generating a first two-dimensional slice representing a first plane, the generating being by a deep machine-learnt generator based on input of the two-dimensional medical image;

forming a slab of the first two-dimensional slice and the two-dimensional medical image;

color rendering, by a renderer, the slab with a cinematic renderer; and

displaying, on a display, the slab as rendered in color as a colorized version of the two-dimensional medical image.

13. The method of claim 12 wherein generating further comprises generating a second two-dimensional slice representing a second plane different than the first plane, the first and second planes different than a third plane represented by the two-dimensional medical image, and wherein forming the slab comprises forming the slab of the first and second two-dimensional slices and the two-dimensional medical image in a stack.

14. The method of claim 12 wherein generating comprises generating with the deep machine-learnt generator comprising a convolutional deep learnt neural network.

15. The method of claim 12 wherein forming the slab comprises forming a volume and wherein color rendering comprises three-dimensional rendering the volume.

16. The method of claim 12 wherein color rendering comprises three-dimensional rendering with path tracing.

17. The method of claim 12 wherein the two-dimensional medical image and the first two-dimensional slice comprise scalar values without color information, and wherein color rendering comprises three-dimensional rendering with a look-up table linking scalar values with colors.

18. The method of claim 12 wherein the two-dimensional medical image includes a first multi-label mask, wherein generating comprises generating the first two-dimensional slice with a second multi-label mask, and wherein color rendering comprises color rendering as a function of the first and second multi-label masks.

19. The method of claim 12 further comprising:

inputting the colorized version of the two-dimensional medical image to a machine-learnt classifier; and

outputting, by the machine-learnt classifier based on the inputting, a diagnostic or therapeutic classification.

20. A system for colorizing two-dimensional medical imaging, the system comprising:

a medical scanner configured to scan a first plane of a patient and generate scan data for the first plane;

a machine configured to output scan data for a second plane by application of the scan data for the first plane to a deep-machine-learnt extrapolator;

a graphics processing unit configured to cinematically render a color image of the first plane from a slab formed from the scan data of the first plane and the scan data of the second plane; and

a display configured to display the color image.

21. A system for colorizing two-dimensional medical imaging, the system comprising:

a medical scanner configured to scan a first plane of a patient and generate scan data for the first plane;

a machine configured to output a color image from the scan data for the first plane by application of the scan data for the first plane to a deep-machine-learnt colorizer trained from cinematic renderings of slabs; and

a display configured to display the color image.