CONTENT-BASED MEDICAL IMAGE RENDERING BASED ON MACHINE LEARNING

• US 20170262598A1
• Filed: 06/23/2016
• Published: 09/14/2017
• Est. Priority Date: 03/10/2016
• Status: Active Grant

• Current Assignee: Siemens Healthineers AG (Siemens AG)
• Original Assignee: Siemens Healthcare GMBH (Siemens AG)
• Inventors: Petkov, Kaloian, Miao, Shun, Yu, Daphne, Georgescu, Bogdan, Engel, Klaus, Mansi, Tommaso, Comaniciu, Dorin
• Granted Patent: US 9,760,690 B1
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/2414 Smoothing the distance, e.g. radial basis function networks [RBFN]
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/006 based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
• CPC Class Codes: G06N 3/02 Neural networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06T 15/005 General purpose rendering architectures
• CPC Class Codes: G06T 15/06 Ray-tracing
• CPC Class Codes: G06T 15/08 Volume rendering
• CPC Class Codes: G06T 15/506 Illumination models
• CPC Class Codes: G06T 2210/41 Medical
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 2201/03 Recognition of patterns in medical or anatomical images
• CPC Class Codes: G16H 30/20 for handling medical images, e.g. DICOM, HL7 or PACS
• CPC Class Codes: G16H 30/40 for processing medical images, e.g. editing
• CPC Class Codes: G16H 50/20 for computer-aided diagnosis, e.g. based on medical expert systems

Abstract

An artificial intelligence agent is machine trained and used to provide physically-based rendering settings. By using deep learning and/or other machine training, settings of multiple rendering parameters may be provided for consistent imaging even in physically-based rendering.

Claims

1. A method for content-based rendering based on machine learning in a rendering system, the method comprising:

loading, from memory, a medical dataset representing a three-dimensional region of a patient;

applying, by a machine, the medical dataset to a machine-learnt model, the machine-learned model trained with deep learning to extract features from the medical dataset and trained to output values for two or more physically-based rendering parameters based on input of the medical dataset, the two or more physically-based rendering parameters being settings of a physically-based renderer, the settings used by the physically-based renderer to control rendering from three dimensions to two-dimensions, at least one of the two or more rendering parameters being controls for data consistency handling, lighting design, viewing design, material propriety, or internal renderer property;

rendering, by the physically-based renderer, a photorealistic image of the three-dimensional region of the patient using the output values resulting from the applying as the settings to control the rendering from the medical dataset, the rendering of the medical dataset of the three-dimensional region being to the photorealistic image in the two-dimensions; and

transmitting the photorealistic image.

2. The method of claim 1 further comprising loading patient information other than the medical dataset representing the three-dimensional region of the patient.

3. The method of claim 1 wherein applying comprises applying to output the values for the two or more physically-based rendering parameters as all of the physically-based rendering parameters other than viewing design based on applying user input of the viewing design and the medical dataset.

4. The method of claim 1 wherein applying comprises applying with the machine-learnt model trained to output the values resulting in the photorealistic image corresponding to a standard image despite differences in the medical dataset.

5. The method of claim 1 wherein applying comprises applying with the machine-learnt model comprising a regression, classification, or reinforcement learnt model.

6. The method of claim 1 wherein applying comprises applying with the deep learning as a deep neural network.

7. The method of claim 1 wherein applying comprises applying with the machine-learnt model as a deep reinforcement learnt model.

8. The method of claim 1 wherein rendering comprises rendering with unbiased path tracing.

9. The method of claim 1 wherein transmitting comprises transmitting as part of a diagnostic report, as an initial image of an interactive viewing, or as an overlay in augmented reality.

10. The method of claim 1 further comprising:

measuring ambient light with a light sensor;

wherein applying comprises applying the ambient light and the medical dataset, the machine-learnt model trained to output the values based in part on the ambient light.

11. A method for machine training for content-based rendering in a machine training system, the method comprising:

inputting first volume data of a volume of a patient, a first image of the volume, and first values of path tracing rendering parameters to training of an artificial intelligence, the path tracing rendering parameters being settings to control rendering from the volume to a two-dimensional image;

machine training, with a machine, the artificial intelligence to output second values of the path tracing rendering parameters for a second volume data where the second values control the rendering from the volume to provide a second rendered image of the second volume modeled on the first image; and

storing the trained artificial intelligence.

12. The method of claim 11 wherein inputting comprises inputting the patient non-image information to the training.

13. The method of claim 11 wherein training comprises training the artificial intelligence to output the second values as two or more of data consistency, transfer function, lighting, and viewing parameters.

14. The method of claim 11 wherein training comprises training the artificial intelligence to output the second values based on user selected viewing camera parameters, the path tracing rendering parameters for which second values are to be output being other than the viewing camera parameters.

15. The method of claim 11 wherein training so the second rendered image is modeled after the first rendered image comprises training with a metric of similarity.

16. The method of claim 11 wherein the first image comprises a reference photograph or video of the patient.

17. The method of claim 11 wherein inputting comprises perturbing the path tracing rendering parameters, creating a collection of sets of the path tracing rendering parameters, and wherein training comprises training based on selection of a sub-set of the sets.

18. The method of claim 17 wherein selection of the sub-set comprises selection by a user based on images rendered using the sets.

19. The method of claim 11 wherein machine training comprises deep learning with regression, classification, or reinforcement learning.

20. The method of claim 19 wherein machine training comprises deep reinforcement learning with a similarity of the second rendered image to the first image as a reinforcement.

21. The method of claim 20 wherein deep reinforcement learning comprises selecting with a probability distribution of different similarities including the similarity.

22. A system for content-based rendering based on machine learning, the system comprising:

a medical scanner configured to scan a patient;

a machine configured to output settings for rendering parameters by application of data from the scan to a machine-learnt model, the rendering parameters being controls for performing rendering from a volume to a two-dimensional image, the rendering parameters including windowing, transfer function, and lighting, and the settings learned to provide a first image from the data similar to one or more second images for a same diagnostic context; and

a graphics processing unit configured to render the first image from the data using the settings output by the application of the data to the machine-learnt model, the first image being a two-dimensional representation.

23. The system of claim 22 wherein the machine-learnt model is machine learnt with deep learning.

24. The system of claim 22 wherein the rendering parameters further comprise material properties, viewing properties, lighting properties and internal renderer properties, and wherein the graphics processing unit is configured to render with path tracing using the settings.