Adaptive visualization for direct physician use

• US 8,938,113 B2
• Filed: 07/22/2011
• Issued: 01/20/2015
• Est. Priority Date: 07/26/2010
• Status: Active Grant

• Current Assignee: AI Visualize, Inc.
• Original Assignee: Kjaya, LLC
• Inventors: Kovalan, Kovey, Brailko, Nickolay
• Primary Examiner(s): BHATNAGAR, ANAND P
• Application Number:
  US13/811,559
• Publication Number:
  US 20130121548A1
• Time in Patent Office: 1,278 Days
• Field of Search: 382/100, 382/128, 382/129, 382/130, 382/131, 382/132, 128/922, 378 4- 27
• US Class Current: 382/131
• CPC Class Codes: A61B 5/055 involving electronic [EMR] or nuclear [NMR] magnetic resonance, e.g. magnetic resonance imaging
• CPC Class Codes: A61B 6/032 Transmission computed tomography [CT]
• CPC Class Codes: A61B 6/466 adapted to display 3D data
• CPC Class Codes: G06N 3/086 using evolutionary algorithms, e.g. genetic algorithms or genetic programming
• CPC Class Codes: G06T 15/08 Volume rendering
• CPC Class Codes: G06T 2207/10072 Tomographic images
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/20104 Interactive definition of region of interest [ROI]
• CPC Class Codes: G06T 2207/30004 Biomedical image processing
• CPC Class Codes: G06T 7/0014 using an image reference approach
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/143 involving probabilistic approaches, e.g. Markov random field [MRF] modelling
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 2201/031 of internal organs

Abstract

A method of modifying a three dimensional (3D) volume visualization image of an anatomical structure in real time to separate desired portions thereof. The method includes providing a two dimensional (2D) image slice of a 3D volume visualization image of an anatomical structure, identifying portions of the anatomical structure of interest, and providing a prototype image of desired portions of the anatomical structure. The method then includes using an evolver to evolve parameters of an algorithm that employs a transfer function to map optical properties to intensity values coinciding with the portions of the anatomical structure of interest to generate an image that sufficiently matches the prototype image. If the parameters match the prototype image, the method then includes applying the transfer function to additional 2D image slices of the 3D volume visualization image to generate a modified 3D volume visualization image of the anatomical structure. The method includes using a pattern recognizer to assist the evolver, to classify whether a view is normal or abnormal, and to extract the characteristic of an abnormality if and when detected.

Claims

1. A method of modifying a three dimensional (3D) volume visualization image of an anatomical structure in real time to separate desired portions thereof comprising:

providing a two dimensional (2D) image slice of a 3D volume visualization image of an anatomical structure;

identifying portions of the anatomical structure of interest;

providing a prototype image of desired portions of the anatomical structure;

using an evolver to evolve parameters of an algorithm that employs a transfer function to map optical properties to intensity values coinciding with the portions of the anatomical structure of interest to generate an image that sufficiently matches the prototype image;

if the parameters match the prototype image, applying the transfer function to additional 2D image slices of the 3D volume visualization image to generate a modified 3D volume visualization image of the anatomical structure.

2. The method of claim 1 wherein the optical properties comprise color and/or transparency values.

3. The method of claim 1 wherein the evolver is a genetic algorithm.

4. The method of claim 3 wherein a pattern recognizer is used to determine a fitness function of the genetic algorithm by evaluating whether the transfer function generated image sufficiently matches the prototype image, and assigning a fitness value between 0.0 and 1.0 reflecting the degree at which the images match.

5. The method of claim 4 wherein the pattern recognizer is a neural network model.

6. The method of claim 5 wherein the neural network model has been previously trained to recognize analogous images.

7. The method of claim 1 wherein the parameter values define a set of graphical elements selecting a histogram.

8. The method of claim 1 wherein the algorithm employs graphical elements selecting a histogram to generate a transfer function.

9. The method of claim 1 wherein a pattern recognizer is used to determine whether the view is flagged as normal or abnormal by the centralized server application and identifies the anomaly in the view.

10. The method of claim 1 wherein a neural network model is used to determine whether the transfer function generated image matches the prototype image, the neural network being adapted to recognize visual patterns directly from pixel images of the transfer function generated image and the prototype image.

11. The method of claim 1 wherein the neural network employs a convolution neural network model.

12. The method of claim 1 wherein a root mean squared error (Ei) fitness function is used to determine whether the transfer function generated image sufficiently matches the prototype image, and assigning a fitness value between 0.0 and 1.0 reflecting the degree at which the images match.

13. The method of claim 1 further including, if the transfer function generated image does not sufficiently match the prototype image, then continue evolving the parameters of the algorithm, and repeating the aforementioned steps.

14. The method of claim 1 further including using the matched genotype or transfer function in future modifications of a 3D volume visualization image of an anatomical structure to separate desired portions thereof, using the aforementioned steps.

15. The method of claim 1 further including using the matched genotype or transfer function to generate views in lieu of scan technologist segmentation protocols.

16. A method of displaying adaptive visualization on a remote client application and a three dimensional (3D) volume visualization image at centralized server application thereof comprising:

providing a two dimensional (2D) image slice of a 3D volume visualization image of an anatomical structure;

identifying a view of interest on the remote client application;

transmitting view of interest information to the centralized server application;

using an evolver to evolve transfer-function-parameters of an algorithm that maps optical properties to intensity values to produce a view of interest in the form of a rendered image;

using an evolver to evolve compression-parameters of an algorithm that optimizes the compression algorithm to compress the rendered image;

using an evolver to evolve transmission-parameters of an algorithm that optimizes the transmission of the compressed image from the centralized server to the remote client application.

17. The method of claim 16 wherein the optimization of compression-parameters produces the highest compression ratio while maintaining diagnostic quality.

18. The method of claim 16 wherein the optimization of transmission-parameters produces the fastest transmission to the remote client application.

19. The method of claim 16 wherein the evolver is a genetic algorithm.

20. The method of claim 16 wherein the transfer-function-parameter values define a set of graphical elements selecting a histogram.

21. The method of claim 16 wherein the algorithm employs graphical elements selecting a histogram to generate the transfer-function-parameters.

22. The method of claim 16 wherein a pattern recognizer is used to determine a fitness function of a genetic algorithm by evaluating whether the rendered image sufficiently matches the view of interest, and assigning a fitness value between 0.0 and 1.0 reflecting the degree at which the images match

23. The method of claim 16 wherein a pattern recognizer is used to determine whether the view is flagged as normal or abnormal by the centralized server application.

24. The method of claim 23 wherein the pattern recognizer is a neural network model.

25. The method of claim 24 wherein a neural network model has been previously trained to recognize analogous rendered images.

26. The method of claim 16 wherein a pattern recognizer is used to determine whether the view is flagged as normal or abnormal by the centralized server application and identifies the anomaly in the view.

27. The method of claim 16 wherein a neural network model is used to determine whether the rendered image matches the prototype image, the neural network being adapted to recognize visual patterns directly from pixel images of the rendered image and the prototype image.

28. The method of claim 27 wherein the neural network employs a convolution neural network model.

29. The method of claim 28 wherein the neural network model has been previously trained to recognize normal or abnormal views.

30. The method of claim 16 wherein a flag or warning is sent to the remote client application to alert a user as to whether the view is normal or abnormal.

31. The method of claim 30 wherein the abnormal section of the view is identified and sent to the remote client application to alert a user.

32. The method of claim 31 wherein the user is a radiologist.

33. The method of claim 31 wherein the user is a physician.

34. The method of claim 16 wherein a root mean squared error (Ei) fitness function is used to determine whether the rendered image sufficiently matches the prototype image, and assigning a fitness value between 0.0 and 1.0 reflecting the degree at which the images match.

35. The method of claim 16 further including, if the rendered image does not sufficiently match the prototype image, then continue evolving the parameters of the algorithm, and repeating the aforementioned steps.

36. The method of claim 16 further including using the transfer function parameters that produce a rendered image that matches the prototype image in future modifications of a 3D volume visualization image of an anatomical structure to separate desired portions thereof, using the aforementioned steps.

37. The method of claim 16 further including using the transfer function parameters that produce a rendered image that matches the prototype image to generate views in lieu of scan technologist segmentation protocols.

38. A method of modifying a three dimensional (3D) volume visualization image of an anatomical structure in real time to separate desired portions thereof comprising:

providing a two dimensional (2D) image slice of a 3D volume visualization image scan of an anatomical structure;

identifying a region of the 2D image slice;

providing a prototype image of desired portions of the anatomical structure;

creating a virtual histogram of the slice corresponding to the prototype image or 3D scan, the histogram including intensity values of the identified region of the image;

using a genetic algorithm, selecting regions of the virtual histogram and marking the selected regions with one or more virtual graphical elements, each of the graphical element comprising a genotype of the genetic algorithm;

using a multi dimensional transfer function (MDTF), mapping optical properties to the intensity values coinciding with the prototype slice in the marked regions of the histogram to generate an image;

comparing the MDTF-generated image of the marked region with the prototype image to determine whether the MDTF-generated image of the genotype sufficiently matches the prototype image;

if the MDTF-generated image sufficiently matches the prototype image, applying the MDTF to additional 2D image slices of the 3D volume visualization image to generate a modified 3D volume visualization image of the anatomical structure.

39. The method of claim 38 wherein the optical properties comprise color and/or transparency values.

40. The method of claim 38 wherein the virtual histogram is represented mathematically.

41. The method of claim 38 wherein the virtual graphical element is represented mathematically.

42. The method of claim 38 wherein the genotype represents the parameters of a mathematical representation that describes the virtual graphical element.

43. The method of claim 38 wherein the genetic algorithm selects regions of the virtual histogram by adapting genotypes to create the one or more graphical elements.

44. The method of claim 38 wherein the genetic algorithm employs a B-spline for the one or more graphical elements.

45. The method of claim 38 wherein the genetic algorithm employs a B-spline for the one or more graphical elements, the B-spline having a surface defined by the parametric equation:

p(u, v)=Σ

_i=0^mΣ

_j=0ⁿN_i,p(u)N_j,q(v)P_i,j;

whereby, N_i,p(u) and N_j,q(v) are the B-spline basis function of degrees p and q respectively.

46. The method of claim 38 wherein the genotype represents the parametric equation that describes the B-spline of the virtual graphical element.

47. The method of claim 38 wherein a neural network model is used to determine a fitness function of the genetic algorithm by evaluating whether the MDTF generated image of the genotype sufficiently matches the prototype image, and assigning a fitness value between 0.0 and 1.0 reflecting the degree at which the images match.

48. The method of claim 47 wherein the neural network model has been previously trained to recognize analogous images.

49. The method of claim 38 wherein a neural network model is used to determine whether the MDTF generated image of the genotype sufficiently matches the prototype image, the neural network being adapted to recognize visual patterns directly from pixel images of the MDTF generated image and the prototype image.

50. The method of claim 49 wherein the neural network employs a convolution neural network model.

51. The method of claim 38 wherein a view of interest is in the form of a MDTF generated image and a neural network model is used to determine whether the view is flagged as normal or abnormal by the centralized server application.

52. The method of claim 51 wherein the neural network model has been previously trained to recognize normal or abnormal views.

53. The method of claim 51 wherein a flag or warning is sent to the remote client application to alert a user as to whether the view is normal or abnormal.

54. The method of claim 53 wherein the abnormal section of the view is identified and sent to the remote client application to alert a user.

55. The method of claim 54 wherein the user is a radiologist.

56. The method of claim 54 wherein the user is a physician.

57. The method of claim 38 wherein a root mean squared error (Ei) fitness function is used to determine whether MDTF generated image of the genotype sufficiently matches the prototype image, and assigning a fitness value between 0.0 and 1.0 reflecting the degree at which the images match.

58. The method of claim 38 further including, if the MDTF generated image of the genotype does not sufficiently match the prototype image, then modifying the genotype to create new graphical elements, and repeating the aforementioned steps.

59. The method of claim 38 further including using the matched genotype or MDTF in future modifications of a 3D volume visualization image of an anatomical structure to separate desired portions thereof, using the aforementioned steps.

60. The method of claim 38 further including using the matched genotype or MDTF to generate views in lieu of scan technologist segmentation protocols.

61. A system for modifying a three dimensional (3D) volume visualization image of an anatomical structure in real time to separate desired portions thereof, the system adapted to receive a virtual histogram of a scan, the scan based on a two dimensional (2D) image slice of a 3D volume visualization image of an anatomical structure, the histogram including intensity values of the identified region of the image;

using a genetic algorithm, enable marking of selected regions of the virtual histogram with one or more virtual graphical elements, each of the graphical element comprising a genotype of the genetic algorithm;

using a multi dimensional transfer function (MDTF), map optical properties to the intensity values in the marked regions of the histogram;

enable comparison of the MDTF-generated optical properties of the marked region with the prototype image to determine whether the genotype sufficiently matches the prototype image; and

if the genotype sufficiently matches the prototype image, apply the genotype or MDTF to additional 2D image slices of the 3D volume visualization image to generate a modified 3D volume visualization image of the anatomical structure.

62. The system of claim 61 wherein a convolution neural network model is used to determine whether the MDTF generated image of the genotype sufficiently matches the prototype image, the neural network being adapted to recognize visual patterns directly from pixel images of the marked region and the prototype image.