Combining information of different levels for content-based retrieval of digital pathology images

• US 9,535,928 B2
• Filed: 03/14/2014
• Issued: 01/03/2017
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Xu, Xun, Dai, Shengyang, Nakamura, Akira, Wang, Su
• Primary Examiner(s): Lefkowitz, Sumati
• Assistant Examiner(s): Wang, Carol
• Application Number:
  US14/212,827
• Publication Number:
  US 20140270432A1
• Time in Patent Office: 1,026 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/56 relating to colour

Abstract

Content-based retrieval of digital pathology images (DPI) is a fundamental component in an intelligent DPI processing and management system. One key issue of content-based DPI retrieval is how to represent an image as a feature vector, capturing its key information that is most relevant to the goal of retrieval. A unified framework of extracting information of different levels for DPI, namely low level color information, middle level texture information and high level diagnostic information is described herein. Such information from all the levels are integrated to the end of content-based DPI retrieval.

Claims

1. A method of extracting a feature of an image programmed in a memory of a device comprising:

a. receiving a pathology image;

b. performing a plurality of modes of quantization on the pathology image, wherein the plurality of modes of quantization include color quantization, texture quantization and diagnostic quantization, wherein the diagnostic quantization is implemented by assigning a label indicating a most probable cancer stage to each pixel in the pathology image using an automatic cancer grading analysis system;

c. extracting features from quantization maps of the pathology image, wherein each quantization map of the quantization maps is an integer array of a size equal to the pathology image; and

d. generating a feature vector of the pathology image, wherein the feature vector comprises multiple co-occurrence feature vectors, deduced from co-occurrence matrices computed at multiple scales, wherein an offset distance parameter is used to determine a scale at which pixel correlation is analyzed.

2. The method of claim 1 wherein the pathology image is quantized into a group of quantization maps corresponding to different quantization modules.

3. The method of claim 2 wherein each element of the quantization maps comprises a quantization label of a corresponding pixel in the pathology image.

4. The method of claim 1 wherein the color quantization is implemented by first learning a Gaussian Mixture Model (GMM) from all pixels in a set of training images, then, based on the GMM, mapping each pixel in the pathology image to a Gaussian kernel label to which a color of the pixel has maximum posterior probability.

5. The method of claim 4 wherein the texture quantization is implemented by transforming each pixel of the pathology image into a filter response vector by filtering the pathology image with a set of predefined filters and quantizing the filter response vector against the GMM, which was learned from filter response vectors computed on the set of training images.

6. The method of claim 1 wherein the quantization maps are converted into feature vectors by aggregating quantization labels.

7. The method of claim 1 wherein the feature vector comprises a histogram of labels.

8. The method of claim 1 wherein a pyramid of images of different resolutions are generated from the input pathology image, and the feature extraction procedure is repeated at each resolution, and the final feature vector is the concatenation of feature vectors computed at all resolutions.

9. A system comprising:

a. a non-transitory memory for storing an application, the application comprising;

i. a mode module configured for performing a plurality of modes of quantization on a pathology image, wherein the plurality of modes of quantization include color quantization, texture quantization and diagnostic quantization, wherein the diagnostic quantization is implemented by assigning a label indicating a most probable cancer stage to each pixel in the pathology image using an automatic cancer grading analysis system;

ii. an extraction module configured for extracting features from quantization maps of the pathology image, wherein each quantization map of the quantization maps is an integer array of a size equal to the pathology image; and

iii. a generation module configured for generating a feature vector of the pathology image, wherein the feature vector comprises multiple co-occurrence feature vectors, deduced from co-occurrence matrices computed at multiple scales, wherein an offset distance parameter is used to determine a scale at which pixel correlation is analyzed; and

b. a processor coupled to the memory, the processor configured for processing the application.

10. The system of claim 9 wherein the pathology image is quantized into a group of quantization maps corresponding to different quantization modules.

11. The system of claim 10 wherein each element of the quantization maps comprises a quantization label of a corresponding pixel in the pathology image.

12. The system of claim 9 wherein the color quantization is implemented by first learning a Gaussian Mixture Model (GMM) from all pixels in a set of training images, then, based on the GMM, mapping each pixel in the pathology image to a Gaussian kernel label to which a color of the pixel has maximum posterior probability.

13. The system of claim 12 wherein the texture quantization is implemented by transforming each pixel of the pathology image into a filter response vector by filtering the pathology image with a set of predefined filters and quantizing the filter response vector against the GMM, which was learned from filter response vectors computed on the set of training images.

14. The system of claim 9 wherein the quantization maps are converted into feature vectors by aggregating quantization labels.

15. The system of claim 9 wherein the feature vector comprises a histogram of labels.

16. The system of claim 9 wherein a pyramid of images of different resolutions are generated from the input pathology image, and the feature extraction procedure is repeated at each resolution, and the final feature vector is the concatenation of feature vectors computed at all resolutions.

17. An apparatus comprising:

b. a non-transitory memory for storing an application, the application for;

i. receiving a pathology image;

ii. performing a plurality of modes of quantization on the pathology image, wherein the plurality of modes of quantization include color quantization, texture quantization and diagnostic quantization, wherein the diagnostic quantization is implemented by assigning a label indicating a most probable cancer stage to each pixel in the pathology image using an automatic cancer grading analysis system;

iii. extracting features from quantization maps of the pathology image, wherein each quantization map of the quantization maps is an integer array of a size equal to the pathology image; and

iv. generating a feature vector of the pathology image, wherein the feature vector comprises multiple co-occurrence feature vectors, deduced from co-occurrence matrices computed at multiple scales, wherein an offset distance parameter is used to determine a scale at which pixel correlation is analyzed; and

c. a processor coupled to the memory, the processor configured for processing the application.

18. The apparatus of claim 17 wherein the pathology image is quantized into a group of quantization maps corresponding to different quantization modules.

19. The apparatus of claim 18 wherein each element of the quantization maps comprises a quantization label of a corresponding pixel in the pathology image.

20. The apparatus of claim 17 wherein the color quantization is implemented by first learning a Gaussian Mixture Model (GMM) from all pixels in a set of training images, then, based on the GMM, mapping each pixel in the pathology image to a Gaussian kernel label to which a color of the pixel has maximum posterior probability.

21. The apparatus of claim 20 wherein the texture quantization is implemented by transforming each pixel of the pathology image into a filter response vector by filtering the pathology image with a set of predefined filters and quantizing the filter response vector against the GMM, which was learned from filter response vectors computed on the set of training images.

22. The apparatus of claim 17 wherein the quantization maps are converted into feature vectors by aggregating quantization labels.

23. The apparatus of claim 17 wherein the feature vector comprises a histogram of labels.

24. The apparatus of claim 17 wherein a pyramid of images of different resolutions are generated from the input pathology image, and the feature extraction procedure is repeated at each resolution, and the final feature vector is the concatenation of feature vectors computed at all resolutions.