COMBINING INFORMATION OF DIFFERENT LEVELS FOR CONTENT-BASED RETRIEVAL OF DIGITAL PATHOLOGY IMAGES

• US 20140270432A1
• Filed: 03/14/2014
• Published: 09/18/2014
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Xu, Xun, Dai, Shengyang, Nakamura, Akira, Wang, Su
• Granted Patent: US 9,535,928 B2
• Time in Patent Office: Days
• US Class Current: 382/128
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/56 relating to colour

Abstract

Content-based retrieval of digital pathology images (DPI) is a fundamental component in an intelligent DPI processing and management system. One key issue of content-based DPI retrieval is how to represent an image as a feature vector, capturing its key information that is most relevant to the goal of retrieval. A unified framework of extracting information of different levels for DPI, namely low level color information, middle level texture information and high level diagnostic information is described herein. Such information from all the levels are integrated to the end of content-based DPI retrieval.

Claims

1. A method of extracting a feature of an image programmed in a memory of a device comprising:

a. receiving a pathology image;

b. performing a plurality of modes of quantization on the pathology image;

c. extracting features from quantization maps of the pathology image; and

d. generating a feature vector of the pathology image.

2. The method of claim 1 wherein the pathology image is quantized into a group of quantization maps corresponding to different quantization modules.

3. The method of claim 2 wherein each element of the quantization map comprises a quantization label of a corresponding pixel in the pathology image.

4. The method of claim 1 wherein the plurality of modes of quantization include color quantization, texture quantization and diagnostic quantization.

5. The method of claim 4 wherein the color quantization is implemented by first learning a Gaussian Mixture Model (GMM) from all pixels in a set of training images, then, based on the GMM, mapping each pixel in the pathology image to a Gaussian kernel label to which a color of the pixel has maximum posterior probability.

6. The method of claim 5 wherein the texture quantization is implemented by transforming each pixel of the pathology image into a filter response vector by filtering the pathology image with a set of predefined filters and quantizing the filter response vector against the GMM, which was learned from filter response vectors computed on the set of training images.

7. The method of claim 4 wherein the diagnostic quantization is implemented by assigning a label indicating a most probable cancer stage to each pixel in the pathology image using an automatic cancer grading analysis system.

8. The method of claim 1 wherein the quantization maps are converted into feature vectors by aggregating quantization labels.

9. The method of claim 1 wherein the feature vector comprises a histogram of labels.

10. The method of claim 1 wherein the feature vector also comprises multiple co-occurrence feature vectors, deduced from co-occurrence matrices computed at multiple scales.

11. The method of claim 1 wherein a pyramid of images of different resolutions are generated from the input pathology image, and the feature extraction procedure is repeated at each resolution, and the final feature vector is the concatenation of feature vectors computed at all resolutions.

12. A system programmed in a memory of a device comprising:

a. a mode module configured for performing a plurality of modes of quantization on a pathology image;

b. an extraction module configured for extracting features from quantization maps of the pathology image; and

c. a generation module configured for generating a feature vector of the pathology image.

13. The system of claim 12 wherein the pathology image is quantized into a group of quantization maps corresponding to different quantization modules.

14. The system of claim 13 wherein each element of the quantization map comprises a quantization label of a corresponding pixel in the pathology image.

15. The system of claim 12 wherein the plurality of modes of quantization include color quantization, texture quantization and diagnostic quantization.

16. The system of claim 15 wherein the color quantization is implemented by first learning a Gaussian Mixture Model (GMM) from all pixels in a set of training images, then, based on the GMM, mapping each pixel in the pathology image to a Gaussian kernel label to which a color of the pixel has maximum posterior probability.

17. The system of claim 16 wherein the texture quantization is implemented by transforming each pixel of the pathology image into a filter response vector by filtering the pathology image with a set of predefined filters and quantizing the filter response vector against the GMM, which was learned from filter response vectors computed on the set of training images.

18. The system of claim 15 wherein the diagnostic quantization is implemented by assigning a label indicating a most probable cancer stage to each pixel in the pathology image using an automatic cancer grading analysis system.

19. The system of claim 12 wherein the quantization maps are converted into feature vectors by aggregating quantization labels.

20. The system of claim 12 wherein the feature vector comprises a histogram of labels.

21. The system of claim 12 wherein the feature vector also comprises multiple co-occurrence feature vectors, deduced from co-occurrence matrices computed at multiple scales.

22. The system of claim 12 wherein a pyramid of images of different resolutions are generated from the input pathology image, and the feature extraction procedure is repeated at each resolution, and the final feature vector is the concatenation of feature vectors computed at all resolutions.

23. An apparatus comprising:

a. a non-transitory memory for storing an application, the application for;

i. receiving a pathology image;

ii. performing a plurality of modes of quantization on the pathology image;

iii. extracting features from quantization maps of the pathology image; and

iv. generating a feature vector of the pathology image; and

b. a processing component coupled to the memory, the processing component configured for processing the application.

24. The apparatus of claim 23 wherein the pathology image is quantized into a group of quantization maps corresponding to different quantization modules.

25. The apparatus of claim 24 wherein each element of the quantization map comprises a quantization label of a corresponding pixel in the pathology image.

26. The apparatus of claim 23 wherein the plurality of modes of quantization include color quantization, texture quantization and diagnostic quantization.

27. The apparatus of claim 26 wherein the color quantization is implemented by first learning a Gaussian Mixture Model (GMM) from all pixels in a set of training images, then, based on the GMM, mapping each pixel in the pathology image to a Gaussian kernel label to which a color of the pixel has maximum posterior probability.

28. The apparatus of claim 27 wherein the texture quantization is implemented by transforming each pixel of the pathology image into a filter response vector by filtering the pathology image with a set of predefined filters and quantizing the filter response vector against the GMM, which was learned from filter response vectors computed on the set of training images.

29. The apparatus of claim 26 wherein the diagnostic quantization is implemented by assigning a label indicating a most probable cancer stage to each pixel in the pathology image using an automatic cancer grading analysis system.

30. The apparatus of claim 23 wherein the quantization maps are converted into feature vectors by aggregating quantization labels.

31. The apparatus of claim 23 wherein the feature vector comprises a histogram of labels.

32. The apparatus of claim 23 wherein the feature vector also comprises multiple co-occurrence feature vectors, deduced from co-occurrence matrices computed at multiple scales.

33. The apparatus of claim 23 wherein a pyramid of images of different resolutions are generated from the input pathology image, and the feature extraction procedure is repeated at each resolution, and the final feature vector is the concatenation of feature vectors computed at all resolutions.