Global signatures for large-scale image recognition

• US 10,565,759 B2
• Filed: 06/26/2017
• Issued: 02/18/2020
• Est. Priority Date: 03/05/2015
• Status: Active Grant

• Current Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Original Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Inventors: Song, Bing
• Primary Examiner(s): Niu, Feng
• Application Number:
  US15/633,679
• Publication Number:
  US 20170294040A1
• Time in Patent Office: 967 Days
• Field of Search: None
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/28 Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2210/22 Cropping
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G06V 10/464 using a plurality of salient features, e.g. bag-of-words [BoW] representations
• CPC Class Codes: G06V 10/772 Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries
• CPC Class Codes: G16H 10/60 for patient-specific data, e.g. for electronic patient records
• CPC Class Codes: G16H 30/20 for handling medical images, e.g. DICOM, HL7 or PACS
• CPC Class Codes: G16H 30/40 for processing medical images, e.g. editing

Abstract

Techniques are provided that include obtaining a vocabulary including a set of content indices that reference corresponding cells in a descriptor space based on an input set of descriptors. A plurality of local features of an image are identified based on the vocabulary, the local features being represented by a plurality of local descriptors. An associated visual word in the vocabulary is determined for each of the plurality of local descriptors. A plurality of global signatures for the image are generated based on the associated visual words, wherein some of the plurality of global signatures are generated using local descriptors corresponding to different cropped versions of the image, two or more of the different cropped versions of the image being centered at a same pixel location of the image, and an image recognition search is facilitated using the plurality of global signatures to search a document image dataset.

Claims

1. An image recognition device comprising:

at least one tangible, non-transitory, computer-readable memory configured to store a vocabulary including a set of content indices that reference corresponding cells in a descriptor space based on an input set of descriptors identifying medical information; and

at least one processor coupled with the at least one tangible, non-transitory computer-readable memory and, upon execution of image recognition software instructions, is configured to;

identify a plurality of local features of an image based on the vocabulary, the local features being represented by a plurality of local descriptors;

determine an associated visual word in the vocabulary for each one of the plurality of local descriptors;

generate, prior to facilitating an image recognition search, a plurality of global signatures for the image based on the associated visual words, wherein some of the plurality of global signatures are generated using local descriptors corresponding to different cropped versions of the image;

store the plurality of global signatures; and

facilitate the image recognition search to search a medical record dataset, wherein the image recognition search compares each of the stored plurality of global signatures with a plurality of document image signatures.

2. The device of claim 1, wherein facilitating the image recognition search includes displaying an image recognition search result to a user.

3. The device of claim 1, wherein a first global signature of the plurality of global signatures corresponds to a first cropped version of the image, and wherein at least one of the different cropped versions of the image is centered at a same pixel location as the first cropped version of the image.

4. The device of claim 1, wherein a first global signature of the plurality of global signatures corresponds to a first cropped version of the image, and wherein at least one of the different cropped versions of the image is centered at a different pixel location from the first cropped version of the image.

5. The device of claim 1, wherein the at least one processor is further configured to:

select at least one additional pixel location of the image; and

generate one or more of the plurality of global signatures for the image using local descriptors corresponding to cropped versions of the image centered at the at least one additional pixel location.

6. The device of claim 1, wherein the at least one processor is further configured to determine at least one of the different cropped versions of the image in order to focus a global signature around one or more features or objects in the image.

7. The device of claim 1, wherein the at least one processor is further configured to use the different cropped versions of the image to identify different regions of the image.

8. The device of claim 1, wherein at least one aspect of the different cropped versions of the image is user-selectable.

9. The device of claim 1, wherein one of the plurality of global signatures is generated using local descriptors corresponding to an uncropped version of the image.

10. The device of claim 1, wherein the image is a digital representation of at least one of a medical patient, a face, and biological material.

11. The device of claim 1, wherein the image is one of a query image or a document image.

12. The device of claim 1, wherein the medical record dataset comprises global signatures corresponding to each of a plurality of images.

13. The device of claim 1, wherein the medical record dataset includes digital representations of an object at different scales.

14. The device of claim 1, wherein the medical record dataset comprises at least one of a medical training image, a video frame, and a test library of medical record images.

15. The device of claim 1, wherein the at least one processor is further configured to:

divide the image into k cluster centers; and

assign each one of the plurality of local descriptors to a closest one of the k cluster centers.

16. The device of claim 15, wherein a global signature is a k*128 dimensional vector of locally aggregated descriptors (VLAD) global signature.

17. The device of claim 16, wherein the global signature is a 2048 dimensional VLAD global signature.

18. The device of claim 1, wherein the plurality of global signatures includes at least one of a VLAD signature, a GIST signature or a Deep Learning signature.

19. The device of claim 1, wherein the at least one processor is further configured to generate the plurality of global signatures based on one or more features or objects located at an off-center location in the image.

20. The device of claim 1, wherein the at least one processor is further configured to generate the plurality of global signatures based on one or more different scaled features or objects in the image.

21. The device of claim 1, wherein a distance measurement between global signatures representing images containing an object in common is less than a distance measurement between global signatures representing images that do not contain an object in common.

22. The device of claim 1, wherein the plurality of local descriptors are one of scale-invariant feature transform (SIFT) descriptors, Fast Retina Keypoint (FREAK) descriptors, Histograms of Oriented Gradient (HOG) descriptors, Speeded Up Robust Features (SURF) descriptors, DAISY descriptors, Binary Robust Invariant Scalable Keypoints (BRISK) descriptors, FAST descriptors, Binary Robust Independent Elementary Features (BRIEF) descriptors, Harris Corners descriptors, Edges descriptors, Gradient Location and Orientation Histogram (GLOH) descriptors, Electrooculography (EOG) descriptors or Transform Invariant Low-rank Textures (TILT) descriptors.

23. The device of claim 1, wherein a dictionary is used to determine the associated visual word in the vocabulary with each of the plurality of local descriptors.

24. The device of claim 23, wherein the dictionary comprises a VLAD dictionary in which a plurality of visual words is based on descriptors determined from a training sample of model images.

25. A method for large-scale image recognition, comprising:

storing, by at least one tangible, non-transitory, computer-readable memory, a vocabulary including a set of content indices that reference corresponding cells in a descriptor space based on an input set of descriptors identifying medical information;

identifying, by at least one processor coupled with the at least one tangible, non-transitory computer-readable memory, a plurality of local features of an image based on the vocabulary, the local features being represented by a plurality of local descriptors;

determining, by the at least one processor, an associated visual word in the vocabulary for each one of the plurality of local descriptors;

generating, prior to facilitating an image recognition search, a plurality of global signatures for the image based on the associated visual words, wherein some of the plurality of global signatures are generated using local descriptors corresponding to different cropped versions of the image;

storing the plurality of global signatures; and

facilitating the image recognition search to search a medical record dataset, wherein the image recognition search compares each of the stored plurality of global signatures with a plurality of document image signatures.

26. A computer program product embedded in a non-transitory computer readable medium comprising instructions executable by a computer processor for large-scale image recognition, which, when executed by a processor, cause the processor to perform one or more steps comprising:

storing a vocabulary including a set of content indices that reference corresponding cells in a descriptor space based on an input set of descriptors identifying medical information;

identifying a plurality of local features of an image based on the vocabulary, the local features being represented by a plurality of local descriptors;

determining, by the at least one processor, an associated visual word in the vocabulary for each one of the plurality of local descriptors;

generating, prior to facilitating an image recognition search, a plurality of global signatures for the image based on the associated visual words, wherein some of the plurality of global signatures are generated using local descriptors corresponding to different cropped versions of the image;

storing the plurality of global signatures; and

facilitating the image recognition search to search a medical record dataset, wherein the image recognition search compares each of the stored plurality of global signatures with a plurality of document image signatures.