GLOBAL SIGNATURES FOR LARGE-SCALE IMAGE RECOGNITION

• US 20160259816A1
• Filed: 03/07/2016
• Published: 09/08/2016
• Est. Priority Date: 03/05/2015
• Status: Active Grant

• Current Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Original Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Inventors: Song, Bing
• Granted Patent: US 9,721,186 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/28 Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2210/22 Cropping
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G06V 10/464 using a plurality of salient features, e.g. bag-of-words [BoW] representations
• CPC Class Codes: G06V 10/772 Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries
• CPC Class Codes: G16H 10/60 for patient-specific data, e.g. for electronic patient records
• CPC Class Codes: G16H 30/20 for handling medical images, e.g. DICOM, HL7 or PACS
• CPC Class Codes: G16H 30/40 for processing medical images, e.g. editing

Abstract

Techniques are provided that include obtaining a vocabulary including a set of content indices that reference corresponding cells in a descriptor space based on an input set of descriptors. A plurality of local features of an image are identified based on the vocabulary, the local features being represented by a plurality of local descriptors. An associated visual word in the vocabulary is determined for each of the plurality of local descriptors. A plurality of global signatures for the image are generated based on the associated visual words, wherein some of the plurality of global signatures are generated using local descriptors corresponding to different cropped versions of the image, two or more of the different cropped versions of the image being centered at a same pixel location of the image, and an image recognition search is facilitated using the plurality of global signatures to search a document image dataset.

Claims

1. A method for large-scale image recognition, comprising:

obtaining, by a descriptor engine, a vocabulary including a set of content indices that reference corresponding cells in a descriptor space based on an input set of descriptors;

identifying, by the descriptor engine, a plurality of local features of an image based on the vocabulary, the local features being represented by a plurality of local descriptors;

determining, by the descriptor engine, an associated visual word in the vocabulary for each one of the plurality of local descriptors;

generating, by a signature engine coupled to the descriptor engine, a plurality of global signatures for the image based on the associated visual words, wherein some of the plurality of global signatures are generated using local descriptors corresponding to different cropped versions of the image, two or more of the different cropped versions of the image being centered at a same pixel location of the image; and

facilitating, by the signature engine, an image recognition search using the plurality of global signatures to search a document image dataset.

2. The method of claim 1, wherein the image is associated with an image dataset comprising a plurality of images.

3. The method of claim 2, wherein the image dataset comprises at least 10 million images.

4. The method of claim 1, further comprising:

dividing the image into k cluster centers; and

assigning each one of the plurality of local descriptors to a closest one of the k cluster centers.

5. The method of claim 4, wherein k equals 64 or 256.

6. The method of claim 4, further comprising, for each one of the k cluster centers, vector quantizing the local descriptors by recording residuals between vectors of each one of the local descriptors and the cluster centers to which the local descriptors are assigned.

7. The method of claim 6, further comprising concatenating sums of the residuals for each one of the k cluster centers to generate a global signature.

8. The method of claim 7, wherein the global signature is a k*128 dimensional vector of locally aggregated descriptors (VLAD) global signature.

9. The method of claim 1, further comprising compressing one or more of the plurality of global signatures.

10. The method of claim 9, wherein the one or more of the plurality of global signatures is compressed using a principal components analysis (PCA) projection matrix.

11. The method of claim 10, wherein the PCA projection matrix reduces a number of dimensions of the one or more of the plurality of global signatures by a factor of 100 or more.

12. The method of claim 1, wherein the image recognition search is for an object that is located at an off-center location within the image.

13. The method of claim 1, wherein the image recognition search is for an object that covers less than a majority portion of the image.

14. The method of claim 1, wherein the different cropped versions of the image are centered at two or more pixel locations of the image.

15. The method of claim 1, wherein a one of the plurality of global signatures generated using local descriptors corresponding to a cropped version of the image is centered at a different pixel location of the image with respect to another one of the plurality of global signatures.

16. The method of claim 1, wherein one of the plurality of global signatures is generated using local descriptors corresponding to an uncropped version of the image.

17. The method of claim 1, wherein the image is a query image.

18. The method of claim 1, wherein the plurality of global signatures includes at least four global signatures.

19. The method of claim 1, wherein the plurality of local descriptors are one of scale-invariant feature transform (SIFT) descriptors, Fast Retina Keypoint (FREAK) descriptors, Histograms of Oriented Gradient (HOG) descriptors, Speeded Up Robust Features (SURF) descriptors, DAISY descriptors, Binary Robust Invariant Scalable Keypoints (BRISK) descriptors, FAST descriptors, Binary Robust Independent Elementary Features (BRIEF) descriptors, Harris Corners descriptors, Edges descriptors, Gradient Location and Orientation Histogram (GLOH) descriptors, Electrooculography (EOG) descriptors or Transform Invariant Low-rank Textures (TILT) descriptors.

20. The method of claim 1, wherein a dictionary is used to determine the associated visual word in the vocabulary with each of the plurality of local descriptors.

21. The method of claim 20, wherein the dictionary comprises a VLAD dictionary in which a plurality of visual words are based on descriptors determined from a training sample of model images.

22. The method of claim 1, wherein the same pixel location corresponds to a center point of the image.

23. The method of claim 1, further comprising selecting at least one additional pixel location of the image;

and

generating, by the signature engine, one or more of the plurality of global signatures for the image using local descriptors corresponding to cropped versions of the image centered at the at least one additional pixel location.

24. The method of claim 1, wherein the plurality of global signatures includes at least one of a VLAD signature, a GIST signature or a Deep Learning signature.

25. The method of claim 1, wherein a distance measurement between global signatures representing images containing an object in common is less than a distance measurement between global signatures representing images that do not contain an object in common.

26. The method of claim 1, wherein the document image dataset comprises global signatures corresponding to each of a plurality of document images.

27. The method of claim 1, wherein the image is a digital representation of at least one of the following:

a vehicle, a face, a purchasable product, a printed media item, a vending machine, an appliance, a plant, a sign, biological material, a medical patient and a game component.

28. A system for large-scale image recognition, comprising:

a processor;

a main memory device;

a persistent storage device;

a descriptor engine executable on the processor according to software instructions stored in one of the main memory device and the persistent storage device and configured to;

obtain a vocabulary including a set of content indices that reference corresponding cells in a descriptor space based on an input set of descriptors;

identify a plurality of local features of an image based on the vocabulary, the local features being represented by a plurality of local descriptors; and

determine an associated visual word in the vocabulary for each one of the plurality of local descriptors; and

a signature engine, coupled to the descriptor engine, executable on the processor according to software instructions stored in one of the main memory device and the persistent storage device and configured to;

generate a plurality of global signatures for the image based on the associated visual words, wherein some of the plurality of global signatures are generated using local descriptors corresponding to different cropped versions of the image, two or more of the different cropped versions of the image being centered at a same pixel location of the image; and

facilitate an image recognition search using the plurality of global signatures to search a document image dataset.

29. A computer program product embedded in a non-transitory computer readable medium comprising instructions executable by a computer processor for large-scale image recognition, which, when executed by a processor, cause the processor to perform one or more steps comprising:

obtaining a vocabulary including a set of content indices that reference corresponding cells in a descriptor space based on an input set of descriptors;

identifying a plurality of local features of an image based on the vocabulary, the local features being represented by a plurality of local descriptors;

determining an associated visual word in the vocabulary for each one of the plurality of local descriptors;

generating a plurality of global signatures for the image based on the associated visual words, wherein some of the plurality of global signatures are generated using local descriptors corresponding to different cropped versions of the image, two or more of the different cropped versions of the image being centered at a same pixel location of the image; and

facilitating an image recognition search using the plurality of global signatures to search a document image dataset.

30. A system for large-scale image recognition, comprising:

means for obtaining a vocabulary including a set of content indices that reference corresponding cells in a descriptor space based on an input set of descriptors;

means for identifying a plurality of local features of an image based on the vocabulary, the local features being represented by a plurality of local descriptors;

means for determining an associated visual word in the vocabulary for each one of the plurality of local descriptors;

means for generating a plurality of global signatures for the image based on the associated visual words, wherein some of the plurality of global signatures are generated using local descriptors corresponding to different cropped versions of the image, two or more of the different cropped versions of the image being centered at a same pixel location of the image; and

means for facilitating an image recognition search using the plurality of global signatures to search a document image dataset.