Methods and apparatus for visual search

• US 8,805,116 B2
• Filed: 03/29/2012
• Issued: 08/12/2014
• Est. Priority Date: 09/17/2011
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Lin, Zhe, Brandt, Jonathan W.
• Primary Examiner(s): Patel, Nirav G
• Assistant Examiner(s): Mistry, Oneal R
• Application Number:
  US13/434,061
• Publication Number:
  US 20130121570A1
• Time in Patent Office: 866 Days
• Field of Search: None
• US Class Current: 382/286
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5862 using texture
• CPC Class Codes: G06V 10/464 using a plurality of salient features, e.g. bag-of-words [BoW] representations

Abstract

For each image of a set of images, the each image is characterized with a set of fixed-orientation texture descriptors and a set of color descriptors. The set of images is indexed in a color index and a texture index. Similarly, a query image is characterized with a set of fixed-orientation texture descriptors. The set of fixed orientation texture descriptors of the query image includes a set of fixed orientation descriptors for each of a set of rotated query images, and a set of color descriptors of the query image. A rotated local Bag-of-Features (BoF) operation is performed upon the set of rotated query images and the set of images. Each of the set of images is ranked based on the rotated local Bag-of-Features operation.

Claims

1. A method, comprising:

executing an image analysis module on one or more processors of a computing device to perform;

characterizing each image in a set of images with a set of fixed-orientation texture descriptors and a set of color descriptors;

indexing the set of images in a color index comprising the set of color descriptors for each image of the set of images, and in a texture index comprising the set of fixed-orientation texture descriptors for each image of the set of images, the indexing comprising encoding spatial information in the set of color descriptors and in the set of fixed-orientation texture descriptors, the spatial information including feature location and scale as encoded in both the set of color descriptors and in the set of fixed-orientation texture descriptors;

characterizing a query image with the set of fixed-orientation texture descriptors of the query image, the set of fixed-orientation texture descriptors of the query image comprising a set of fixed orientation descriptors for each of a set of rotated query images, and with the set of color descriptors of the query image;

performing a rotated local Bag-of-Features (BoF) operation upon the set of rotated query images and the set of images; and

ranking each of the set of images based on the rotated local Bag-of-Features operation.

2. The method of claim 1, wherein the characterizing each image further comprises:

extracting from each image of the set of images a set of sparse feature regions and a set of dense feature regions;

calculating a texture descriptor for each of the set of sparse feature regions and each of the set of dense feature regions; and

calculating a color descriptor for each of the set of sparse feature regions and each of the set of dense feature regions.

3. The method of claim 1, wherein the indexing the set of images comprises:

building based on the set of color descriptors the color index as an inverted file; and

separately building based on the set of fixed orientation texture descriptors the texture index as another inverted file.

4. The method of claim 1, wherein the characterizing the query image comprises:

generating the set of rotated query images that represents the query image as transformed through at a set of rotation angles;

calculating the set of fixed orientation texture descriptors of the query image from the set of rotated query images; and

calculating for the query image a color descriptor.

5. The method of claim 4, further comprising:

extracting from the query image a set of sparse feature regions of fixed orientation and a set of dense feature regions of fixed orientation; and

wherein

the calculating the set of fixed orientation texture descriptors of the query image further comprises calculating for each transformed query image of the set of rotated query images;

a set of texture descriptors for the each of the set of sparse feature regions; and

a set of texture descriptors for the each of the set of dense feature regions.

6. The method of claim 1, wherein performing the rotated local BoF operation comprises scoring, for each of the images in the set of images, one or more regions that have similarity to at least one of the rotated query images.

7. The method of claim 6, wherein the scoring comprises taking a maximum of a set of similarity scores for the set of rotated query images with respect to the each of the set of images.

8. The method of claim 6, wherein the scoring further comprises scoring based on a probability of spatial correspondence between features based on scale and translation.

9. A system, comprising:

at least one processor; and

a memory comprising program instructions that are executable by the at least one processor as an image analysis module configured to;

characterize each image in a set of images with a set of fixed-orientation texture descriptors, and a set of color descriptors;

index the set of images in a color index comprising the set of color descriptors for each image of the set of images and in a texture index comprising the set of fixed-orientation texture descriptors for each image of the set of images, the indexing comprising encoding spatial information in the set of color descriptors and in the set of fixed-orientation texture descriptors, the spatial information including feature location and scale as encoded in both the set of color descriptors and in the set of fixed-orientation texture descriptors;

characterize a query image with the set of fixed-orientation texture descriptors of the query image, the set of fixed orientation texture descriptors of the query image comprising a set of fixed orientation descriptors for each of a set of rotated query images, and with the set of color descriptors of the query image;

perform a rotated local Bag-of-Features (BoF) operation upon the set of rotated query images and the set of images; and

rank each of the set of images based on the rotated local Bag-of-Features operation to generate an image ranking

10. The system of claim 9, further comprising instructions are executable by the at least one processor to:

wherein the image analysis module is configured to iteratively refine said ranking based on user selection of images presented as results of said ranking

11. The system of claim 9, wherein for each of the images, the texture index comprises the set of fixed-orientation texture descriptors and the set of color descriptors that characterize a set of dense features and a set of sparse features.

12. The system of claim 9, wherein to index the set of images, the image analysis module is configured to encode spatial information in the set of color descriptors and the set of fixed-orientation texture descriptors.

13. The system of claim 12, wherein the spatial information comprises scale and location information, and wherein to score the set of images, the image analysis module is configured to determine differences between the position of a feature of the query image and a position of a corresponding feature in each of the set of images.

14. The system of claim 9, wherein to perform the rotated local BoF operation, the image analysis module is configured to score, for each of the images in the set of images, one or more regions that have similarity to at least one of the rotated query images based on a user-weighted combination of color and texture.

15. A non-transitory computer-readable storage medium comprising stored program instructions that are computer-executable to implement:

characterizing each image in a set of images with a set of fixed-orientation texture descriptors, and a set of color descriptors;

indexing the set of images in a color index comprising the set of color descriptors for each image of the set of images and in a texture index comprising the set of fixed-orientation texture descriptors for each image of the set of images, the indexing comprising encoding spatial information in the set of color descriptors and in the set of fixed-orientation texture descriptors, the spatial information including feature location and scale as encoded in both the set of color descriptors and in the set of fixed-orientation texture descriptors;

characterizing a query image with the set of fixed-orientation texture descriptors of the query image, the set of fixed-orientation texture descriptors of the query image comprising a set of fixed orientation descriptors for each of a set of rotated query images, and with a set of color descriptors of the query image;

performing a rotated local Bag-of-Features (BoF) operation upon the set of rotated query images and the set of images; and

ranking each of the set of images based on the rotated local Bag-of-Features operation.

16. The non-transitory computer-readable storage medium of claim 15, wherein to implement said characterizing each image, the program instructions implement:

extracting from each image of the set of images a set of sparse feature regions and a set of dense feature regions;

calculating a texture descriptor for each of the set of sparse feature regions and each of the set of dense feature regions; and

calculating a color descriptor for each of the set of sparse feature regions and each of the set of dense feature regions.

17. The non-transitory computer-readable storage medium of claim 15, wherein to implement said indexing the set of images, the program instructions implement:

building based on the set of color descriptors the color index as an inverted file; and

separately building based on the set of fixed orientation texture descriptors the texture index as another inverted file.

18. The non-transitory computer-readable storage medium of claim 15, wherein to implement said characterizing the query image, the program instructions implement:

generating the set of rotated query images that represents the query image as transformed through at a set of rotation angles;

calculating the set of fixed orientation texture descriptors of the query image from the set of rotated query images; and

calculating for the query image a color descriptor.

19. The non-transitory computer-readable storage medium of claim 18, wherein the program instructions implement extracting from the query image a set of sparse feature regions of fixed orientation and a set of dense feature regions of fixed orientation, and wherein to implement said calculating the set of fixed orientation texture descriptors of the query image, the program instructions implement calculating for each transformed query image of the set of rotated query images:

a set of texture descriptors for the each of the set of sparse feature regions; and

a set of texture descriptors for the each of the set of dense feature regions.

20. The non-transitory computer-readable storage medium of claim 15, wherein to implement said performing the rotated local BoF operation, the program instructions implement scoring, for each of the images in the set of images, one or more regions that have similarity to at least one of the rotated query images.