APPARATUS AND METHOD FOR PERFORMING VISUAL SEARCH

• US 20140016863A1
• Filed: 06/27/2013
• Published: 01/16/2014
• Est. Priority Date: 07/06/2012
• Status: Active Grant

• Current Assignee: Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.)
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Fernandes, Felix Carlos, Saxena, Ankur, Bucak, Serhat, Nagar, Abhishek, Srivastava, Gaurav
• Granted Patent: US 9,256,617 B2
• Time in Patent Office: Days
• US Class Current: 382/165
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06V 10/464 using a plurality of salient features, e.g. bag-of-words [BoW] representations
• CPC Class Codes: G06V 10/56 relating to colour

Abstract

A method comprises computing a color factor value indicating an amount of color gradients in at least one color channel from the query image. The method comprises combining the color-keypoints with the gray-keypoints when the color factor value is greater than a threshold. A method for performing a visual search comprises extracting a plurality of local descriptors from a query image and then selecting a subset of them based on various criteria'"'"'s such as visual meaning score. A method comprises aggregating each mean vector for each visual codeword from distances between each visual codeword and local descriptors. The method comprises aggregating variance vector for each visual codeword from the distance between each visual codeword, and local descriptors. The method comprises transmitting aggregated mean vector information and aggregated variance vector information to a search server for efficient image retrieval.

Claims

1. A method for performing a visual search, the method comprising,

extracting gray-keypoints from a grayscale of a query image;

computing a color factor value indicating an amount of color gradient in at least one color channel from the query image;

extracting color-keypoints from at least one color channel when the value of the color fact is greater than a threshold; and

combining the color-keypoints with the gray-keypoints.

2. The method of claim 1 further comprising closing a switch to activate the extracting color-keypoints from at least one color channel when the color factor value is greater than the predetermined threshold.

3. The method of claim 1, the combining of the color-keypoints comprising:

including extracted gray-keypoints into a pool;

selecting one keypoint with minimum spatial distance from a color-keypoint, out of gray-keypoints of the pool;

computing differences of spatial distance, orientation and scale between the selected one keypoint and the color-keypoint; and

including the color-keypoint into the pool when one of the differences is greater than a corresponding threshold value.

4. The method of claim 3 further comprising:

replacing one keypoint with the color-keypoint when the peak value of the color-keypoint is greater than that of the one keypoint.

5. The method of claim 1, wherein the color factor is calculated as follows:

6. The method of claim 5, wherein at least one color channel includes red, green, and blue channels, and the color factor is calculated as follows:

7. The method of claim 1, wherein the color factor is appended to a global descriptor transmitted to a search server.

8. An apparatus for performing a visual search, the apparatus comprising:

a controller configured to

extract gray-keypoints from a grayscale of a query image;

compute a color factor value indicating an amount of color gradient in at least one color channel from the query image;

extract color-keypoints from at least one color channel when the value of the color factor is greater than a predetermined threshold; and

combine the color-keypoints with the gray-keypoints.

9. The apparatus of claim 8, further comprising a switch configured to activate extracting color-keypoints from at least one color channel when the color factor value is greater than the predetermined threshold.

10. A method for performing a visual search, the method comprising:

extracting gray-keypoints from a grayscale of a query image;

computing color factor values indicating an amount of color gradient in at least one color channel separately from various patches or blocks of an image;

extracting color-keypoints from at least one color channel and from patches or blocks when the value of the color factor corresponding to that patch or block is greater than a threshold; and

combining the color-keypoints with the gray-keypoints.

11. The method of claim 10, wherein the patches have different granularities.

12. An apparatus for performing a visual search, the apparatus comprising:

a controller configured to

extract gray-keypoints from a grayscale of a query image;

compute color factor values indicating an amount of color gradient in at least one color channel separately from various patches or blocks of an image;

extract color-keypoints from at least one color channel and from patches or blocks when the value of the color factor corresponding to that patch or block is greater than a threshold; and

combine the color-keypoints with the gray-keypoints.

13. The apparatus of claim 12, wherein patches have different granularities.

14. A method for performing a visual search, the method comprising:

extracting gray-keypoints from a grayscale of a query image;

selecting a plurality of local descriptors whose scale is greater than a predetermined size;

calculating a convex hull of the selected local descriptors; and

distinguishing local descriptors which are located within the convex hull from the other local descriptors.

15. The method of claim 14, wherein the distinguishing of local descriptors is based on scoring the local descriptors which are located within the convex hull higher than the other local descriptors.

16. A method for performing a visual search, the method comprising,

extracting local descriptors from a query image;

computing a visual meaning score based on a distance between each local descriptor and a closest visual word;

calculating a score for each local descriptor based on the visual meaning score; and

selecting at least one local descriptor based on the score.

17. The method of claim 16, wherein the distance between the local descriptor and the visual word that is closest to the local descriptor can be formulated as follows:

18. A method for performing a visual search, the method comprising:

extracting a plurality of local descriptors from a query image;

categorizing local descriptors matching each visual word as positive class, and non-matching local descriptors as negative class;

calculating a ratio of sizes of the positive class and the negative class for each visual word; and

weighting higher the local descriptors in the positive class in proportion to the ratio.

19. A method for performing a visual search, the method comprising, extracting a plurality of local descriptors from a query image;

selecting m of local descriptors where m is greater than a transmission bitrate;

obtaining each quantizing error from quantizing each selected local descriptor;

assigning scores to each selected local descriptor based on each quantizing error; and

re-selecting a portion of m of local descriptors based on the scores.

20. A method for performing a visual search, the method comprising:

extracting a plurality of local descriptors from a query image;

aggregating each mean vector for each visual codeword from distances between each visual codeword and local descriptors;

aggregating variance vector for each visual codeword from each mean vector; and

transmitting aggregated mean vector information and aggregated variance vector information to a search server.

21. The method of claim 20, wherein the mean vector, S_i is calculated as follows:

22. The method of claim 21, wherein the variance vector, V_i is calculated as follows:

V_i=(1/N_iΣ

_j=1^Ni(v_ij)²)−

(S_i)²

where square operation on vectors S_i and v_ij are performed element-wise.

23. The method of claim 21, wherein the aggregated mean vector information and the aggregated variance vector information are transmitted without increasing a bit-rate.

24. The method of claim 23, wherein the aggregated mean vector information is transmitted using a K dimensional vector and the aggregated variance vector information is transmitted using a (32−

K) dimensional vector, where K<

32.

25. An apparatus for performing a visual search, the apparatus comprising:

a controller configured to

extract a plurality of local descriptors from a query image;

aggregate each mean vector for each visual codeword from distances between each visual codeword and local descriptors;

aggregate variance vector for each visual codeword from each mean vector; and

transmit aggregated mean vector information and aggregated variance vector information to a search server.

26. A method for performing a visual search, the method comprising:

receiving aggregated mean vector information and aggregated variance vector information from a client device;

retrieving each mean vector for each visual codeword from mean vector information;

retrieving each variance vector for each visual codeword from the variance vector information, using each mean vector; and

searching for a matching image, using each mean vector and each variance vector for each visual codeword.

27. An apparatus for performing a visual search, the apparatus comprising:

a controller configured to

receive aggregated mean vector information and aggregated variance vector information from a client device;

retrieve each mean vector for each visual codeword from mean vector information;

retrieve each variance vector for each visual codeword from the variance vector information, using each mean vector; and

search for a matching image, using each mean vector and each variance vector for each visual codeword.