INCREMENTAL VISUAL QUERY PROCESSING WITH HOLISTIC FEATURE FEEDBACK

• US 20140101195A1
• Filed: 10/08/2013
• Published: 04/10/2014
• Est. Priority Date: 10/10/2012
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Saxena, Ankur, Nagar, Abhishek, Srivastava, Gaurav, Bhat, Kong Posh, Li, Zhu
• Granted Patent: US 9,727,586 B2
• Time in Patent Office: Days
• US Class Current: 707/769
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]

Abstract

To reduce communication costs and computational complexity, only a subset of ranked SIFT points within a query image for a visual search request is transmitted to the visual search server in each iteration of an incremental search. For each candidate match, a flag identifying the matching points is returned by the server for use in computing holistic (e.g., histogram) information for a bounding box within the query image including the matching points. Distance from that query image holistic information is used to reject images from a short list used for a subsequent iteration, if any. If all images are rejected or a match criteria is met during one iteration, the search may terminate early without consideration of remaining SIFT points.

Claims

1. A method, comprising:

receiving, at a visual search server, information relating to a first subset of distinctive features within a query image for a visual search request;

for at least a subset of stored image information accessible to the visual search server, determining, at a visual search server, one or more sets of stored image information in which information relating to distinctive features for a respective image corresponds to at least a subset of the query image information; and

for each of the one or more sets of stored image information in which information relating to distinctive features for a respective image corresponds to at least a subset of the query image information,

transmitting a flag identifying the corresponding stored image information from the visual search server to a source of the visual search request, and

using holistic information received at the visual search server from the visual search request source, the holistic information based upon a region of the query image selected using the flag, to evaluate a match between the respective image and the query image.

2. The method according to claim 1, wherein the subset of stored image information comprises a short list of sets of stored image information in which information relating to distinctive features for a respective image corresponded to at least a subset of information relating to a second subset of distinctive features within the query image.

3. The method according to claim 1, wherein the holistic information comprises one or more of a color histogram for the query image region, a pixel intensity histogram for the query image region, an edge histogram for the query image region, and an appearance model for the query image region.

4. The method according to claim 1, wherein the information relating to the first subset of distinctive features within the query image comprises one or more of a ranked edge strength and a ranked peak strength.

5. The method according to claim 1, wherein the first subset of distinctive features within the query image comprise k of n distinctive features, where k<

n.

6. A visual search server, comprising:

a network connection configured to receive information relating to a first subset of distinctive features within a query image for a visual search request; and

a processor configured

to determine, for at least a subset of stored image information accessible to the visual search server, one or more sets of stored image information in which information relating to distinctive features for a respective image corresponds to at least a subset of the query image information, and

for each of the one or more sets of stored image information in which information relating to distinctive features for a respective image corresponds to at least a subset of the query image information,

to transmit a flag identifying the corresponding stored image information from the visual search server to a source of the visual search request, and

to use holistic information received at the visual search server from the visual search request source, the holistic information based upon a region of the query image selected using the flag, to evaluate a match between the respective image and the query image.

7. The visual search server according to claim 6, wherein the subset of stored image information comprises a short list of sets of stored image information in which information relating to distinctive features for a respective image corresponded to at least a subset of information relating to a second subset of distinctive features within the query image.

8. The visual search server according to claim 6, wherein the holistic information comprises one or more of a color histogram for the query image region, a pixel intensity histogram for the query image region, an edge histogram for the query image region, and an appearance model for the query image region.

9. The visual search server according to claim 6, wherein the information relating to the first subset of distinctive features within the query image comprises one or more of a ranked edge strength and a ranked peak strength.

10. The visual search server according to claim 6, wherein the first subset of distinctive features within the query image comprise k of n distinctive features, where k<

n.

11. A method, comprising:

transmitting a visual search request containing information relating to a first subset of distinctive features within a query image from a mobile device to a visual search server;

receiving, for each of the one or more sets of stored image information accessible to the visual search server in which information relating to distinctive features for a respective image corresponds to at least a subset of the query image information, a flag identifying the corresponding stored image information from the visual search server;

computing holistic information based upon a region of the query image selected using the flag; and

transmitting the holisitic information to the visual search server for use in evaluating a match between the respective image and the query image.

12. The method according to claim 11, wherein the subset of stored image information comprises a short list of sets of stored image information in which information relating to distinctive features for a respective image corresponded to at least a subset of information relating to a second subset of distinctive features within the query image.

13. The method according to claim 11, wherein the holistic information comprises one or more of a color histogram for the query image region, a pixel intensity histogram for the query image region, an edge histogram for the query image region, and an appearance model for the query image region.

14. The method according to claim 11, wherein the information relating to the first subset of distinctive features within the query image comprises one or more of a ranked edge strength and a ranked peak strength.

15. The method according to claim 11, wherein the first subset of distinctive features within the query image comprise k of n distinctive features, where k<

n.

16. A mobile device, comprising:

a wireless data connection configured

to transmit a visual search request containing information relating to a first subset of distinctive features within a query image from a mobile device to a visual search server, and

to receive, for each of the one or more sets of stored image information accessible to the visual search server in which information relating to distinctive features for a respective image corresponds to at least a subset of the query image information, a flag identifying the corresponding stored image information from the visual search server; and

a processor configured to compute holistic information based upon a region of the query image selected using the flag,

wherein the wireless data connection is further configured to transmit the holisitic information to the visual search server for use in evaluating a match between the respective image and the query image.

17. The mobile device according to claim 16, wherein the subset of stored image information comprises a short list of sets of stored image information in which information relating to distinctive features for a respective image corresponded to at least a subset of information relating to a second subset of distinctive features within the query image.

18. The mobile device according to claim 16, wherein the holistic information comprises one or more of a color histogram for the query image region, a pixel intensity histogram for the query image region, an edge histogram for the query image region, and an appearance model for the query image region.

19. The mobile device according to claim 16, wherein the information relating to the first subset of distinctive features within the query image comprises one or more of a ranked edge strength and a ranked peak strength.

20. The mobile device according to claim 16, wherein the first subset of distinctive features within the query image comprise k of n distinctive features, where k<

n.