GESTURE-BASED VISUAL SEARCH

• US 20120294520A1
• Filed: 05/17/2011
• Published: 11/22/2012
• Est. Priority Date: 05/17/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Mei, Tao, Li, Shipeng, Xu, Ying-Qing, Chen, Zheng, Sun, Jian-Tao, Zhang, Ning
• Granted Patent: US 8,553,981 B2
• Time in Patent Office: Days
• US Class Current: 382/164
• CPC Class Codes: G06F 16/434 using image data, e.g. images, photos, pictures taken by a user
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5846 using extracted text
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A user may perform an image search on an object shown in an image. The user may use a mobile device to display an image. In response to displaying the image, the client device may send the image to a visual search system for image segmentation. Upon receiving a segmented image from the visual search system, the client device may display the segmented image to the user who may select one or more segments including an object of interest to instantiate a search. The visual search system may formulate a search query based on the one or more selected segments and perform a search using the search query. The visual search system may then return search results to the client device for display to the user.

Claims

1. A method comprising:

under control of one or more processors configured with executable instructions;

receiving an image from a client;

segmenting the image into a plurality of segments;

sending at least a portion of the segmented image to the client;

receiving, from the client, one or more selected segments of the plurality of segments; and

in response to receiving the one or more selected segments from the client, formulating a query based on the one or more selected segments.

2. The method as recited in claim 1, further comprising:

obtaining search results based on the query; and

sending the search results to the client.

3. The method as recited in claim 1, further comprising receiving contextual information of the image, wherein formulating the query is further based on the received contextual information of the image.

4. The method as recited in claim 1, wherein formulating the query comprises:

recognizing an object in the one or more selected segments; and

formulating the query based on the recognized object in the one or more selected segment.

5. The method as recited in claim 1, wherein formulating the query comprises:

extracting features from the one or more selected segments;

determining visual words from a vocabulary tree based on the extracted features; and

formulating the query based on the determined visual words.

6. The method as recited in claim 1, wherein formulating the query comprising:

recognizing a text in the one or more selected segments; and

formulating the query based on the recognized text.

7. The method as recited in claim 1, wherein the image includes an image captured by the client, an image shown in a web page of a web browser application, an image selected from an album of the client and/or an image frame of a video played on the client.

8. The method as recited in claim 1, wherein segmenting the image comprises segmenting the image based on color of the image.

9. The method as recited in claim 1, further comprising receiving type information of an object of interest in the image from the client, wherein segmenting the image comprises segmenting the image based on the received type information.

10. The method as recited in claim 9, wherein segmenting the image based on the received type information comprises:

recognizing one or more objects having a type in accordance with the received type information from the image;

segmenting the one or more objects from the image; and

rendering the one or more objects as the plurality of segments.

11. The method as recited in claim 1, wherein the portion of the segmented image comprises less than all of the segmented image.

12. One or more computer-readable media configured with computer-executable instructions that, when executed by one or more processors, configure the one or more processors to perform acts comprising:

displaying an image on a display of a client device, the image including a plurality of segments;

receiving a selection gesture to select one or more segments from the plurality of segments;

instantiating a search query based on the one or more selected segments.

13. The one or more computer-readable media as recited in claim 12, the acts further comprising, prior to displaying the image segmenting, at the client device, the image into the plurality of segments.

14. The one or more computer-readable media as recited in claim 12, the acts further comprising, prior to displaying the image:

providing the image to a gesture-based visual search system for segmentation; and

receiving the plurality of segments of the image from the gesture-based visual search system.

15. The one or more computer-readable media as recited in claim 12, wherein the acts further comprise:

sending the search query to a search engine; and

receiving search results from the search engine.

16. The one or more computer-readable media as recited in claim 12, wherein receiving the selection gesture comprises receiving one or more touch inputs to a user interface of the client device, the one or more touch inputs corresponding to the one or more selected segments.

17. The one or more computer-readable media as recited in claim 12, wherein receiving the selection gesture comprises receiving a bounding gesture that bounds or substantially bounds the one or more selected segments.

18. The one or more computer-readable media as recited in claim 12, the acts further comprising, prior to displaying the image:

obtaining the image by;

capturing the image through a camera of the client device;

selecting the image from a photo application of the client device;

or

selecting the image from media stored on or accessible to the client device.

19. The one or more computer-readable media as recited in claim 12, the acts further comprising receiving contextual information of the image, wherein instantiating the search query is further based on the contextual information of the image.

20. A method comprising:

under control of one or more processors configured with executable instructions;

receiving an image and contextual information associated with the image from a client;

segmenting the image into a plurality of segments based on the contextual information associated with the image;

returning the segmented image in a form of the plurality of segments to the client;

receiving a selection of one or more segments of the plurality of segments from the client;

formulating a search query based on the one or more selected segments and the contextual information associated with the image;

performing a search using the search query to obtain search results; and

returning the search results to the client.