Gesture-based visual search

• US 8,831,349 B2
• Filed: 09/05/2013
• Issued: 09/09/2014
• Est. Priority Date: 05/17/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Mei, Tao, Li, Shipeng, Xu, Ying-Qing, Zhang, Ning, Chen, Zheng, Sun, Jian-Tao
• Primary Examiner(s): PATEL, KANJIBHAI B
• Application Number:
  US14/019,259
• Publication Number:
  US 20140003714A1
• Time in Patent Office: 369 Days
• Field of Search: 382/164, 382/173, 382/282, 382/305, 382/306, 382/312, 700/2, 707/706, 707/713, 707/763, 707/765, 707/767, 345/428, 235/462.45
• US Class Current: 382/173
• CPC Class Codes: G06F 16/434 using image data, e.g. images, photos, pictures taken by a user
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5846 using extracted text
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A user may perform an image search on an object shown in an image. The user may use a mobile device to display an image. In response to displaying the image, the client device may send the image to a visual search system for image segmentation. Upon receiving a segmented image from the visual search system, the client device may display the segmented image to the user who may select one or more segments including an object of interest to instantiate a search. The visual search system may formulate a search query based on the one or more selected segments and perform a search using the search query. The visual search system may then return search results to the client device for display to the user.

Claims

1. A method comprising:

under control of one or more processors of a client device that are configured with executable instructions;

displaying an image on a display of the client device, the image including a plurality of segments;

receiving a selection gesture to select one or more segments from the plurality of segments;

receiving contextual information of the image; and

instantiating a search query based on the one or more selected segments and the contextual information of the image.

2. The method as recited in claim 1, further comprising, prior to displaying the image, segmenting, at the client device, the image into the plurality of segments.

3. The method as recited in claim 1, further comprising, prior to displaying the image:

providing the image to a gesture-based visual search system for segmentation; and

receiving the plurality of segments of the image from the gesture-based visual search system.

4. The method as recited in claim 1, further comprising:

sending the search query to a search engine; and

receiving search results from the search engine.

5. The method as recited in claim 1, wherein receiving the selection gesture comprises receiving one or more touch inputs to a user interface of the client device, the one or more touch inputs corresponding to the one or more selected segments.

6. The method as recited in claim 1, wherein receiving the selection gesture comprises receiving a bounding gesture that bounds or substantially bounds the one or more selected segments.

7. The method as recited in claim 1, further comprising, prior to displaying the image:

obtaining the image by;

capturing the image through a camera of the client device;

receiving a selection of the image from a photo application of the client device;

or

receiving a selection of the image from media stored on or accessible to the client device.

8. A client device comprising:

a display;

one or more processors;

memory storing executable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising;

displaying an image on the display, the image including a plurality of segments;

receiving a selection gesture to select one or more segments from the plurality of segments;

receiving contextual information of the image; and

instantiating a search query based on the one or more selected segments and the contextual information of the image.

9. The client device as recited in claim 8, the acts further comprising, prior to displaying the image segmenting, at the client device, the image into the plurality of segments.

10. The client device as recited in claim 8, the acts further comprising, prior to displaying the image:

providing the image to a gesture-based visual search system for segmentation; and

receiving the plurality of segments of the image from the gesture-based visual search system.

11. The client device as recited in claim 8, the acts further comprising:

sending the search query to a search engine; and

receiving search results from the search engine.

12. The client device as recited in claim 8, wherein receiving the selection gesture comprises receiving one or more touch inputs to a user interface of the client device, the one or more touch inputs corresponding to the one or more selected segments.

13. The client device as recited in claim 8, wherein receiving the selection gesture comprises receiving a bounding gesture that bounds or substantially bounds the one or more selected segments.

14. The client device as recited in claim 8, the acts further comprising, prior to displaying the image:

obtaining the image by;

capturing the image through a camera of the client device;

receiving a selection of the image from a photo application of the client device;

or

receiving a selection of the image from media stored on or accessible to the client device.

15. One or more computer-storage media configured with computer-executable instructions that, when executed by one or more processors, configure the one or more processors to perform acts comprising:

receiving an indication of an object of interest in an image that is displayed on a client device;

sending the indication and the image to a search system; and

receiving the image having a first area and a second area, the first area being equal to or greater than an area corresponding to the indication of the object of interest and the second area being a remainder of the image that is different from the first area, wherein the first area is segmented without segmenting the second area.

16. The one or more computer-storage media as recited in claim 15, wherein receiving the indication of the object of interest comprises receiving a bounding gesture that substantially bounds the object of interest.

17. The one or more computer-storage media as recited in claim 15, the acts further comprise:

receiving a selection gesture to select one or more segments from the area corresponding to the indication; and

instantiating a search query based on the one or more selected segments.

18. The one or more computer-storage media as recited in claim 17, the acts further comprising:

sending the search query to a search engine; and

receiving search results from the search engine.