Object outlining to initiate a visual search

• US 9,536,354 B2
• Filed: 12/04/2015
• Issued: 01/03/2017
• Est. Priority Date: 01/06/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Starner, Thad Eugene, Essa, Irfan
• Primary Examiner(s): Vaughn, Jr., William C
• Assistant Examiner(s): TEKLE, DANIEL T
• Application Number:
  US14/959,245
• Publication Number:
  US 20160086383A1
• Time in Patent Office: 396 Days
• Field of Search: 348/143
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20096 Interactive definition of curve of interest
• CPC Class Codes: G06T 2207/20112 Image segmentation details
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

Methods and devices for initiating a search of an object are disclosed. In one embodiment, a method is disclosed that includes receiving sensor data from a sensor on a wearable computing device and, based on the sensor data, detecting a movement that defines an outline of an area in the sensor data. The method further includes identifying an object that is located in the area and initiating a search on the object. In another embodiment, a server is disclosed that includes an interface configured to receive sensor data from a sensor on a wearable computing device, at least one processor, and data storage comprising instructions executable by the at least one processor to detect, based on the sensor data, a movement that defines an outline of an area in the sensor data, identify an object that is located in the area, and initiate a search on the object.

Claims

1. A method comprising:

receiving video data from a camera on a wearable computing device, wherein a real-world field of view corresponds to the wearable computing device;

based on the video data, detecting a first movement of a pointing device in the real-world field of view;

determining that a real-world object is partially outlined by the first movement;

determining an identity of an object that is located in an area of the video data corresponding to the partial outline defined by the first movement of the pointing device;

initiating a search on the real-world object and outputting corresponding information related to the real-world object at the wearable computing device;

detecting, based on the video data, a second movement of a pointing device in the real-world field of view, wherein the second movement is a continuation of the first movement that extends the partial outline to provide a further-defined outline the real-world object and

responsive to detecting the second movement extending the outline of the real-world object, outputting additional information related to the real-world object at the wearable computing device.

2. The method of claim 1, further comprising causing the wearable computing device to display the additional information.

3. The method of claim 1, wherein initiating the search comprises:

beginning the search before the movement is complete.

4. The method of claim 3, wherein:

beginning the search comprises causing the wearable computing device to provide information corresponding to results of the search.

5. The method of claim 1, further comprising:

during the search, caching information corresponding to results of the search; and

causing the wearable computing device to provide at least some of the cached information.

6. The method of claim 1, wherein the search comprises a visual search.

7. The method of claim 1, wherein the pointing device comprises a finger.

8. The, method of claim 1 wherein:

the sensor is configured to be offset from an eye of a user by an angle and a distance; and

detecting the movement is further based on the angle and the distance.

9. The method of claim 1, wherein determining the identity of the object comprises:

using edge detection to detect edges of the object; and

aligning the outline with at least one edge of the object.

10. The method of claim 1, wherein determining the identity of the object comprises:

segmenting the sensor data into layers; and

aligning the outline with at least one layer.

11. The method of claim 1, wherein outputting the information related to the real-world object at the wearable computing device comprises playing out audio.

12. The method of claim 1, wherein outputting the additional information related to the real-world object at the wearable computing device comprises playing out audio.

13. A wearable computing device comprising:

an interface configured to receive video data from a camera on a wearable computing device, wherein a real-world field of view corresponds to the wearable computing device;

at least one processor; and

data storage comprising instructions executable by the at least one processor to;

based on the video data, detect a first movement of a pointing device in the real-world field of view;

determining that a real-world object is partially outlined by the first movement;

determine an identity of an object that is located in an area of the video data corresponding to the partial outline defined by the first movement of the pointing device;

initiate a search on the real-world object and outputting corresponding information related to the real-world object at the wearable computing device;

detect, based on the video data, a second movement of a pointing device in the real-world field of view, wherein the second movement is a continuation of the first movement that extends the partial outline to provide a further-defined outline the real-world object; and

responsive to detecting the second movement extending the outline of the real-world object, output additional information related to the real-world object at the wearable computing device.

14. The wearable computing device of claim 13, further comprising a speaker configured to provide audio information corresponding to results of the search.

15. The wearable computing device of claim 14, wherein the device comprises a head-mountable device.

16. The wearable computing device of claim 13, wherein performing the search comprises:

beginning the search before the movement is complete.

17. A non-transitory computer-readable medium having stored therein instructions executable by a computing device to cause the computing device to perform functions comprising:

receiving video data from a camera on a wearable computing device, wherein a real-world field of view corresponds to the wearable computing device;

based on the video data, detecting a first movement of a pointing device in the real-world field of view;

determining that a real-world object is partially outlined by the first movement;

determining an identity of an object that is located in an area of the video data corresponding to the partial outline defined by the first movement of the pointing device;

initiating a search on the real-world object and outputting corresponding information related to the real-world object at the wearable computing device;

detecting, based on the video data, a second movement of a pointing device in the real-world field of view, wherein the second movement is a continuation of the first movement that extends the partial outline to provide a further-defined outline the real-world object;

responsive to detecting the second movement extending the outline of the real-world object, outputting additional information related to the real-world object at the wearable computing device.

18. The non-transitory computer-readable medium of claim 17, wherein performing the search comprises:

beginning the search before the movement is complete.