Object outlining to initiate a visual search

• US 9,230,171 B2
• Filed: 02/20/2012
• Issued: 01/05/2016
• Est. Priority Date: 01/06/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Starner, Thad Eugene, Essa, Irfan
• Primary Examiner(s): Vaughn, Jr., William C
• Assistant Examiner(s): TEKLE, DANIEL T
• Application Number:
  US13/400,258
• Publication Number:
  US 20150227795A1
• Time in Patent Office: 1,415 Days
• Field of Search: 348/143
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20096 Interactive definition of curve of interest
• CPC Class Codes: G06T 2207/20112 Image segmentation details
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

Methods and devices for initiating a search of an object are disclosed. In one embodiment, a method is disclosed that includes receiving video data from a camera on a wearable computing device and, based on the video data, detecting a movement that defines an outline of an area in the video data. The method further includes identifying an object that is located in the area and initiating a search on the object. In another embodiment, a server is disclosed that includes an interface configured to receive video data from a camera on a wearable computing device, at least one processor, and data storage comprising instructions executable by the at least one processor to detect, based on the video data, a movement that defines an outline of an area in the video data, identify an object that is located in the area, and initiate a search on the object.

Claims

1. A method comprising:

receiving video data from a camera on a wearable computing device, wherein a real-world field of view corresponds to the wearable computing device;

based on the video data, detecting a movement of a pointing device within the real-world field of view;

determining that the movement of the pointing device defines at least a portion of an outline of an area in the video data;

identifying a portion of the video data that represents a real-world object and is substantially located in the determined area; and

initiating a visual search on the portion of the video data that represents the real-world object.

2. The method of claim 1, further comprising causing the wearable computing device to display information corresponding to results of the search.

3. The method of claim 1, wherein initiating the visual search comprises:

beginning the visual search before the movement is complete; and

periodically updating the visual search during the movement.

4. The method of claim 3:

wherein beginning the visual search comprises causing the wearable computing device to display information corresponding to results of the search; and

wherein periodically updating the visual search comprises causing the wearable computing device to display updated information corresponding to results of the updated search.

5. The method of claim 1, further comprising:

during the visual search, caching information corresponding to results of the visual search; and

causing the wearable computing device to display at least some of the cached information.

6. The method of claim 1, wherein:

the movement comprises movement of a pointing device; and

detecting the movement comprises tracking the pointing device.

7. The method of claim 6, wherein the pointing device comprises a finger.

8. The, method of claim 1 wherein:

the camera is configured to be offset from an eye of a user by an angle and a distance; and

detecting the movement is further based on the angle and the distance.

9. The method of claim 1, wherein identifying the object comprises:

using edge detection to detect one or more edges of the object in the outlined area in the video data, wherein the detected edges define the portion of the video data that represents the real-world object.

10. The method of claim 1, wherein identifying the object comprises:

segmenting the video data into layers; and

aligning the outline with at least one layer.

11. A device comprising:

an interface configured to receive video data from a camera on a wearable computing device, wherein a real-world field of view corresponds to the wearable computing device;

at least one processor; and

data storage comprising instructions executable by the at least one processor to;

based on the video data, detect a movement of a pointing device within the real-world field of view;

determining that the movement of the pointing device defines at least a portion of an outline of an area in the video data;

identify a portion of the video data that represents a real-world object and-is substantially located in the determined area; and

initiate a visual search on the portion of the video data that represents the real-world object.

12. The device of claim 11, further comprising a display configured to display information corresponding to results of the visual search.

13. The device of claim 12, wherein the display comprises a head-mounted display.

14. The device of claim 12, wherein the display is further configured to display the video data.

15. The device of claim 14, wherein the visual search comprises:

beginning the visual search before the movement is complete; and

periodically updating the visual search during the movement.

16. The device of claim 11, wherein:

the movement comprises movement of a pointing device; and

detecting the movement comprises tracking the pointing device.

17. A non-transitory computer-readable medium having stored therein instructions executable by a computing device to cause the computing device to perform functions comprising:

receiving video data from a camera on a wearable computing device, wherein a real-world field of view corresponds to the wearable computing device;

based on the video data, detecting a movement of a pointing device within the real-world field of view;

determining that the movement of the pointing device defines at least a portion of an outline of an area in the video data;

identifying a portion of the video data that represents a real-world object and is substantially located in the determined area; and

initiating a visual search on the portion of the video data that represents the real-world object.

18. The non-transitory computer-readable medium of claim 17, the functions further comprising causing the wearable computing device to display information corresponding to results of the visual search.

19. The non-transitory computer-readable medium of claim 17, wherein performing the visual search comprises:

beginning the visual search before the movement is complete; and

periodically updating the visual search during the movement.