Object occlusion to initiate a visual search

• US 9,052,804 B1
• Filed: 03/12/2012
• Issued: 06/09/2015
• Est. Priority Date: 01/06/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Starner, Thad Eugene, Essa, Irfan, Raffle, Hayes Solos, Aminzade, Daniel
• Primary Examiner(s): Eisen, Alexander
• Assistant Examiner(s): Patel, Sanjiv D
• Application Number:
  US13/417,508
• Time in Patent Office: 1,184 Days
• Field of Search: 382/165, 382/199
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 16/738 Presentation of query results
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Methods and devices for initiating a search of an object are disclosed. In one embodiment, a method is disclosed that includes receiving video data recorded by a camera on a wearable computing device, where the video data comprises at least a first frame and a second frame. The method further includes, based on the video data, detecting an area in the first frame that is at least partially bounded by a pointing device and, based on the video data, detecting in the second frame that the area is at least partially occluded by the pointing device. The method still further includes initiating a search on the area.

Claims

1. A method comprising:

receiving video data recorded by an outward-facing camera on a wearable computing device, wherein the video data comprises at least a first frame and a second frame, wherein the video data captures a real-world environment of the wearable computing device, and wherein an object from the real-world environment is captured in the first frame;

detecting an object from the real-world environment in the first frame;

detecting a pinch gesture that corresponds to the object from the real-world environment, wherein detecting the pinch gesture comprises;

detecting an area in the first frame that is at least partially bounded by a pointing device;

determining that the at least partially-bounded area of the first frame includes at least a portion of the object;

within a predetermined period of time from detecting the area in the first frame that is at least partially bounded by the pointing device, detecting an area in the second frame that is at least partially occluded by the pointing device; and

determining that the at least partially-occluded area of the second frame includes at least a portion of the object; and

in response to detecting the pinch gesture that corresponds to the object from the real-world environment, initiating a search on the object from the real-world environment.

2. The method of claim 1, wherein the pointing device comprises at least two fingers.

3. The method of claim 1, wherein initiating the search on the object comprises initiating a visual search on image content comprising the object.

4. The method of claim 1, further comprising causing the wearable computing device to display information corresponding to results of the search.

5. The method of claim 1, further comprising:

between recording the first frame and the second frame, detecting movement of the wearable computing device; and

aligning the second frame to the first frame based on the movement;

wherein determining that the at least partially-occluded area of the second frame includes at least a portion of the object comprises determining that the at least partially-occluded area of the second frame is in a substantially same area of the frame as the at least partially bounded area in the first frame.

6. The method of claim 1, wherein:

the camera is configured to be offset from an eye of a user by an angle and a distance; and

detecting the area in the first frame is based on the angle and the distance.

7. The method of claim 1, wherein detecting the area in the first frame comprises:

segmenting the first frame into layers; and

selecting at least one layer as the area.

21. The method of claim 1, further comprising:

aligning the second frame to the first frame based on the movement;

wherein determining that the at least partially-occluded area of the second frame includes at least a portion of the object comprises determining that the at least partially-occluded area of the second frame is in a substantially same area of the frame as the at least partially bounded area in the first frame.

8. A wearable computing device comprising:

an outward-facing camera configured to record video data comprising at least a first frame and a second frame, wherein the video data captures a real-world environment of the wearable computing device, and wherein an object from the real-world environment is captured in the first frame;

at least one processor; and

data storage comprising instructions executable by the at least one processor to;

detect an object from the real-world environment in the first frame;

detect a pinch gesture that corresponds to the object from the real-world environment, wherein detection of the pinch gesture comprises both;

detection of an area in the first frame that is at least partially bounded by a pointing device;

a determination that the at least partially-bounded area of the first frame includes at least a portion of the object;

within a predetermined period of time from detection of the area in the first frame that is at least partially bounded by the pointing device, detection of an area in the second frame that is at least partially occluded by the pointing device; and

a determination that the at least partially-occluded area of the second frame includes at least a portion of the object; and

in response to detection of the pinch gesture that corresponds to the object from the real-world environment, initiate a search on the object from the real-world environment.

9. The wearable computing device of claim 8, further comprising a display configured to display information corresponding to results of the search.

10. The wearable computing device of claim 9, wherein the display comprises a head-mounted display.

11. The wearable computing device of claim 9, wherein the display is further configured to display the first frame and the second frame.

12. The wearable computing device of claim 8, wherein the pointing device comprises at least two fingers.

13. The wearable computing device of claim 8, wherein the search comprises a visual search.

14. The wearable computing device of claim 8, further comprising a movement sensor configured to sense movement of the wearable computing device.

15. The wearable computing device of claim 14, wherein the instructions are further executable by the at least one processor to align the second frame to the first frame based on the movement.

16. The wearable computing device of claim 8, wherein:

the camera is configured to be offset from an eye of a user by an angle and a distance; and

detecting the area in the first frame is based on the angle and the distance.

17. The wearable computing device of claim 8, wherein detecting the area in the first frame comprises:

segmenting the first frame into layers; and

selecting as the area at least one layer.

18. A non-transitory computer-readable medium having stored therein instructions executable by a computing device to cause the computing device to perform functions comprising:

using an outward-facing camera on a wearable computing device to record a video comprising at least a first frame and a second frame, wherein the video data captures a real-world environment of the wearable computing device, and wherein an object from the real-world environment is captured in the first frame;

detecting an object from the real-world environment in the first frame;

detecting a pinch gesture that corresponds to the object from the real-world environment, wherein detecting the pinch gesture comprises both;

detecting an area in the first frame that is at least partially bounded by a pointing device;

determining that the at least partially-bounded area of the first frame includes at least a portion of the object;

within a predetermined period of time from detecting the area in the first frame that is at least partially bounded by the pointing device, detecting an area in the second frame that is at least partially occluded by the pointing device; and

determining that the at least partially-occluded area of the second frame includes at least a portion of the object; and

in response to detecting the pinch gesture that corresponds to the object from the real-world environment, initiating a search on the object from the real-world environment.

19. The non-transitory computer-readable medium of claim 18, wherein the pointing device comprises at least two fingers.

20. The non-transitory computer-readable medium of claim 18, wherein the search comprises a visual search.