Object occlusion to initiate a visual search

• US 10,437,882 B2
• Filed: 05/07/2015
• Issued: 10/08/2019
• Est. Priority Date: 01/06/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Starner, Thad Eugene, Essa, Irfan, Raffle, Hayes Solos, Aminzade, Daniel
• Primary Examiner(s): Patel, Sanjiv D.
• Application Number:
  US14/706,839
• Publication Number:
  US 20150242414A1
• Time in Patent Office: 1,615 Days
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 16/738 Presentation of query results
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Methods and devices for initiating a search of an object are disclosed. In one embodiment, a method is disclosed that includes receiving video data recorded by a camera on a wearable computing device, where the video data comprises at least a first frame and a second frame. The method further includes, based on the video data, detecting an area in the first frame that is at least partially bounded by a pointing device and, based on the video data, detecting in the second frame that the area is at least partially occluded by the pointing device. The method still further includes initiating a search on the area.

Claims

1. A method comprising:

receiving video data recorded by a camera on a wearable computing device, wherein the video data comprises at least a first frame and a second frame;

detecting a pinch gesture captured in the video data, wherein detecting the pinch gesture comprises both;

(a) detecting an area in the first frame that is at least partially bounded, in the first frame, by opposing digits of a pointing device, wherein the area comprises image data of real-world subject matter including an object; and

(b) within a predetermined period of time from detecting the area in the first frame that is at least partially bounded by the pointing device, detecting, in the second frame, that the opposing digits moved to at least partially occlude the real-world object; and

in response to detecting the pinch gesture, initiating a search on image content of the real-world object in the detected area in the first frame.

2. The method of claim 1, wherein the pointing device comprises a finger of a user.

3. The method of claim 1, wherein the search comprises a visual search.

4. The method of claim 1, further comprising causing the wearable computing device to display information corresponding to results of the search.

5. The method of claim 1, further comprising:

between recording the first frame and the second frame, detecting movement of the wearable computing device; and

aligning the second frame to the first frame based on the movement.

6. The method of claim 1, wherein:

the camera is configured to be offset from an eye of a user by an angle and a distance; and

detecting the area in the first frame is based on the angle and the distance.

7. The method of claim 1, wherein detecting the area in the first frame comprises:

segmenting the first frame into layers; and

selecting at least one layer as the area.

8. The method of claim 1, wherein initiating a search on image content in the detected area in the first frame comprises:

analyzing the video data to determine an aspect of the real-world environment that is captured in the detected area in the first frame; and

initiating a visual search on image content comprising the determined aspect of the real-world environment.

9. A wearable computing device comprising:

a camera configured to record video data comprising at least a first frame and a second frame;

at least one processor; and

data storage comprising instructions executable by the at least one processor to;

detect a pinch gesture captured in the video data, wherein detection of the pinch gesture comprises both;

(a) detection of an area in the first frame that is at least partially bounded, in the first frame, by opposing digits of a pointing device, wherein the area comprises image data of real-world subject matter including an object; and

(b) within a predetermined period of time from detection of the area in the first frame that is at least partially bounded by the pointing device, detection, in the second frame, that the opposing digits moved to at least partially occlude the real-world object; and

in response to detecting the pinch gesture, initiate a search on image content of the real-world object in the detected area in the first frame.

10. The wearable computing device of claim 9, further comprising a display configured to display information corresponding to results of the search.

11. The wearable computing device of claim 10, wherein the display comprises a head-mounted display.

12. The wearable computing device of claim 10, wherein the display is further configured to display the first frame and the second frame.

13. The wearable computing device of claim 9, wherein the pointing device comprises a finger of a user.

14. The wearable computing device of claim 9, wherein the search comprises a visual search.

15. The wearable computing device of claim 9, further comprising a movement sensor configured to sense movement of the wearable computing device.

16. The wearable computing device of claim 15, wherein the instructions are further executable by the at least one processor to align the second frame to the first frame based on the movement.

17. The wearable computing device of claim 9, wherein:

the camera is configured to be offset from an eye of a user by an angle and a distance; and

detecting the area in the first frame is based on the angle and the distance.

18. The wearable computing device of claim 9, wherein detecting the area in the first frame comprises:

segmenting the first frame into layers; and

selecting as the area at least one layer.

19. A method comprising:

receiving video data recorded by a camera on a computing device, wherein the video data comprises at least a first frame and a second frame;

detecting a pinch gesture, wherein detecting the pinch gesture comprises both;

(a) detecting an area in the first frame that is at least partially bounded, in the first frame, by opposing digits of a pointing device, wherein the area comprises image data of real-world subject matter including an object; and

(b) within a predetermined period of time from detecting the area in the first frame that is at least partially bounded by the pointing device, detecting, in the second frame, that the opposing digits moved to at least partially occlude the real-world object;

determining an adjustment based at least in part on a location of the camera relative to an eye location;

applying the determined adjustment to the first area to determine a second area in the first frame that corresponds to the pinch gesture in a field of view associated with the computing device; and

in response to detecting the pinch gesture, initiating a search on image content of the real-world object corresponding to the second area in the first frame.

20. The method of claim 19, wherein the pointing device comprises a finger of a user.