Using visual layers to aid in initiating a visual search

• US 9,405,977 B2
• Filed: 05/28/2015
• Issued: 08/02/2016
• Est. Priority Date: 01/06/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Starner, Thad Eugene, Essa, Irfan
• Primary Examiner(s): Haley, Joseph
• Assistant Examiner(s): FRANK, EMILY J
• Application Number:
  US14/724,524
• Publication Number:
  US 20150286877A1
• Time in Patent Office: 432 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/033 Pointing devices displaced or positioned by the user , e.g. mice, trackballs, pens or joysticks; Accessories therefor digitisers characterised by the transducing means G06F3/041
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes

Abstract

Methods and devices for initiating a search are disclosed. In one embodiment, a method is disclosed that includes causing a camera on a wearable computing device to record video data, segmenting the video data into a number of layers and, based on the video data, detecting that a pointing object is in proximity to a first layer. The method further includes initiating a first search on the first layer. In another embodiment, a wearable computing device is disclosed that includes a camera configured to record video data, a processor, and data storage comprising instructions executable by the processor to segment the video data into a number of layers and, based on the video data, detect that a pointing object is in proximity to a first layer. The instructions are further executable by the processor to initiate a first search on the first layer.

Claims

1. A method comprising:

causing a camera on a wearable computing device to record video data;

segmenting the video data into a number of layers that comprise at least a first layer and a second layer;

based on the video data, detecting a pointing object in a field of view of the camera;

determining that the pointing object is in proximity to the first layer, wherein the first layer corresponds to a first object;

in response to determining that the pointing object is in proximity to the first layer, initiating a visual search on at least a portion of the first layer; and

providing search results from the visual search to facilitate display of information related to the first object on a display that is associated with the wearable computing device.

2. The method of claim 1, further comprising causing the display to display first information corresponding to the search results of the visual search.

3. The method of claim 2, further comprising:

based on the video data, detecting that the pointing object is no longer in proximity to the first layer and is in proximity to the second layer;

initiating a second visual search on the second layer; and

replacing, on the display, the first information with second information corresponding to results of the second search.

4. The method of claim 1, further comprising:

in further response to determining that the pointing object is in proximity to the first layer, highlighting the first layer.

5. The method of claim 4, further comprising:

based on the video data, detecting that the pointing object is no longer in proximity to the first layer and is in proximity to the second layer; and

responsively highlighting the second layer.

6. The method of claim 1, wherein each layer exhibits at least one of motion coherence and appearance coherence.

7. The method of claim 1, further comprising receiving an acceptance of the first layer.

8. The method of claim 7, wherein receiving the acceptance comprises receiving at least one of a tactile input, an audible input, and a visual input.

9. The method of claim 1, wherein the pointing device comprises a finger.

10. The method of claim 1, wherein the camera is configured to be offset from an eye of a user by an angle and a distance, and wherein the determining that the pointing object is in proximity to the first layer is based on the angle and the distance.

11. The method of claim 1, wherein segmenting the video data into the number of layers comprises:

segmenting the video data into a first set of layers in which the first layer comprises both the first object and a second object;

based on the video data, determining that the pointing object is behind the first object and in front of the second object; and

segmenting the video data into a second set of layers in which the first layer comprises only one of the first object and the second object.

12. A wearable computing device comprising:

a camera interface operable to receive video data from a camera;

a display interface operable to output graphic data to a display;

at least one processor; and

data storage comprising instructions executable by the at least one processor to;

receive the video data via the camera interface;

segment the video data into a plurality of layers that comprise at least a first layer and a second layer;

based on the video data, detecting a pointing object in a field of view of the camera;

determine that the pointing object is in proximity to the first layer, wherein the first layer corresponds to a first object;

in response to determining that the pointing object is in proximity to the first layer, initiate a visual search on at least a portion of the first layer;

generate graphic data comprising information corresponding to search results from the visual search, wherein the information is related to the first object; and

output, via the display interface, the graphic data to the display.

13. The wearable computing device of claim 12, further comprising a display configured to display an indication of the layers.

14. The wearable computing device of claim 13, wherein the display is further configured to display the video data.

15. The wearable computing device of claim 13, wherein the display is further configured to display first information corresponding to results of the first search.

16. The wearable computing device of claim 13, wherein the display is further configured to highlight the first layer.

17. The wearable computing device of claim 12, further comprising a user interface configured to receive an acceptance of the first layer.

18. The wearable computing device of claim 17, wherein the user interface comprises at least one of a touchpad and a microphone.

19. A computing device comprising:

a communication interface operable to communicate with a wearable computing device;

at least one processor; and

data storage comprising instructions executable by the at least one processor to;

receive video data that was generated at the wearable computing device and is indicative of a field of view corresponding to the wearable computing;

segment the video data into a plurality of layers that comprise at least a first layer and a second layer;

based on the video data, detecting a pointing object in a field of view of the camera;

determine that the pointing object is in proximity to the first layer, wherein the first layer corresponds to a first object;

in response to determining that the pointing object is in proximity to the first layer, perform a visual search on at least a portion of the first layer to determine search results related to the first object; and

send at least a portion of the search results related to the wearable computing device, so as to facilitate display of information related to the first object on a display that is associated with the wearable computing device.

20. The computing device of claim 19, wherein the segmentation of the video data into the number of layers comprises:

segmentation of the video data into a first set of layers in which the first layer comprises both the first object and a second object;

a determination, based on the video data, that the pointing object is behind the first object and in front of the second object; and

segmentation of the video data into a second set of layers in which the first layer comprises only one of the first object and the second object.