Hand gestures to signify what is important

• US 9,024,842 B1
• Filed: 09/04/2013
• Issued: 05/05/2015
• Est. Priority Date: 07/08/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Prada Gomez, Luis Ricardo, Wheeler, Aaron
• Primary Examiner(s): Cheng, Joe H
• Application Number:
  US14/017,573
• Time in Patent Office: 608 Days
• Field of Search: 345/7, 345/8, 345/156, 345/173, 345/418, 345/619, 345/620, 345/660, 345/661, 349/11, 349/13, 705/14.58, 351/158, 700/258, 700/259, 359/630, 715/716, 715/764, 715/848
• US Class Current: 345/7
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

In accordance with example embodiments, hand gestures can be used to provide user input to a wearable computing device, and in particular to identify, signify, or otherwise indicate what may be considered or classified as important or worthy of attention or notice. A wearable computing device, which could include a head-mounted display (HMD) and a video camera, may recognize known hand gestures and carry out particular actions in response. Particular hand gestures could be used for selecting portions of a field of view of the HMD, and generating images from the selected portions. The HMD could then transmit the generated images to one or more applications in a network server communicatively connected with the HMD, including a server or server system hosting a social networking service.

Claims

1. In a wearable head-mounted display (HMD), a computer-implemented method comprising:

making a determination that video data from a video camera of the wearable HMD includes a hand gesture of a thumb and forefinger of a particular hand forming a shape of a symbolic right angle in a field of view (FOV) of the wearable HMD, the symbolic right angle having a first leg with a first length corresponding to a length of the thumb projected in the FOV and a second, perpendicular leg with a second length corresponding to a length of the forefinger projected in the FOV;

determining that the particular hand is associated with the wearable HMD via analysis of the video data including the hand gesture;

determining a closed area in the FOV bounded by a rectangle having a corner at a vertex of the symbolic right angle in the FOV, and having two perpendicular sides coinciding with, and having lengths respectively equal to, the first and second legs of the symbolic right angle;

generating an image from the video data of a sub-FOV within the closed area;

matching the shape of the symbolic right angle with a predetermined action associated with the image; and

executing the predetermined action on a processor of the wearable HMD.

2. The method of claim 1, wherein determining that the particular hand is associated with the wearable HMD comprises:

detecting a particular marking on the particular hand; and

matching the particular marking with a predetermined identifier associated with the wearable HMD.

3. The method of claim 1, wherein determining that the particular hand is associated with the wearable HMD via analysis of the video data including the hand gesture comprises determining that a particular hand gesture made by the particular hand matches a predetermined identifying hand gesture associated with the wearable HMD.

4. The method of claim 1, wherein generating the image from the video data of the sub-FOV within the closed area comprises determining a set of image pixels contained within the closed area.

5. The method of claim 1, wherein executing the predetermined action comprises storing the image.

6. The method of claim 1, wherein executing the predetermined action comprises:

generating a message containing the image; and

transmitting the message to a network server in a network communicatively coupled with the wearable HMD.

7. The method of claim 6, wherein generating the message comprises including in the message information associating an identified subscriber account of a social networking service with a request to post the image on a web page associated with the identified subscriber account,

and wherein transmitting the message to the network server comprises transmitting the message to the social network service.

8. The method of claim 7, further comprising:

responsive to transmitting the message to the social networking service, receiving a response message from the social networking service containing information indicative of a result of a network-based information search carried out based on the image; and

displaying an aspect of the contained information.

9. The method of claim 7, further comprising transmitting a log-in message to the social networking service, the log-in message including a request to log in to the identified subscriber account.

10. The method of claim 1, wherein executing the predetermined action comprises:

generating a message containing the image and a request for a network-based information search based on the image; and

transmitting the message to a network server in a network communicatively coupled with the wearable HMD.

11. The method of claim 10, further comprising:

receiving a response message from the network server containing information indicative of a search result; and

displaying an aspect of the contained information.

12. A wearable head-mounted display (HMD) comprising:

a display device;

a video camera;

one or more processors;

memory; and

machine-readable instructions stored in the memory that, when executed by the one or more processors, cause the wearable HMD to carry out functions including;

making a determination that video data from the video camera includes a hand gesture of a thumb and forefinger of a particular hand forming a shape of a symbolic right angle in a field of view (FOV) of the wearable HMD, the symbolic right angle having a first leg with a first length corresponding to a length of the thumb projected in the FOV and a second, perpendicular leg with a second length corresponding to a length of the forefinger projected in the FOV,

determining that the particular hand is associated with the wearable HMD via analysis of the video data including the hand gesture,

determining a closed area in the FOV bounded by a rectangle having a corner at a vertex of the symbolic right angle in the FOV, and having two perpendicular sides coinciding with, and having lengths respectively equal to, the first and second legs of the symbolic right angle,

generating an image from the video data of a sub-FOV within the closed area,

matching the shape of the symbolic right angle with a predetermined action associated with the image,

and executing the predetermined action on a processor of the wearable HMD.

13. A nontransitory computer-readable medium having instructions stored thereon that, upon execution by one or more processors of a wearable head-mounted display (HMD), cause the wearable HMD to carry out functions comprising:

making a determination that video data from a video camera of the wearable HMD includes a hand gesture of a thumb and forefinger of a particular hand forming a shape of a symbolic right angle in a field of view (FOV) of the wearable HMD, the symbolic right angle having a first leg with a first length corresponding to a length of the thumb projected in the FOV and a second, perpendicular leg with a second length corresponding to a length of the forefinger projected in the FOV;

determining that the particular hand is associated with the wearable HMD via analysis of the video data including the hand gesture;

determining a closed area in the FOV bounded by a rectangle having a corner at a vertex of the symbolic right angle in the FOV, and having two perpendicular sides coinciding with, and having lengths respectively equal to, the first and second legs of the symbolic right angle;

generating an image from the video data of a sub-FOV within the closed area;

matching the shape of the symbolic right angle with a predetermined action associated with the image; and

executing the predetermined action on a processor of the wearable HMD.

14. In a wearable head-mounted display (HMD), a computer-implemented method comprising:

making a determination that video data from a video camera of the wearable HMD includes a hand gesture of a forefinger of a particular hand pointing to an object in a field of view (FOV) of the wearable HMD;

determining that the particular hand is associated with the wearable HMD via analysis of the video data including the hand gesture;

highlighting the object to associate the object with a high qualitative ranking;

responsive to both making the determination and highlighting the object, identifying a predetermined action associated with the high qualitative ranking; and

executing the predetermined action on a processor of the wearable HMD.

15. The method of claim 14, wherein executing the predetermined action comprises:

generating an image of the object in the FOV; and

storing the generated image in a data structure together with information associating the image with the high qualitative ranking.

16. A wearable head-mounted display (HMD) comprising:

a display device;

a video camera;

one or more processors;

memory; and

machine-readable instructions stored in the memory that, when executed by the one or more processors, cause the wearable HMD to carry out functions including;

making a determination that video data from the video camera includes a hand gesture of a forefinger of a particular hand pointing to an object in a field of view (FOV) of the wearable HMD,

determining that the particular hand is associated with the wearable HMD via analysis of the video data including the hand gesture,

highlighting the object to associate the object with a high qualitative ranking,

responsive to both making the determination and highlighting the object, identifying a predetermined action associated with the high qualitative ranking,

and executing the predetermined action on a processor of the wearable HMD.

17. A nontransitory computer-readable medium having instructions stored thereon that, upon execution by one or more processors of a wearable head-mounted display (HMD), cause the wearable HMD to carry out functions comprising:

making a determination that video data from a video camera of the wearable HMD includes a hand gesture of a forefinger of a particular hand pointing to an object in a field of view (FOV) of the wearable HMD;

determining that the particular hand is associated with the wearable HMD via analysis of the video data including the hand gesture;

highlighting the object to associate the object with a high qualitative ranking;

responsive to both making the determination and highlighting the object, identifying a predetermined action associated with the high qualitative ranking; and

executing the predetermined action on a processor of the wearable HMD.

18. In a wearable head-mounted display (HMD), a computer-implemented method comprising:

making a determination that video data from a video camera of the wearable HMD includes a hand gesture of a forefinger of a particular hand pointing to text in a field of view (FOV) of the wearable HMD;

determining that the particular hand is associated with the wearable HMD via analysis of the video data including the hand gesture;

underlining the text;

responsive to both making the determination and underlining the text, identifying a predetermined action associated with the text; and

executing the predetermined action on a processor of the wearable HMD.

19. The method of claim 18, wherein executing the predetermined action comprises:

storing the text.

20. A wearable head-mounted display (HMD) comprising:

a display device;

a video camera;

one or more processors;

memory; and

machine-readable instructions stored in the memory that, when executed by the one or more processors, cause the wearable HMD to carry out functions including;

making a determination that video data from the video camera includes a hand gesture of a forefinger of a particular hand pointing to text in a field of view (FOV) of the wearable HMD,

determining that the particular hand is associated with the wearable HMD via analysis of the video data including the hand gesture,

underlining the text,

responsive to both making the determination and underlining the text, identifying a predetermined action associated with the text,

and executing the predetermined action on a processor of the wearable HMD.

21. A nontransitory computer-readable medium having instructions stored thereon that, upon execution by one or more processors of a wearable head-mounted display (HMD), cause the wearable HMD to carry out functions comprising:

making a determination that video data from a video camera of the wearable HMD includes a hand gesture of a forefinger of a particular hand pointing to text in a field of view (FOV) of the wearable HMD;

determining that the particular hand is associated with the wearable HMD via analysis of the video data including the hand gesture;

underlining the text;

responsive to both making the determination and underlining the text, identifying a predetermined action associated with the text; and

executing the predetermined action on a processor of the wearable HMD.

22. In a wearable head-mounted display (HMD), a computer-implemented method comprising:

capturing video data from a video camera of the wearable HMD of a particular hand gesture of at least one particular hand in a field of view (FOV) of the wearable HMD;

determining that the at least one particular hand is associated with the wearable HMD by determining that the wearable HMD is operating in a recording mode;

analyzing the video data to generate an analytical description of the particular hand gesture; and

recording the generated analytical description in storage memory of the wearable HMD, together with an identifier for each of one or more associated particular actions.

23. The method of claim 22, further comprising:

subsequent to recording the generated analytical description in the storage memory of the wearable HMD, together with the identifier for each of the one or more associated particular actions, making a determination that video data from the video camera of the wearable HMD includes at least one hand making a recognized hand gesture in the FOV of the wearable HMD that matches the generated analytical description;

based on the match of the recognized hand gesture and the generated analytical description, determining each of the one or more associated particular actions; and

executing each of the one or more associated particular actions on a processor of the wearable HMD.

24. The method of claim 23, wherein the particular hand gesture is selected from a group consisting of:

(i) two hands in a shape of a symbolic heart in projection, (ii) a thumb and forefinger of a hand forming a shape of a symbolic right angle moving from a first projected location to a second projected location, and (iii) a forefinger of a hand tracing a closed-loop shape in projection,

and wherein one of the one or more associated particular actions comprises generating an image from video data of a sub-FOV within a closed area defined by the recognized hand gesture.

25. A wearable head-mounted display (HMD) comprising:

a display device;

a video camera;

one or more processors;

memory; and

machine-readable instructions stored in the memory that, when executed by the one or more processors, cause the wearable HMD to carry out functions including;

capturing video data from the video camera of a particular hand gesture of at least one particular hand in a field of view (FOV) of the wearable HMD,

determining that the at least one particular hand is associated with the wearable HMD by determining that the wearable HMD is operating in a recording mode,

analyzing the video data to generate an analytical description of the particular hand gesture,

and recording the generated analytical description in storage memory of the wearable HMD, together with an identifier for each of one or more associated particular actions.

26. A nontransitory computer-readable medium having instructions stored thereon that, upon execution by one or more processors of a wearable head-mounted display (HMD), cause the wearable HMD to carry out functions comprising:

capturing video data from a video camera of the wearable HMD of a particular hand gesture of at least one particular hand in a field of view (FOV) of the wearable HMD;

determining that the at least one particular hand is associated with the wearable HMD by determining that the wearable HMD is operating in a recording mode;

analyzing the video data to generate an analytical description of the particular hand gesture; and

recording the generated analytical description in storage memory of the wearable HMD, together with an identifier for each of one or more associated particular actions.