Hand gestures to signify what is important

• US 8,558,759 B1
• Filed: 07/08/2011
• Issued: 10/15/2013
• Est. Priority Date: 07/08/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Prada Gomez, Luis Ricardo, Wheeler, Aaron
• Primary Examiner(s): Cheng, Joe H
• Application Number:
  US13/179,190
• Time in Patent Office: 830 Days
• Field of Search: 345/7, 345/8, 345/156, 345/173, 345/418, 345/619, 345/620, 345/660, 345/661, 349/11, 349/13, 705/14.58, 359/630, 715/716, 715/764, 715/848, 351/158, 700/258, 700/259
• US Class Current: 345/7
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

In accordance with example embodiments, hand gestures can be used to provide user input to a wearable computing device, and in particular to identify, signify, or otherwise indicate what may be considered or classified as important or worthy of attention or notice. A wearable computing device, which could include a head-mounted display (HMD) and a video camera, may recognize known hand gestures and carry out particular actions in response. Particular hand gestures could be used for selecting portions of a field of view of the HMD, and generating images from the selected portions. The HMD could then transmit the generated images to one or more applications in a network server communicatively connected with the HMD, including a server or server system hosting a social networking service.

Claims

1. In a wearable head-mounted display (HMD), a computer-implemented method comprising:

making a determination that video data from a video camera of the wearable HMD includes a hand gesture forming an area bounded by two hands in a shape of a symbolic heart;

determining that the two hands are associated with the wearable HMD via analysis of the video data including the hand gesture;

generating an image from the video data, wherein the image includes the area bounded by the two hands;

matching the shape of the symbolic heart with a predetermined action associated with the image; and

executing the predetermined action on a processor of the wearable HMD.

2. The method of claim 1, wherein determining that the two hands are associated with the wearable HMD comprises:

detecting a particular marking on at least one of the two hands; and

matching the particular marking with a predetermined identifier associated with the wearable HMD.

3. The method of claim 1, wherein determining that the two hands are associated with the wearable HMD via analysis of the video data including the hand gesture comprises determining that a particular hand gesture made by at least one of the two hands matches a predetermined identifying hand gesture associated with the wearable HMD.

4. The method of claim 1, wherein generating the image from the video data comprises determining a set of image pixels contained within the area bounded by the two hands.

5. The method of claim 1, wherein executing the predetermined action comprises:

determining a qualitative ranking for the image based on the determination; and

storing a data structure containing information associating the image with the qualitative ranking.

6. The method of claim 1, wherein executing the predetermined action comprises:

generating a message containing the image and information indicative of a qualitative ranking of the image; and

transmitting the message to a network server in a network communicatively coupled with the wearable HMD.

7. The method of claim 6, wherein generating the message comprises including in the message information associating an identified subscriber account of a social networking service with an indication of the image being liked,

and wherein transmitting the message to the network server comprises transmitting the message to the social networking service.

8. The method of claim 7, further comprising:

responsive to transmitting the message to the social networking service, receiving a response message from the social networking service containing information indicative of a result of a network-based information search carried out based on the image and on the indication of the image being liked; and

displaying an aspect of the contained information.

9. The method of claim 7, further comprising transmitting a log-in message to the social networking service, the log-in message including a request to log in to the identified subscriber account.

10. The method of claim 1, wherein executing the predetermined action comprises:

generating a message containing the image and a request for a network-based information search based on the image; and

transmitting the message to a network server in a network communicatively coupled with the wearable HMD.

11. The method of claim 10, further comprising:

receiving a response message from the network server containing information indicative of a search result; and

displaying an aspect of the contained information.

12. In a wearable head-mounted display (HMD), a computer-implemented method comprising:

making a determination that video data from a video camera of the wearable HMD includes a hand gesture of a thumb and forefinger of a particular hand forming a shape of a symbolic right angle moving from a first location to a second location in a field of view (FOV) of the wearable HMD;

determining that the particular hand is associated with the wearable HMD via analysis of the video data including the hand gesture;

determining a closed area in the FOV bounded by a rectangle having a first corner at a vertex of the symbolic right angle at the first location and a diagonally-opposite second corner at a vertex of the symbolic right angle at the second location;

generating an image from the video data of a sub-FOV within the closed area;

matching the shape of the symbolic right angle with a predetermined action associated with the image; and

executing the predetermined action on a processor of the wearable HMD.

13. The method of claim 12, wherein determining that the particular hand is associated with the wearable HMD comprises:

detecting a particular marking on the particular hand; and

matching the particular marking with a predetermined identifier associated with the wearable HMD.

14. The method of claim 12, wherein determining that the particular hand is associated with the wearable HMD via analysis of the video data including the hand gesture comprises determining that a particular hand gesture made by the particular hand matches a predetermined identifying hand gesture associated with the wearable HMD.

15. The method of claim 12, wherein generating the image from the video data of the sub-FOV within the closed area comprises determining a set of image pixels contained within the closed area.

16. The method of claim 12, wherein executing the predetermined action comprises storing the image.

17. The method of claim 12, wherein executing the predetermined action comprises:

generating a message containing the image; and

transmitting the message to a network server in a network communicatively coupled with the wearable HMD.

18. The method of claim 17, wherein generating the message comprises including in the message information associating an identified subscriber account of a social networking service with a request to post the image on a web page associated with the identified subscriber account,

and wherein transmitting the message to the network server comprises transmitting the message to the social network service.

19. The method of claim 18, further comprising:

responsive to transmitting the message to the social networking service, receiving a response message from the social networking service containing information indicative of a result of a network-based information search carried out based on the image; and

displaying an aspect of the contained information.

20. The method of claim 18, further comprising transmitting a log-in message to the social networking service, the log-in message including a request to log in to the identified subscriber account.

21. The method of claim 12, wherein executing the predetermined action comprises:

generating a message containing the image and a request for a network-based information search based on the image; and

transmitting the message to a network server in a network communicatively coupled with the wearable HMD.

22. The method of claim 21, further comprising:

receiving a response message from the network server containing information indicative of a search result; and

displaying an aspect of the contained information.

23. In a wearable head-mounted display (HMD), a computer-implemented method comprising:

making a determination that video data from a video camera of the wearable HMD includes a hand gesture of a forefinger of a particular hand tracing a closed-loop shape in a field of view (FOV) of the wearable HMD;

determining that the particular hand is associated with the wearable HMD via analysis of the video data including the hand gesture;

determining a closed area in the FOV bounded by the closed-loop shape;

generating an image from the video data of a sub-FOV within the closed area;

responsive to both making the determination and generating the image, identifying a predetermined action associated with the image; and

executing the predetermined action on a processor of the wearable HMD.

24. The method of claim 23, wherein determining that the particular hand is associated with the wearable HMD comprises:

detecting a particular marking on the particular hand; and

matching the particular marking with a predetermined identifier associated with the wearable HMD.

25. The method of claim 23, wherein determining that the particular hand is associated with the wearable HMD via analysis of the video data including the hand gesture comprises determining that a particular hand gesture made by the particular hand matches a predetermined identifying hand gesture associated with the wearable HMD.

26. The method of claim 23, wherein generating the image from the video data of the sub-FOV within the closed area comprises determining a set of image pixels contained within the closed area.

27. The method of claim 23, wherein executing the predetermined action comprises storing the image.

28. The method of claim 23, wherein executing the predetermined action comprises:

generating a message containing the image; and

transmitting the message to a network server in a network communicatively coupled with the wearable HMD.

29. The method of claim 28, wherein generating the message comprises including in the message information associating an identified subscriber account of a social networking service with a request to post the image on a web page associated with the identified subscriber account,

and wherein transmitting the message to the network server comprises transmitting the message to the social networking service.

30. The method of claim 29, further comprising:

responsive to transmitting the message to the social networking service, receiving a response message from the social networking service containing information indicative of a result of a network-based information search carried out based on the image; and

displaying an aspect of the contained information.

31. The method of claim 29, further comprising transmitting a log-in message to the social networking service, the log-in message including a request to log in to the identified subscriber account.

32. The method of claim 23, wherein executing the predetermined action comprises:

generating a message containing the image and a request for a network-based information search based on the image; and

transmitting the message to a network server in a network communicatively coupled with the wearable HMD.

33. The method of claim 32, further comprising:

receiving a response message from the network server containing information indicative of a search result; and

displaying an aspect of the contained information.

34. A wearable head-mounted display (HMD) comprising:

a display device;

a video camera;

one or more processors;

memory; and

machine-readable instructions stored in the memory that, when executed by the one or more processors, cause the wearable HMD to carry out functions including;

making a determination that video data from the video camera includes a hand gesture forming an area bounded by two hands in a shape of a symbolic heart,

determining that the two hands are associated with the wearable HMD via analysis of the video data including the hand gesture,

generating an image from the video data, wherein the image includes the area bounded by the two hands,

matching the shape of the symbolic heart with a predetermined action associated with the image,

and executing the predetermined action on a processor of the wearable HMD.

35. A nontransitory computer-readable medium having instructions stored thereon that, upon execution by one or more processors of a wearable head-mounted display (HMD), cause the wearable HMD to carry out functions comprising:

making a determination that video data from a video camera of the wearable HMD includes a hand gesture forming an area bounded by two hands in a shape of a symbolic heart;

determining that the two hands are associated with the wearable HMD via analysis of the video data including the hand gesture;

generating an image from the video data, wherein the image includes the area bounded by the two hands;

matching the shape of the symbolic heart with a predetermined action associated with the image; and

executing the predetermined action on a processor of the wearable HMD.

36. A wearable head-mounted display (HMD) comprising:

a display device;

a video camera;

one or more processors;

memory; and

machine-readable instructions stored in the memory that, when executed by the one or more processors, cause the wearable HMD to carry out functions including;

making a determination that video data from the video camera includes a hand gesture of a thumb and forefinger of a particular hand forming a shape of a symbolic right angle moving from a first location to a second location in a field of view (FOV) of the wearable HMD,

determining that the particular hand is associated with the wearable HMD via analysis of the video data including the hand gesture,

determining a closed area in the FOV bounded by a rectangle having a first corner at a vertex of the symbolic right angle at the first location and a diagonally-opposite second corner at a vertex of the symbolic right angle at the second location,

generating an image from the video data of a sub-FOV within the closed area,

matching the shape of the symbolic right angle with a predetermined action associated with the image,

and executing the predetermined action on a processor of the wearable HMD.

37. A nontransitory computer-readable medium having instructions stored thereon that, upon execution by one or more processors of a wearable head-mounted display (HMD), cause the wearable HMD to carry out functions comprising:

making a determination that video data from a video camera of the wearable HMD includes a hand gesture of a thumb and forefinger of a particular hand forming a shape of a symbolic right angle moving from a first location to a second location in a field of view (FOV) of the wearable HMD;

determining that the particular hand is associated with the wearable HMD via analysis of the video data including the hand gesture;

determining a closed area in the FOV bounded by a rectangle having a first corner at a vertex of the symbolic right angle at the first location and a diagonally-opposite second corner at a vertex of the symbolic right angle at the second location;

generating an image from the video data of a sub-FOV within the closed area;

matching the shape of the symbolic right angle with a predetermined action associated with the image; and

executing the predetermined action on a processor of the wearable HMD.

38. A wearable head-mounted display (HMD) comprising:

a display device;

a video camera;

one or more processors;

memory; and

machine-readable instructions stored in the memory that, when executed by the one or more processors, cause the wearable HMD to carry out functions including;

making a determination that video data from the video camera includes a hand gesture of a forefinger of a particular hand tracing a closed-loop shape in a field of view (FOV) of the wearable HMD,

determining that the particular hand is associated with the wearable HMD via analysis of the video data including the hand gesture,

determining a closed area in the FOV bounded by the closed-loop shape,

generating an image from the video data of a sub-FOV within the closed area,

responsive to both making the determination and generating the image, identifying a predetermined action associated with the image,

and executing the predetermined action on a processor of the wearable HMD.

39. A nontransitory computer-readable medium having instructions stored thereon that, upon execution by one or more processors of a wearable head-mounted display (HMD), cause the wearable HMD to carry out functions comprising:

making a determination that video data from a video camera of the wearable HMD includes a hand gesture of a forefinger of a particular hand tracing a closed-loop shape in a field of view (FOV) of the wearable HMD;

determining that the particular hand is associated with the wearable HMD via analysis of the video data including the hand gesture;

determining a closed area in the FOV bounded by the closed-loop shape;

generating an image from the video data of a sub-FOV within the closed area;

responsive to both making the determination and generating the image, identifying a predetermined action associated with the image; and

executing the predetermined action on a processor of the wearable HMD.