Visual gestures for a head mounted device

• US 9,652,047 B2
• Filed: 02/24/2016
• Issued: 05/16/2017
• Est. Priority Date: 02/25/2015
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: DAQRI, LLC
• Inventors: Crispin, Sterling, Kazansky, Lucas, Mullins, Brian, Kammerait, Matthew, Broaddus, Christopher
• Primary Examiner(s): OSORIO, RICARDO
• Application Number:
  US15/052,775
• Publication Number:
  US 20160246384A1
• Time in Patent Office: 447 Days
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 2203/011 Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34

Abstract

Visual gestures in a display device allow a user to select and activate features in a display of the display device. A sensor of the display device tracks an eye gaze of a user directed at a display of the display device. A visual gesture module identifies a predefined trigger zone in the display. A virtual object application displays a virtual object in the display based on the eye gaze of the user and the predefined trigger zone.

Claims

1. A head mounted display device comprising:

a transparent display operatively positioned to present virtual content to a user wearing the head mounted display device while allowing the user to view physical objects through the transparent display;

one or more eye gaze sensors configured to track at least a first pupil of the user to determine an eye gaze of the user, wherein the eye gaze of the user indicates a location on the transparent display in a line of sign of the user;

one or more computer processors; and

a memory storing instructions that, when executed by the one or computer processors, cause the head mounted display device to;

monitor the eye gaze of the user wearing the head mounted display device;

determine, based on monitoring the eye gaze of the user, that a duration of time that the line of sight of the user is within a predefined trigger zone of the transparent display exceeds a threshold duration for the predefined trigger zone, wherein the predefined trigger zone corresponds to a first physical object visible to the user through the predefined trigger zone of the head mounted display; and

in response to determining that the duration of time that the line of sight of the user is within the predefined trigger zone of the transparent display exceeds the threshold duration, cause a display of a virtual object in the predefined trigger zone of the transparent display.

2. The head mounted display device of claim 1, wherein the one or more eye gaze sensors are configured to identify a direction and a duration of the eye gaze of the user.

3. The head mounted display device of claim 2, wherein the instructions further cause the head mounted display device to change a state of the virtual object based on a change in the direction and duration of the eye gaze of the user.

4. The head mounted display device of claim 2, wherein the predefined trigger zone includes a plurality of gradient zones, each gradient zones associated with a corresponding predefined trigger duration.

5. The head mounted display device of claim 2, wherein the instructions further cause the head mounted display device to:

identify a spatial and temporal pattern of the eye gaze of the user;

identify a predefined spatial and temporal pattern; and

display the virtual object in the display based on the spatial and temporal pattern of the eye gaze of the user matching the predefined spatial and temporal pattern.

6. The head mounted display device of claim 2, wherein the instructions further cause the head mounted display device to:

identify a focal depth of the eye gaze of the user;

identify a predefined focal depth range; and

display the virtual object in the transparent display based on the focal depth of the eye gaze of the user being with the predefined focal depth range.

7. The head mounted display device of claim 2, further comprising:

a voltage sensor connected to a skin of the user, the voltage sensor configured to identify a physical and mental state of the user, wherein the instructions further cause the head mounted display device to;

identify a predefined physical and mental state corresponding to a second virtual object; and

display the second virtual object in the transparent display based on a combination of the physical and mental state of the user corresponding to the predefined physical and mental state and the eye gaze of the user intersecting the predefined trigger zone.

8. The head mounted display device of claim 2, wherein the instructions further cause the head mounted display device to:

identify that the eye gaze of the user is within a predefined distance threshold of the predefined trigger zone; and

display a visual indicator being pulled towards the predefined trigger zone.

9. The head mounted display device of claim 2, wherein the instructions further cause the head mounted display device to:

identify a predefined trigger point associated with a body part of the user; and

display a second virtual object in the transparent display based on a first eye gaze of the user directed at the body part of the user followed by a second eye gaze of the user directed at the predefined trigger zone associated with the body part.

10. A computer-implemented method comprising:

monitoring, by a head mounted display device, an eye gaze of a user wearing the head mounted display device, the head mounted display device including a transparent display operatively positioned to present virtual content to the user while allowing the user to view physical objects through the transparent display, the head mounted display including one or more eye gaze sensors configured to track at least a first pupil of the user to determine the eye gaze of the user, wherein the eye gaze of the user indicates a location on the transparent display in a line of sight of the user;

determining, based on monitoring the eye gaze of the user, that a duration of time that the line of sight of the user is within a predefined trigger zone of the transparent display exceeds a threshold duration for the predefined trigger zone, wherein the predefined trigger zone corresponds to a first physical object visible to the user through the predefined trigger zone of the head mounted display; and

in response to determining that the duration of time that the line of sight of the user is within the predefined trigger zone of the transparent display exceeds the threshold duration, causing a display of a virtual object in the predefined trigger zone of the transparent display.

11. The computer-implemented method of claim 10, further comprising:

identifying a direction and a duration of the eye gaze of the user.

12. The computer-implemented method of claim 11, further comprising:

changing a state of the virtual object based on a change in the direction and duration of the eye gaze of the user.

13. The computer-implemented method of claim 11, wherein the predefined trigger zone includes a plurality of gradient zones, each gradient zones associated with a corresponding predefined trigger duration.

14. The computer-implemented method of claim 11, further comprising:

identifying a spatial and temporal pattern of the eye gaze of the user;

identifying a predefined spatial and temporal pattern; and

displaying the virtual object in the display based on the spatial and temporal pattern of the eye gaze of the user matching the predefined spatial and temporal pattern.

15. The computer-implemented method of claim 11, further comprising:

identifying a focal depth of the eye gaze of the user;

identifying a predefined focal depth range; and

displaying the virtual object in the transparent display based on the focal depth of the eye gaze of the user being with the predefined focal depth range.

16. The computer-implemented method of claim 11, further comprising:

identifying a physical and mental state of the user using a voltage sensor connected to a skin of the user;

identifying a predefined physical and mental state corresponding to a second virtual object; and

displaying the second virtual object in the transparent display based on a combination of the physical and mental state of the user corresponding to the predefined physical and mental state and the eye gaze of the user intersecting the predefined trigger zone.

17. The computer-implemented method of claim 11, further comprising:

identifying a predefined trigger point associated with a body part of the user; and

displaying a second virtual object in the transparent display based on a first eye gaze of the user directed at the body part of the user followed by a second eye gaze of the user directed at the predefined trigger zone associated with the body part.

18. A non-transitory machine-readable medium comprising instructions that, when executed by one or more processors of a head mounted display device, cause the head mounted display device to perform operations comprising:

monitoring an eye gaze of a user wearing the head mounted display device, the head mounted display device including a transparent display operatively positioned to present virtual content to the user while allowing the user to view physical objects through the transparent display, the head mounted display including one or more eye gaze sensors configured to track at least a first pupil of the user to determine the eye gaze of the user, wherein the eye gaze of the user indicates a location on the transparent display in a line of sight of the user;

determining, based on monitoring the eye gaze of the user, that a duration of time that the line of sight of the user is within a predefined trigger zone of the transparent display exceeds a threshold duration for the predefined trigger zone, wherein the predefined trigger zone corresponds to a first physical object visible to the user through the predefined trigger zone of the head mounted display; and

in response to determining that the duration of time that the line of sight of the user is within the predefined trigger zone of the transparent display exceeds the threshold duration, causing a display of a virtual object in the predefined trigger zone of the transparent display.