VISUAL GESTURES FOR A HEAD MOUNTED DEVICE

• US 20160246384A1
• Filed: 02/24/2016
• Published: 08/25/2016
• Est. Priority Date: 02/25/2015
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: Brian Mullins, Matthew Kammerait
• Inventors: KAMMERAIT, MATTHEW, BROADDUS, CHRISTOPHER, CRISPIN, STERLING, KAZANSKY, LUCAS, MULLINS, BRIAN
• Granted Patent: US 9,652,047 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 2203/011 Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34

Abstract

Visual gestures in a display device allow a user to select and activate features in a display of the display device. A sensor of the display device tracks an eye gaze of a user directed at a display of the display device. A visual gesture module identifies a predefined trigger zone in the display. A virtual object application displays a virtual object in the display based on the eye gaze of the user and the predefined trigger zone.

Claims

1. A display device comprising:

a display;

a sensor configured to identify an eye gaze of a user directed at the display;

a visual gesture module configured to identify a predefined trigger zone in the display; and

a virtual object application, implemented by a processor of the display device, configured to display a virtual object in the display based on the eye gaze of the user and the predefined trigger zone.

2. The display device of claim 1, wherein the sensor is configured to identify a direction and a duration of the eye gaze of the user, wherein the display includes a transparent display.

3. The display device of claim 2, wherein the virtual object application is configured to change a state of the virtual object based on a change in the direction and duration of the eye gaze of the user.

4. The display device of claim 2, wherein the virtual object application is configured to change a state of the virtual object in response to determining that a line of sight of the eye gaze intersects the predefined trigger zone in the transparent display and that the duration of the eye gaze is greater than a predefined trigger duration.

5. The display device of claim 2, wherein the predefined trigger zone includes a plurality of gradient zones, each gradient zones associated with a corresponding predefined trigger duration.

6. The display device of claim 2, wherein the sensor is configured to identify a spatial and temporal pattern of the eye gaze of the user, wherein the visual gesture module is configured to identify a predefined spatial and temporal pattern, wherein the virtual object application is configured to display the virtual object in the display based on the spatial and temporal pattern of the eye gaze of the user matching the predefined spatial and temporal pattern.

7. The display device of claim 2, wherein the sensor is configured to identify a focal depth of the eye gaze of the user, wherein the visual gesture module is configured to identify a predefined focal depth range, wherein the virtual object application is configured to display the virtual object in the transparent display based on the focal depth of the eye gaze of the user being with the predefined focal depth range.

8. The display device of claim 2, further comprising:

a voltage sensor connected to a skin of the user, the voltage sensor configured to identify a physical and mental state of the user,

wherein the visual gesture module is configured to identify a predefined physical and mental state corresponding to a second virtual object, and

wherein the virtual object application is configured to display the second virtual object in the transparent display based on a combination of the physical and mental state of the user corresponding to the predefined physical and mental state and the eye gaze of the user intersecting the predefined trigger zone.

9. The display device of claim 2, wherein the virtual object application is configured to identify that the eye gaze of the user is within a predefined distance threshold of the predefined trigger zone, and to display a visual indicator being pulled towards the predefined trigger zone.

10. The display device of claim 2, wherein the visual gesture module configured to identify a predefined trigger points associated with a body part of the user, wherein the virtual object application is configured to display a second virtual object in the transparent display based on a first eye gaze of the user directed at the body part of the user followed by a second eye gaze of the user directed at the predefined trigger zone.

11. A computer-implemented method comprising:

identifying, with a sensor of a display device, an eye gaze of a user directed at a display of the display device;

identifying a predefined trigger zone in the display; and

causing a display of a virtual object in the display based on the eye gaze of the user and the predefined trigger zone.

12. The computer-implemented method of claim 11, further comprising:

identifying a direction and a duration of the eye gaze of the user, wherein the display includes a transparent display.

13. The computer-implemented method of claim 12, further comprising:

changing a state of the virtual object based on a change in the direction and duration of the eye gaze of the user.

14. The computer-implemented method of claim 12, further comprising:

determining that a line of sight of the eye gaze intersects the predefined trigger zone in the transparent display and that the duration of the eye gaze is greater than a predefined trigger duration; and

changing a state of the virtual object in response to determining that the line of sight of the eye gaze intersects the predefined trigger zone in the transparent display and that the duration of the eye gaze is greater than the predefined trigger duration.

15. The computer-implemented method of claim 12, wherein the predefined trigger zone includes a plurality of gradient zones, each gradient zones associated with a corresponding predefined trigger duration.

16. The computer-implemented method of claim 12, further comprising:

identifying a spatial and temporal pattern of the eye gaze of the user;

identifying a predefined spatial and temporal pattern; and

displaying the virtual object in the display based on the spatial and temporal pattern of the eye gaze of the user matching the predefined spatial and temporal pattern.

17. The computer-implemented method of claim 12, further comprising:

identifying a focal depth of the eye gaze of the user;

identifying a predefined focal depth range; and

displaying the virtual object in the transparent display based on the focal depth of the eye gaze of the user being with the predefined focal depth range.

18. The computer-implemented method of claim 12, further comprising:

identifying a physical and mental state of the user using a voltage sensor connected to a skin of the user;

identifying a predefined physical and mental state corresponding to a second virtual object; and

displaying the second virtual object in the transparent display based on a combination of the physical and mental state of the user corresponding to the predefined physical and mental state and the eye gaze of the user intersecting the predefined trigger zone.

19. The computer-implemented method of claim 12, further comprising:

identifying a predefined trigger points associated with a body part of the user; and

displaying a second virtual object in the transparent display based on a first eye gaze of the user directed at the body part of the user followed by a second eye gaze of the user directed at the predefined trigger zone.

20. A non-transitory machine-readable medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

identifying, with a sensor of a display device, an eye gaze of a user directed at a display of the display device;

identifying a predefined trigger zone in the display; and

causing a display of a virtual object in the display based on the eye gaze of the user and the predefined trigger zone.