Controlling a near eye display

• US 10,019,843 B2
• Filed: 08/07/2014
• Issued: 07/10/2018
• Est. Priority Date: 08/08/2013
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Grossinger, Nadav, Alpem, Yair
• Primary Examiner(s): Wang, Yuehan
• Application Number:
  US14/453,733
• Publication Number:
  US 20150042680A1
• Time in Patent Office: 1,433 Days
• Field of Search: 345633
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A method and a system for controlling a near eye display using a virtual navigation space are provided herein. The system may include: a wearable near eye display; a sensor having a field of view, attached to the wearable near eye display and configured to capture a scene; a transmitter attached to the wearable near eye display said transmitter is configured to transmit a structured light pattern onto a navigation space, wherein the sensor is configured to capture reflections of the specified pattern coming from the navigation space; and a computer processor configured to analyze said reflections and control a visual indicator presented to a user over the wearable near eye display. The method implements the aforementioned logic without being limited to the architecture.

Claims

1. An apparatus comprising:

a wearable near eye display;

a sensor attached to the wearable near eye display and configured to capture a scene;

a transmitter attached to the wearable near eye display, wherein the transmitter is configured to transmit a structured light pattern onto a navigation space and onto a hand of a user in the navigation space, and wherein the sensor is further configured to capture reflections of the structured light pattern coming from the navigation space; and

a computer processor configured to

analyze the reflections of the structured light pattern coming from the navigation space;

identify a gesture of the hand from a set of predefined gestures based on the reflections analysis, each gesture in the set of predefined gestures comprising a rotation or a posture change of the hand in the navigation space;

control a cursor presented to the user over the wearable near eye display according to the identified gesture of the hand, the cursor being a graphical pointing indication on the wearable near eye display interacting with content presented over the wearable near eye display;

invoke a corresponding action associated with the cursor over the wearable near eye display based on the identified gesture of the hand; and

move the cursor along an axis at the wearable near eye display based on the rotation of the hand along the axis at the navigation space, the hand remaining in a form of a first during the rotation.

2. The apparatus of claim 1, wherein the computer processor is further configured to instruct the wearable near eye display to present a computer generated scene to the user providing a virtual reality environment to the user.

3. The apparatus of claim 1, wherein the computer processor is further configured to instruct the wearable near eye display to present the captured scene to the user providing an augmented reality environment to the user.

4. The apparatus of claim 1, wherein the computer processor is further configured to generate a 3D depth map based on the reflections and to analyze the 3D depth map.

5. The apparatus of claim 1, wherein the computer processor is further configured to instruct the transmitter to illuminate only a portion of the navigation space comprising the hand.

6. The apparatus of claim 1, wherein the computer processor is further configured to present, over the wearable near eye display, a computer generated scene with which the cursor interacts.

7. The apparatus of claim 1, wherein the sensor and the transmitter are operable only within visible light bandwidth.

8. The apparatus of claim 1, wherein the sensor that captures the scene and the sensor that captures the reflections is a single element.

9. The apparatus of claim 1, wherein

the computer processor is further configured to move the cursor along one axis at the wearable near eye display based on the posture change of a finger of the hand along a defined axis at the navigation space.

10. The apparatus of claim 1, wherein the computer processor is further configured to move the cursor along the axis at the wearable near eye display based on tilting a palm of the hand along a specified arc at the navigation space.

11. The apparatus of claim 1, wherein a depth of field of the structured light pattern is selected to match a predefined location of the navigation space.

12. The apparatus of claim 1, wherein the structured light pattern is continuous along a first axis and non-continuous along a second axis perpendicular to the first axis.

13. The apparatus of claim 1, wherein the structured light pattern covers less than 10% of a field of view of the sensor.

23. The apparatus of claim 1, wherein the computer processor is further configured to identify the gesture of the hand from the set of predefined gestures by detecting segmentation of the structured light pattern reflected from the hand and captured by the sensor.

24. The apparatus of claim 1, wherein the computer processor is further configured to:

generate a 3D depth map of the navigation space based on the reflections of the structured light pattern captured by the sensor; and

analyze segmentation in the 3D depth map of the navigation space to identify the gesture of the hand from the set of predefined gestures.

14. A system comprising:

a transmitter attachable to a wearable near eye display;

a computer readable code executable on a computer processor and configured, when executed, to invoke the computer processor to;

instruct the transmitter to transmit a structured light pattern onto a navigation space and onto a hand of a user in the navigation space;

instruct a sensor attached to the wearable near eye display to capture a scene;

instruct the sensor to capture reflections of the structured light pattern coming from the navigation space;

analyze the reflections of the structured light pattern coming from the navigation space;

identify a gesture of the hand from a set of predefined gestures based on the reflections analysis, each gesture in the set of predefined gestures comprising a rotation or a posture change of the hand in the navigation space;

control a cursor presented to the user over the wearable near eye display according to the identified gesture of the hand, the cursor being a graphical pointing indication on the wearable near eye display interacting with content presented over the wearable near eye display;

invoke a corresponding action associated with the cursor over the wearable near eye display based on the identified gesture of the hand; and

move the cursor along an axis at the wearable near eye display based on the rotation of the hand along the axis at the navigation space, the hand remaining in a form of a first during the rotation.

15. A method comprising:

capturing a scene from a view point of a user;

transmitting a structured light pattern from the view point onto a navigation space and onto a hand of the user in the navigation space;

capturing reflections of the structured light pattern coming from the navigation space;

analyzing the reflections of the structured light pattern coming from the navigation space;

identifying a gesture of the hand from a set of predefined gestures based on the reflections analysis, each gesture in the set of predefined gestures comprising a rotation or a posture change of the hand in the navigation space;

controlling a cursor presented to the user over the wearable near eye display according to the identified gesture of the hand, the cursor being a graphical pointing indication on the wearable near eye display interacting with content presented over the wearable near eye display;

invoking a corresponding action associated with the cursor over the wearable near eye display based on the identified gesture of the hand; and

moving the cursor along an axis at the wearable near eye display based on the rotation of the hand along the axis at the navigation space, the hand remaining in a form of a first during the rotation.

16. The method of claim 15, further comprising instructing the wearable near eye display to present a computer generated scene to the user providing a virtual reality environment to the user.

17. The method of claim 15, further comprising instructing the wearable near eye display to present the captured scene to the user providing an augmented reality environment to the user.

18. The method of claim 15, wherein analyzing the reflections comprises generating a 3D depth map based on the reflections and analyzing the 3D depth map.

19. The method of claim 15, further comprising illuminating with the structured light pattern only a portion of the navigation space comprising the hand.

20. The method of claim 15, further comprising presenting over the wearable near eye display, a computer generated scene with which the cursor interacts.

21. The method of claim 15, further comprising moving the cursor along one axis at the wearable near eye display based on the posture change of a finger of the hand along a defined axis at the navigation space.

22. The method of claim 15, further comprising moving the cursor along the axis at the wearable near eye display based on tilting a palm of the hand along a specified arc at the navigation space.

25. The method of claim 15, further comprising identifying the gesture of the hand from the set of predefined gestures by detecting segmentation of the structured light pattern reflected from the hand.

26. The method of claim 15, further comprising:

generating a 3D depth map of the navigation space based on the captured reflections of the structured light pattern; and

analyzing segmentation in the 3D depth map of the navigation space to identify the gesture of the hand from the set of predefined gestures.