Systems and methods to facilitate interactions with virtual content in an augmented reality environment

• US 10,254,846 B1
• Filed: 03/15/2017
• Issued: 04/09/2019
• Est. Priority Date: 03/15/2017
• Status: Active Grant

• Current Assignee: Meta View, Inc.
• Original Assignee: Meta Company
• Inventors: Kinstner, Zachary R., Bott, Jared, Goodrich, Ryan
• Primary Examiner(s): Marinelli, Patrick F
• Application Number:
  US15/459,852
• Time in Patent Office: 755 Days
• CPC Class Codes: G06F 2203/04801 Cursor retrieval aid, i.e. visual aspect modification, blinking, colour changes, enlargement or other visual cues, for helping user do find the cursor in graphical user interfaces
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/0487 using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
• CPC Class Codes: G06F 9/453 Help systems
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/557 from light fields, e.g. from plenoptic cameras
• CPC Class Codes: G06T 7/73 using feature-based methods

Abstract

A system configured to facilitate interactions with virtual content in an augmented reality environment may comprise one or more of a hand tracking device, a light source, an optical element, one or more physical processor, non-transitory electronic storage, and/or other components. The hand tracking device may be configured to generate output signals conveying positions of a digit of a user'"'"'s hand over time. The light source may be configured to emit light forming images of virtual content. The optical element being configured to provide the light emitted from the light source into one or more eyes of a user. A visual indicator may be provide that visually indicates a perceived distance between the virtual content and the digit.

Claims

1. A system configured to facilitate interactions with virtual content in an augmented reality environment, the system comprising:

a hand tracking device configured to generate output signals conveying positions of a digit of a user'"'"'s hand over time;

a light source configured to emit light forming images of virtual content;

an optical element, the optical element being configured to provide the light emitted from the light source to an eye of the user to generate a perceived three-dimensional light field within the user'"'"'s field-of-view;

one or more physical processors configured by machine-readable instructions to;

control the light source to emit the light forming the images of the virtual content, the virtual content being formed by the three-dimensional light field so that the virtual content is perceived to be within the user'"'"'s field of view beyond the optical element, the virtual content including a first virtual object perceived at a first position within the user'"'"'s field of view at a point in time;

determine, based on the positions of the virtual content and the digit, perceived distances between the virtual content and the digit over time, including determining a first perceived distance between the first virtual object and the digit at the point in time;

control the light source to emit light forming images of a visual indicator on the first virtual object, the visual indicator visually indicating perceived distance between the virtual content and the digit and that the perceived distance indicated is for the first virtual object and the digit, such that at the point in time the visual indicator visually indicates the first perceived distance between the digit and the first virtual object and that the first perceived distance is indicated for the first virtual object and the digit,

wherein the first virtual object includes a first content element, the first content element being configured for interaction with the digit, and wherein a positioning of the visual indicator at or near the first content element indicates the interaction of the digit with the first content element;

determine, from the output signals, one or more gesture-based inputs being provided by the user, the one or more gesture-based inputs including a first gesture based input; and

disable the first gesture-based input based on the positioning of the visual indicator at or near the first content element indicating the interaction of the digit with the first content element.

2. The system of claim 1, wherein the visual indicator includes a circle having a centroid and a size defined by a diameter, wherein positioning of the centroid of the circle on the first virtual object conveys positioning of the digit relative the first virtual object, and the diameter of the circle conveys the first perceived distance between the digit and first virtual object.

3. The system of claim 1, wherein the position of the digit over time includes positions of a tip of the digit.

4. The system of claim 1, wherein the first virtual object has a planar shape.

5. The system of claim 1, wherein the first content element includes a hyperlink.

6. The system of claim 1, wherein the light source, the optical element, and the one or more physical processors are included in a headset configured to be worn by a user.

7. The system of claim 1, wherein the optical element is a waveguide or a partially reflective material.

8. A system configured to facilitate interactions with virtual content in an augmented reality environment, the system comprising:

a hand tracking device configured to generate output signals conveying positions of a digit of a user'"'"'s hand over time;

a light source configured to emit light forming images of virtual content;

an optical element, the optical element being configured to provide the light emitted from the light source to an eye of the user to generate a perceived three-dimensional light field within the user'"'"'s field-of-view;

one or more physical processors configured by machine-readable instructions to;

control the light source to emit the light forming the images of the virtual content, the virtual content being formed by the three-dimensional light field so that the virtual content is perceived to be within the user'"'"'s field of view beyond the optical element, the virtual content including a first virtual object perceived at a first position within the user'"'"'s field of view at a point in time;

determine, based on the positions of the virtual content and the digit, perceived distances between the virtual content and the digit over time, including determining a first perceived distance between the first virtual object and the digit at the point in time;

control the light source to emit light forming images of a visual indicator on the first virtual object, the visual indicator visually indicating perceived distance between the virtual content and the digit and that the perceived distance indicated is for the first virtual object and the digit, such that at the point in time the visual indicator visually indicates the first perceived distance between the digit and the first virtual object and that the first perceived distance is indicated for the first virtual object and the digit,

wherein the first virtual object includes a first content element, the first content element being configured for interaction with the digit, and wherein a positioning of the visual indicator at or near the first content element indicates the interaction of the digit with the first content element;

determine, from the output signals, one or more gesture-based inputs being provided by the user, the one or more gesture-based inputs including a first gesture-based input; and

disable the interaction of the digit with the first content element based on determining the first gestured-based input is being provided by the user.

9. A method to facilitate interactions with virtual content in an augmented reality environment, the method being implemented in a system comprising a hand tracking device, a light source, an optical element, one or more physical processors, and storage media storing machine-readable instructions, the method comprising:

generating, with the hand tracking device, output signals conveying positions of a digit of a user'"'"'s hand over time;

emitting, with the light source, light forming images of virtual content;

generating a perceived three-dimensional light field within the user'"'"'s field-of-view by providing, via the optical element, the light emitted from the light source to an eye of the user;

controlling, with the one or more physical processors, the light source to emit the light forming the images of the virtual content, the virtual content being formed by the three-dimensional light field so that the virtual content is perceived to be within the user'"'"'s field of view beyond the optical element, the virtual content including a first virtual object perceived at a first position within the user'"'"'s field of view at a point in time;

determining, with the one or more physical processors based on the positions of the virtual content and the digit, perceived distances between the virtual content and the digit over time, including determining a first perceived distance between the first virtual object and the digit at the point in time;

controlling, with the one or more physical processors, the light source to emit light forming images of a visual indicator on the first virtual object, the visual indicator visually indicating perceived distance between the virtual content and the digit and that the perceived distance indicated is for the first virtual object and the digit, such that at the point in time the visual indicator visually indicates the first perceived distance between the digit and the first virtual object and that the first perceived distance is indicated for the first virtual object and the digit,

wherein the first virtual object includes a first content element, the first content element being configured for interaction with the digit, and wherein a positioning of the visual indicator at or near the first content element indicates the interaction of the digit with the first content element;

determining, using the one or more physical processors and from the output signals, one or more gesture-based inputs being provided by the user, the one or more gesture-based inputs including a first gesture based input; and

disabling, using the one or more physical processors, the first gesture-based input based on the positioning of the visual indicator at or near the first content element facilitating the interaction of the digit with the first content element.

10. The method of claim 9, wherein the visual indicator includes a circle having a centroid and a size defined by a diameter, wherein positioning of the centroid of the circle on the first virtual object conveys positioning of the digit relative the first virtual object, and the diameter of the circle conveys the first perceived distance between the digit and first virtual object.

11. The method of claim 9, wherein the position of the digit over time includes positions of a tip of the digit.

12. The method of claim 9, wherein the first virtual object has a planar shape.

13. The method of claim 9, wherein the first content element includes a hyperlink.

14. The method of claim 9, wherein the light source, the optical element, and the one or more physical processors are included in a headset configured to be worn by a user.

15. The method of claim 9, wherein the optical element is a waveguide or a partially reflective material.

16. A method to facilitate interactions with virtual content in an augmented reality environment, the method being implemented in a system comprising a hand tracking device, a light source, an optical element, one or more physical processors, and storage media storing machine-readable instructions, the method comprising:

generating, with the hand tracking device, output signals conveying positions of a digit of a user'"'"'s hand over time;

emitting, with the light source, light forming images of virtual content;

generating a perceived three-dimensional light field within the user'"'"'s field-of-view by providing, via the optical element, the light emitted from the light source to an eye of the user;

controlling, with the one or more physical processors, the light source to emit the light forming the images of the virtual content, the virtual content being formed by the three-dimensional light field so that the virtual content is perceived to be within the user'"'"'s field of view beyond the optical element, the virtual content including a first virtual object perceived at a first position within the user'"'"'s field of view at a point in time;

determining, with the one or more physical processors based on the positions of the virtual content and the digit, perceived distances between the virtual content and the digit over time, including determining a first perceived distance between the first virtual object and the digit at the point in time;

controlling, with the one or more physical processors, the light source to emit light forming images of a visual indicator on the first virtual object, the visual indicator visually indicating perceived distance between the virtual content and the digit and that the perceived distance indicated is for the first virtual object and the digit, such that at the point in time the visual indicator visually indicates the first perceived distance between the digit and the first virtual object and that the first perceived distance is indicated for the first virtual object and the digit,

wherein the first virtual object includes a first content element, the first content element being configured for interaction with the digit, and wherein a positioning of the visual indicator at or near the first content element indicates the interaction of the digit with the first content element;

determining, using the one or more physical processors and from the output signals, one or more gesture-based inputs being provided by the user, the one or more gesture-based inputs including a first gesture-based input; and

disabling, using the one or more physical processors, the interaction of the digit with the first content element based on determining the first gestured-based input is being provided by the user.