Eye tracking systems and method for augmented or virtual reality

• US 10,825,248 B2
• Filed: 05/08/2015
• Issued: 11/03/2020
• Est. Priority Date: 04/18/2014
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Miller, Samuel A.
• Primary Examiner(s): Mengistu, Amare
• Assistant Examiner(s): Mathews, Crystal
• Application Number:
  US14/707,000
• Publication Number:
  US 20150301599A1
• Time in Patent Office: 2,006 Days
• CPC Class Codes: A63F 13/56 Computing the motion of game characters with respect to other game characters, game objects or elements of the game scene, e.g. for simulating the behaviour of a group of virtual soldiers or for path finding
• CPC Class Codes: A63F 13/57 Simulating properties, behaviour or motion of objects in the game world, e.g. computing tyre load in a car race game A63F13/56 takes precedence
• CPC Class Codes: A63F 13/577 using determination of contact between game characters or objects, e.g. to avoid collision between virtual racing cars
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0101 characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0179 Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 3/167 : Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06Q 30/02 : Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06T 11/60 : Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 13/40 : of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 13/80 : 2D [Two Dimensional] animation, e.g. using sprites
• CPC Class Codes: G06T 15/10 : Geometric effects
• CPC Class Codes: G06T 19/003 : Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/04 : involving 3D image data
• CPC Class Codes: G06T 2213/08 : Animation software package
• CPC Class Codes: G06T 2215/16 : Using real world measurements to influence rendering
• CPC Class Codes: G06T 2219/024 : Multi-user, collaborative environment
• CPC Class Codes: G06T 7/60 : Analysis of geometric attributes
• CPC Class Codes: G06T 7/70 : Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/10 : Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/653 : by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 20/80 : Recognising image objects characterised by unique random patterns
• CPC Class Codes: H04S 2400/11 : Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2400/15 : Aspects of sound capture and related signal processing for recording or reproduction
• CPC Class Codes: H04S 7/304 : For headphones

Abstract

An augmented reality display system comprises a passable world model data comprises a set of map points corresponding to one or more objects of the real world. The augmented reality system also comprises a processor to communicate with one or more individual augmented reality display systems to pass a portion of the passable world model data to the one or more individual augmented reality display systems, wherein the piece of the passable world model data is passed based at least in part on respective locations corresponding to the one or more individual augmented reality display systems.

Claims

1. A method, comprising:

rendering one or more virtual objects in a virtual or augmented reality environment with a virtual image generation subsystem of a virtual content display system altering, with a variable focus element, one or more rays or cones of light among multiple planes that correspond to respective virtual depths towards at least one eye of a user, wherein

the virtual image generation subsystem comprising a graphics processing unit, at least one projector, the variable focus element, a first assembly, and a second assembly that is operatively coupled to the first assembly,

the first assembly and second assembly receive power from a portable power source separate from the virtual image generation subsystem,

the first assembly includes a first board and a projector driver providing image information and signals to the at least one projector,

the second assembly includes a second board, a microprocessor, the graphics processing unit, and one or more motion sensors or transducers, and

the at least one projector projects image information of the one or more virtual objects to the at least one eye of the user;

representing the at least one eye with an eye model that comprises a first circular shape or circle and a second circular shape or circle, wherein the second circular shape or circle represents a cornea of the at least one eye and is layered on top of the first circular shape or circle;

detecting, with at least an eye tracking device, one or more characteristics pertaining to an interaction between the at least one eye of the user and reflected light from the at least one eye at least by;

capturing, by a first set of sensors or transducers in the first assembly, a first pattern emitted or reflected from one or more ambient light sources in a real-world environment;

projecting, with at least the projector driver in the first assembly and the at least one projector, a second light pattern generated by a set of light sources in the first assembly to the at least one eye of the user;

in response to the first light pattern, detecting the reflected light from the at least one eye using one or more second sensors or transducers in the first assembly; and

determining, by the microprocessor in the second assembly, the interaction at least by correlating the reflected light with the first light pattern and the second light pattern;

determining, by the microprocessor in the second assembly, an eye pointing vector and a center of rotation of the at least one eye in the eye model using at least the first circular shape or circle and the second circular shape or circle based at least in part upon a characteristic of a cross-section of the at least one eye and a range of movement of the at least one eye;

determining, by the microprocessor in the second assembly, a vectored distance for the eye pointing vector based at least in part upon the one or more characteristics and the center of rotation for the at least one eye in the eye model; and

determining, by the microprocessor in the second assembly, at least one movement or pose for both the at least one eye and another eye of the user at least by using the vectored distance of the at least one eye and further by extrapolating one or more eye movement or pose characteristics with at least one or more parameters pertaining to the interaction and captured by one or more sensors for the at least one eye of the user.

2. The method of claim 1, wherein at least one characteristic of the one or more characteristics pertains to light reflected back from the at least one eye, and the plurality of eye pointing vectors are determined without using cameras.

3. The method of claim 1, wherein at least one characteristic of the one or more characteristics pertains to one or more reflections of one or more objects from a structure of the at least one eye of the user, and the one or more characteristics comprise an extent of backscattered light detected by the first set of sensors or transducers in the first assembly.

4. The method of claim 1, further comprising varying at least one parameter of the one or more rays or cones of light.

5. The method of claim 4, further comprising pseudo-randomly varying the at least one parameter.

6. The method of claim 4, wherein the at least one parameter corresponds to a length of emission of the light source.

7. The method of claim 1, wherein the one or more rays or cones of light are emitted in a predetermined pattern.

8. The method of claim 7, further comprising correlating the one or more characteristics with a set of one or more known characteristics to determine the at least one movement or pose.

9. The method of claim 1, further comprising determining the at least one movement or pose based at least in part on a known distance of the at least one eye from one or more sensors detecting a first characteristic of the interaction between the one or more rays or cones of light and the at least one eye of the user and a plurality of light sources emitting the one or more rays or cones of light to the at least one eye of the user.

10. The method of claim 1, further comprising:

configuring a heat map that tracks and records at least one or more numbers of instances of poses or movements of the at least one eye of the user in relation to one or more virtual objects or one or more real-world objects;

storing information of at least one virtual object or at least one real-world object that produces a most number of instances of the poses or the movements in the heat map; and

rendering virtual contents to the user based in part or in whole upon the heat map or upon the one or more characteristics pertaining to the interaction.

11. A virtual content display system for tracking one or more movements or poses of a user'"'"'s eyes, comprising:

a virtual image generation subsystem configured to rendering one or more virtual objects in a virtual or augmented reality environment with a virtual image generation subsystem of a virtual content display system altering, with a variable focus element, one or more rays or cones of light among multiple planes that corresponds to respective virtual depths towards at least one eye of a user, wherein

the virtual image generation subsystem comprising a graphics processing unit, at least one projector, the variable focus element, a first assembly, and a second assembly that is operatively coupled to the first assembly,

the first assembly and second assembly receive power from a portable power source separate from the virtual image generation subsystem,

the first assembly includes a first board and a projector driver providing image information and signals to the at least one projector,

the second assembly includes a second board, a microprocessor, the graphics processing unit, and one or more motion sensors or transducers, and

the at least one projector projects image information of the one or more virtual objects to the at least one eye of the user;

an eye model that represents the at least one eye with a first circular shape or circle and a second circular shape or circle that models a cornea of the at least one eye and is layered on top of the first circular shape or circle;

an eye tracking device positioned in relation to the virtual image generation subsystem as well as the at least one eye of the user and configured at least to;

detect one or more characteristics pertaining to an interaction between the at least one eye of the user and reflected light from the at least one eye at least by;

capturing, by a first set of sensors or transducers in the first assembly, a first pattern emitted or reflected from one or more ambient light sources in a real-world environment;

projecting, with at least the projector driver in the first assembly and the at least one projector, a second light pattern generated by a set of light sources in the first assembly to the at least one eye of the user;

in response to the first light pattern, detecting the reflected light from the at least one eye using one or more second sensors or transducers in the first assembly; and

determining, by the microprocessor in the second assembly, the interaction at least by correlating the reflected light with the first light pattern and the second light pattern;

the microprocessor in the second assembly further configured to determine an eye pointing vector and a center of rotation of the at least one eye in the eye model using at least the first circular shape or circle and the second circular shape or circle based at least in part upon a characteristic of a cross-section of the at least one eye and a range of movement of the at least one eye;

the microprocessor in the second assembly further configured to determine a vectored distance for the eye pointing vector based at least in part upon the one or more characteristics and the center of rotation for the at least one eye in the eye model; and

the microprocessor in the second assembly further configured to determine at least one movement or pose for both the at least one eye and another eye of the user at least by using the vectored distance of the at least one eye and further by extrapolating one or more eye movement or pose characteristics with at least one or more parameters pertaining to the interaction and captured by one or more sensors for the at least one eye of the user.

12. The virtual content display system of claim 11, wherein the microprocessor is configured to determine an eye pointing vector for the at least one eye without using cameras.

13. The virtual content display system of claim 11, wherein the virtual content display system is instrumented with the eye tracking device only for the at least one eye rather than for the both eyes of the user.

14. The virtual content display system of claim 11, further comprising:

a heat map that is configured to track and record at least one or more numbers of instances of poses or movements of the at least one eye of the user in relation to one or more virtual objects or one or more real-world objects.

15. The virtual content display system of claim 14, wherein microprocessor and the eye tracking device are configured to store information of at least one virtual object or at least one real-world object that produces a most number of instances of the poses or the movements in the heat map.

16. The virtual content display system of claim 14, wherein virtual content display system is further configured to render virtual contents to the user based in part or in whole upon the heat map or upon the one or more characteristics pertaining to the interaction.

17. The virtual content display system of claim 11, wherein the eye tracking device is configured to constantly track the one or more characteristics pertaining to the interaction, and the one or more characteristics comprise at least one of a location of the interaction, a force of the interaction, a direction of the interaction, a frequency of the interaction, a number of the interaction, or a nature of the interaction.

18. The virtual content display system of claim 11, further comprising:

a mapping table that is configured to correlate the at least one movement or pose with a command or an action; and

an execution module that is configured to execute the command or the action in response to the at least one movement or pose.

19. The virtual content display system of claim 11, further comprising:

a pseudo random pattern generation module that is operatively coupled to the light field generation subsystem and is configured to generate a patterned code that is used to pseudo-randomly vary an emission characteristic of the one or more rays or cones of light.

20. The virtual content display system of claim 11, wherein the eye tracking device is configured to detect the one or more characteristics based in part or in whole upon a vector distance that is determined by overlaying two geometric objects on top of each other based in part or in whole on geometric characteristics of the at least one eye of the user.