Eye tracking calibration techniques

• US 10,671,160 B2
• Filed: 05/30/2018
• Issued: 06/02/2020
• Est. Priority Date: 05/31/2017
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Uscinski, Benjamin Joseph, Xu, Yan, Stuart, Bradley Vincent
• Primary Examiner(s): Pervan, Michael
• Application Number:
  US15/993,371
• Publication Number:
  US 20180348861A1
• Time in Patent Office: 734 Days
• Field of Search: 345 7- 8, 351208-209
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/22 Setup operations, e.g. calibration, key configuration or button assignment
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06T 15/06 Ray-tracing
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/19 Sensors therefor

Abstract

Systems and methods for eye tracking calibration in a wearable system are described. The wearable system can present three-dimensional (3D) virtual content and allow a user to interact with the 3D virtual content using eye gaze. During an eye tracking calibration, the wearable system can validate that a user is indeed looking at a calibration target while the eye tracking data is acquired. The validation may be performed based on data associated with the user'"'"'s head pose and vestibulo-ocular reflex.

Claims

1. A wearable system for eye tracking calibration comprising:

an image capture device configured to capture eye images of one or two eyes of a user of the wearable system;

non-transitory memory configured to store the eye images;

a display system through which the user can perceive a virtual eye calibration target in an environment of the user and a virtual reticle;

a pose sensor configured to track a head pose of the user;

a hardware processor in communication with the non-transitory memory and the display system, the hardware processor programmed to;

cause the virtual eye calibration target to be perceivable via the display system;

determine one or more parameters comprising at least one of;

one or more characteristics of the user or one or more characteristics of the display system;

determine a representation of the virtual reticle based on the one or more parameters;

cause the representation of the virtual reticle to be rendered via the display system;

identify the user'"'"'s head pose based on data acquired from the pose sensor;

calculate a position of the virtual reticle based on the user'"'"'s head pose;

determine whether the virtual reticle aligns with the virtual eye calibration target based at least partly on the position of the virtual reticle and the virtual eye calibration target; and

in response to a determination that the virtual reticle aligns with the virtual eye calibration target, instruct the image capture device to capture the eye images and initiate storage of the eye images into the non-transitory memory.

2. The wearable system of claim 1, wherein the image capture device comprises an eye camera configured to image the user'"'"'s eye region.

3. The wearable system of claim 1, wherein the pose sensor comprises an inertial measurement unit.

4. The wearable system of claim 1, wherein the virtual eye calibration target is an object in a world space and the virtual reticle is in a rig space.

5. The wearable system of claim 4, wherein the world space comprises a three-dimensional (3D) coordinate system with reference to the environment of the user and the rig space comprises a coordinate system with reference to the display system.

6. The wearable system of claim 5, wherein to determine whether the virtual reticle aligns with the virtual eye calibration target, the hardware processor is programmed to:

determine an offset between the rig space and the world space;

calculate a first coordinate value of the virtual reticle in the rig space and a second coordinate value of the virtual eye calibration target in the world space; and

calculate an alignment based on the first coordinate value, the second coordinate value, and the offset.

7. The wearable system of claim 6, wherein to calculate the alignment, the hardware processor is programmed to convert the first coordinate value in the rig space to a third coordinate value in the world space based on the offset and determine whether the third coordinate value and the second coordinate value are within a threshold range.

8. The wearable system of claim 1, wherein the user'"'"'s head pose comprises a position, an orientation, or a combination of the position and orientation of the user'"'"'s head.

9. The wearable system of claim 1, wherein to determine whether the virtual reticle aligns with the virtual eye calibration target, the hardware processor is programmed to:

perform a ray cast based on the position of the virtual reticle; and

determine whether a portion of a ray in the ray cast intersects with the virtual eye calibration target.

10. The wearable system of claim 1, wherein to determine whether the virtual reticle aligns with the virtual eye calibration target, the hardware processor is programmed to:

access a target head pose wherein the target head pose is a pose that the user assumes when the virtual reticle aligns with the virtual eye calibration target; and

determine whether the user'"'"'s head pose is at the target head pose.

11. The wearable system of claim 1, wherein in response to the determination that the virtual reticle aligns with the virtual eye calibration target, the hardware processor is further programmed to:

provide a visual, audible, or tactile effect indicating that the virtual reticle is aligned with the virtual eye calibration target.

12. The wearable system of claim 1, wherein in response to a determination that the virtual reticle aligns with the virtual eye calibration target, the hardware processor is further programmed to instruct the display system to present another virtual reticle.

13. The wearable system of claim 1, wherein the hardware processor is further programmed to dynamically update an appearance of the virtual reticle based at least partly on contextual information comprising at least one of:

a size of the eye calibration target, a color of the virtual eye calibration target, or a required precision associated with the eye tracking calibration.

14. The wearable system of claim 13, wherein the appearance of the virtual reticle resembles an appearance of the virtual eye calibration target.

15. The wearable system of claim 13, wherein the hardware processor is programmed to reduce a size of the reticle in response to a high required precision or a small eye calibration target.

16. A method for eye tracking calibration comprising:

under control of a hardware processor;

identifying a virtual eye calibration target in a world space associated with an environment of a user;

determining one or more parameters comprising at least one of;

one or more characteristics of the user or one or more characteristics of the display system;

determining a representation of a virtual reticle based on the one or more parameters;

instructing a wearable device of the user to render the representation of the virtual reticle in a rig space associated with the wearable device of the user;

obtaining head pose data of the user;

determining a current head pose of the user based on the obtained head pose data;

determining a relative position of the virtual reticle with respect to the virtual eye calibration target based at least partly on the user'"'"'s current head pose;

determining whether the virtual reticle aligns with the virtual eye calibration target based at least in part on the relative position of the virtual reticle with respect to the virtual eye calibration target; and

in response to a determination that the virtual reticle aligns with the virtual eye calibration target, instructing an inward-facing imaging system of the wearable device to acquire eye calibration data.

17. The method of claim 16, wherein the head pose data is obtained from at least one of:

an outward-facing imaging system of the wearable device, an inertial measurement unit of the wearable device, or a camera in the environment of the user.

18. The method of claim 16, wherein the eye calibration data comprises images of one or both eyes of the user.

19. The method of claim 18, wherein the virtual eye calibration target is at a fixed location in the world space, or wherein the virtual reticle is rendered at a fixed location in the rig space.

20. The method of claim 19, further comprising in response to a determination that the virtual reticle aligns with the virtual eye calibration target, instructing the wearable device to render the virtual reticle at a different location in the rig space.

21. The method of claim 16, wherein the virtual reticle is rendered at or near a center of a field of view of the user, wherein the field of view comprises a portion of the user'"'"'s environment that the user can perceive at a given time.

22. The method of claim 16, wherein determining a current head pose of the user based on the obtained head pose data comprises calculating a position or an orientation of the user'"'"'s head based on the head pose data.

23. The method of claim 16, further comprising:

accessing a range of head poses and wherein determining whether the virtual reticle aligns with the virtual eye calibration target comprises determining whether the current head pose is within the range of head poses.

24. The method of claim 16, wherein the virtual eye calibration target is a virtual object in a game application and the method for eye tracking calibration is performed as part of the game application.

25. A method for eye tracking calibration comprising:

under control of a hardware processor;

identifying a target region in an environment of a user;

determining one or more parameters comprising at least one of;

one or more characteristics of a user or one or more characteristics of a wearable device configured to display a virtual reticle;

determining a representation of the virtual reticle based on the one or more parameters;

instructing the wearable device to display the virtual reticle at a position in a rig space associated with a wearable device of the user;

obtaining head pose data of the user;

determining a current head pose of the user based on the obtained head pose data;

updating the position of the virtual reticle relative to the target region based at least partly on the user'"'"'s current head pose;

determining whether the virtual reticle aligns with the target region; and

in response to a determination that the virtual reticle aligns with the target region, providing an indication that an alignment has been achieved.

26. The method of claim 25, wherein the target region comprises at least a virtual object, and wherein determining whether the virtual reticle aligns with the target region comprises determining whether the virtual reticle aligns with the virtual object.

27. The method of claim 25, wherein the target region comprises at least a portion of a physical object, and wherein determining whether the virtual reticle aligns with the target region comprises determining whether the virtual reticle aligns with the portion of the physical object.

28. The method of claim 25, wherein the eye calibration data is collected by a capacitive-based or an electrode-based eye tracking system.

29. The method of claim 25, wherein in response to the determination that the virtual reticle aligns with the eye calibration target, the method further comprises causing an eye camera of the wearable device to initiate collection of eye images.

30. The method of claim 25, wherein determining whether the virtual reticle aligns with the target region comprises:

casting at least one virtual ray based at least partly on the user'"'"'s current head pose; and

determining whether the at least one virtual ray intersects with the target region.

31. The method of claim 30, wherein casting at least one virtual ray based at least partly on the user'"'"'s current head pose comprises:

casting at least one virtual ray from a location on the user or the wearable device, and through the position of the virtual reticle.

32. The method of claim 30, wherein casting at least one virtual ray based at least partly on the user'"'"'s current head pose comprises:

casting a virtual geometric cone through the location of the virtual reticle.

33. The method of claim 32, wherein a shape of the virtual reticle corresponds to a cross-sectional shape of the virtual geometric cone at the location of the virtual reticle.