GAZE DETECTION IN A SEE-THROUGH, NEAR-EYE, MIXED REALITY DISPLAY

• US 20130286178A1
• Filed: 03/15/2013
• Published: 10/31/2013
• Est. Priority Date: 08/29/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Lewis, John R., Wei, Yichen, Crocco, Robert L., Vaught, Benjamin I., Kipman, Alex Aben-Athar, Perez, Kathryn Stone
• Granted Patent: US 9,110,504 B2
• Time in Patent Office: Days
• US Class Current: 348/78
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 30/34 Stereoscopes providing a stereoscopic pair of separated images corresponding to parallactically displaced views of the same object, e.g. 3D slide viewers
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06V 40/19 Sensors therefor

Abstract

The technology provides various embodiments for gaze determination within a see-through, near-eye, mixed reality display device. In some embodiments, the boundaries of a gaze detection coordinate system can be determined from a spatial relationship between a user eye and gaze detection elements such as illuminators and at least one light sensor positioned on a support structure such as an eyeglasses frame. The gaze detection coordinate system allows for determination of a gaze vector from each eye based on data representing glints on the user eye, or a combination of image and glint data. A point of gaze may be determined in a three-dimensional user field of view including real and virtual objects. The spatial relationship between the gaze detection elements and the eye may be checked and may trigger a re-calibration of training data sets if the boundaries of the gaze detection coordinate system have changed.

Claims

1. A method for determining gaze in a near-eye mixed reality display device comprising:

determining boundaries of a three dimensional gaze detection coordinate system for each eye based on positions of glints detected on the respective eye, positions on the near-eye display device of illuminators for generating the glints, and a position of at least one sensor for capturing reflected eye data of the respective eye from which the positions of glints are detected;

automatically checking periodically the reflected eye data for any change along a depth axis of the gaze detection coordinate system, the depth axis extending from a respective display optical system of the near-eye, mixed reality display device toward the respective eye;

responsive to detecting any change along at least one depth axis, triggering redetermination of the boundaries of the three dimensional gaze detection coordinate system;

determining a gaze vector for each eye based on reflected eye data including the glints and the boundaries of the three dimensional gaze detection coordinate system;

determining a point of gaze based on the gaze vectors for the eyes in a three-dimensional (3D) user field of view of the near-eye mixed reality display device including real and virtual objects; and

identifying any object at the point of gaze in the 3D user field of view.

2. The method of claim 1 wherein the at least one sensor for capturing reflected eye data includes a camera which captures image data and wherein automatically checking periodically for any changes along the depth axis of the gaze detection coordinate system further comprises:

determining a change along the depth axis based on a change of size of a facial feature in the image data captured by the camera.

3. The method of claim 2 wherein the facial feature comprises an iris.

4. A method for determining gaze in a near-eye mixed reality display device comprising:

determining boundaries of a three dimensional gaze detection coordinate system for each eye based on positions of glints detected on the respective eye, positions on the near-eye display device of illuminators for generating the glints, and a position of at least one sensor for capturing reflected eye data of the respective eye from which the positions of glints are detected;

generating and storing respective training gaze data sets based on the boundaries of the three dimensional gaze detection coordinate system, each training gaze data set including pupil position data and a gaze vector;

based on the reflected eye data of the respective eye, determining current pupil position data for the respective eye;

determining a gaze vector for each eye based on comparison of its current pupil position data with its training gaze data sets;

determining a point of gaze based on the gaze vectors for the eyes in a three-dimensional (3D) user field of view of the near-eye mixed reality display device including real and virtual objects;

identifying any object at the point of gaze in the 3D user field of view;

automatically checking periodically the reflected eye data for any change indicating the training gaze data sets are to be re-calibrated; and

responsive to detecting any change indicating the training gaze data sets are to be re-calibrated along at least one of the depth axes, triggering re-calibration of the training gaze data sets for the eyes.

5. The method of claim 4 wherein determining a gaze vector for each eye based on comparison of its current pupil position data with its training gaze data sets further comprises:

determining one or more distance vectors between the current pupil position data and the pupil position data of at least a subset of the training gaze data sets in accordance with a mapping criteria;

selecting the training gaze data set with the smallest sum for its one or more distance vectors;

interpolating a position change vector between the pupil position data of the selected training gaze data set to the current pupil position data; and

estimating as the gaze vector a vector representing the gaze vector of the selected training gaze data set moved by the position change vector.

6. The method of claim 5 wherein automatically checking periodically the reflected eye data for any change indicating the training gaze data sets are to be re-calibrated further comprises automatically checking periodically for any change along a depth axis of the gaze detection coordinate system, the depth axis extending from a respective display optical system of the near-eye, mixed reality display device toward the respective eye.

7. The method of claim 6 wherein the at least one sensor for capturing reflected eye data includes a camera which captures image data and wherein automatically checking periodically for any changes along the depth axis of the gaze detection coordinate system further comprises:

determining a change along the depth axis based on a change of size of a facial feature in the image data captured by the camera.

8. The method of claim 7 wherein the facial feature comprises an iris.

9. The method of claim 5 wherein automatically checking periodically the reflected eye data for any change indicating the training gaze data sets are to be re-calibrated further comprises automatically checking periodically whether a lighting change has occurred in accordance with a criteria.

10. A mixed reality display system with gaze determination comprising:

a near-eye, mixed reality display device;

at least one image generation unit for generating at least one virtual image for display by the near-eye, mixed reality display device;

a respective arrangement of gaze detection elements positioned on the display device for forming a spatial relationship between the gaze detection elements and each eye, the gaze detection elements including

a set of illuminators for generating glints on the respective eye each illuminator positioned on the see-through display device at a respective predetermined position and generating illumination about a predetermined wavelength, and

at least one respective sensor for capturing light reflected from the respective eye and generating data representing the captured reflected light, the at least one respective sensor positioned at a predetermined position in relation to predetermined positions of the set of illuminators on the see-through display device;

a memory for storing software and the data; and

one or more processors communicatively coupled to the at least one respective sensor to receive the data representing the captured reflected light and having access to the memory for storing the data, the one or more processors determining a gaze vector for each respective eye based on the data representing the captured reflected light and a point of gaze based on the gaze vectors in a three-dimensional (3D) user field of view;

wherein data representing the captured reflected light includes glint intensity data and wherein for each eye, the one or more processors determining the gaze vector comprises;

in a time period, using a first technique that comprises determining a center of a cornea and a center of the pupil based on image data of the respective eye a first number of times, and

during the same time period, using a second different technique based on the glint intensity data independent of the image data of the respective eye for a second number of times;

the one or more processors automatically checking periodically the reflected eye data for any change along a depth axis of the gaze detection coordinate system, the depth axis extending from a respective display optical system of the near-eye, mixed reality display device toward the respective eye; and

responsive to detecting any change along at least one depth axis, the one or more boundaries triggering redetermination of the boundaries of the three dimensional gaze detection coordinate system.

11. The system of claim 10 further comprising:

wherein the illuminators are infra-red (IR) illuminators which transmit about a predetermined wavelength;

at least one light sensor is an IR camera for generating image data of the glints and a pupil; and

wherein the first technique comprises the one or more processors determining the gaze vector including the center of the cornea and the center of the pupil based on the image data of glints and the pupil and the spatial relationship.

12. The system of claim 10 wherein

the first technique that comprises determining a center of a cornea and a center of the pupil based on image data of the respective eye a first number of times further comprises

determining boundaries of a gaze detection coordinate system representing the spatial relationship between the gaze detection elements and each eye, based on positions of glints detected on a user eye, positions on the near-eye display system of illuminators for generating the glints, and a position of at least one sensor for detecting the glints.

13. The system of claim 12 wherein determining boundaries of a gaze detection coordinate system representing the spatial relationship between the gaze detection elements and each eye, based on positions of glints detected on a user eye, positions on the near-eye display system of illuminators for generating the glints, and a position of at least one sensor for detecting the glints further comprises:

determining a position of a center of a cornea of each respective eye with respect to the positions of the illuminators and the at least one sensor based on positions of glints detected on the cornea surface of the respective eye;

determining a pupil center from image data generated from the at least one sensor; and

determining a position of a fixed center of eyeball rotation of each eye based on the position of the cornea center and the pupil center.

14. The method of claim 13, further comprising:

determining a position from the fixed center of eyeball rotation of each eye with respect to the respective illuminators and the at least one sensor for determining movement of the display device with respect to the respective eye.

15. The system of claim 10 further comprising:

wherein the one or more processors determining the gaze vector using the first technique is performed at a first rate; and

the one or more processors determining the gaze vector by the second technique is performed at a second rate faster than the first rate.

16. The system of claim 15 further comprising:

wherein the second different technique based on the glint intensity data independent of the image data of the respective eye further comprises

the one or more processors communicatively coupled to the at least one respective sensor to receive the glint data and having access to the memory for storing the glint data, the one or more processors determining one or more glint positions relative to an eye part based on the glint intensity data generated by the at least one respective sensor, the respective predetermined position of each illuminator in the set of infra-red illuminators, and the at least one predetermined position of the at least one respective sensor for detecting the glints reflected from each eye;

the one or more processors determining a gaze vector for each eye based on the one or more glint positions relative to an eye part; and

the one or more processors determining a point of gaze based on the gaze vectors in a three dimensional user field of view.

17. The system of claim 16 further comprising:

the one or more processors being communicatively coupled to the at least one image generation unit for controlling the at least one image generation unit for displaying the at least one virtual image at different locations in the user field of view viewable through the see-through display device;

the at least one respective sensor capturing a calibration glint data set of intensity values for each glint during display of the at least one virtual image at each different location;

the one or more processors storing the calibration glint data sets of intensity values as indicators of calibration gaze vectors; and

the one or more processors determines the gaze vector of a user based on comparisons of the captured glint data with the calibration glint data sets.

18. The system of claim 16 further comprising:

wherein the at least one respective sensor is a position sensitive detector and the one or more processors turns each illuminator on and off in a predetermined sequence for capturing one or more data sets of glint intensity values.

19. The system of claim 16 wherein the values of the glint intensity data are correlated with stored values for specular reflectivities of different eye parts.

20. The system of claim 19 wherein the different eye parts comprise an iris or a pupil or a sclera or a combination of two or more of an iris, pupil and sclera.