Gaze detection in a see-through, near-eye, mixed reality display

• US 8,928,558 B2
• Filed: 07/12/2013
• Issued: 01/06/2015
• Est. Priority Date: 08/29/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Lewis, John R., Wei, Yichen, Crocco, Robert L., Vaught, Benjamin I., Kipman, Alex Aben-Athar, Perez, Kathryn Stone
• Primary Examiner(s): Sharifi-Tafreshi, Koosha
• Application Number:
  US13/941,439
• Publication Number:
  US 20130300653A1
• Time in Patent Office: 543 Days
• Field of Search: 345/7, 345/8, 345/9, 345/156, 359/630, 396/51
• US Class Current: 345/8
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 30/34 Stereoscopes providing a stereoscopic pair of separated images corresponding to parallactically displaced views of the same object, e.g. 3D slide viewers
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06V 40/19 Sensors therefor

Abstract

The technology provides various embodiments for gaze determination within a see-through, near-eye, mixed reality display device. In some embodiments, the boundaries of a gaze detection coordinate system can be determined from a spatial relationship between a user eye and gaze detection elements such as illuminators and at least one light sensor positioned on a support structure such as an eyeglasses frame. The gaze detection coordinate system allows for determination of a gaze vector from each eye based on data representing glints on the user eye, or a combination of image and glint data. A point of gaze may be determined in a three-dimensional user field of view including real and virtual objects. The spatial relationship between the gaze detection elements and the eye may be checked and may trigger a re-calibration of training data sets if the boundaries of the gaze detection coordinate system have changed.

Claims

1. A gaze vector determination system for use in a mixed reality display system comprising:

at least one image generation unit for generating at least one virtual image for display by a near-eye, mixed reality display device;

a respective arrangement of gaze detection elements positioned on the display device for forming a spatial relationship between the gaze detection elements and at least one eye, the gaze detection elements including;

a set of illuminators for generating glints on the at least one eye each illuminator positioned on the display device at a respective predetermined position and generating illumination about a predetermined wavelength, and

at least one sensor for capturing light reflected from the at least one eye and generating data representing the captured reflected light, the at least one sensor positioned at a predetermined position in relation to predetermined positions of the set of illuminators on the display device;

a memory for storing software and the data; and

one or more processors communicatively coupled to the at least one sensor to receive the data representing the captured reflected light and having access to the memory for storing the data, the one or more processors determining a gaze vector for the at least one eye based on the data representing the captured reflected light, wherein data representing the captured reflected light includes glint intensity data and wherein for the at least one eye, the one or more processors determining the gaze vector comprises;

in a time period, using a first technique that comprises determining at least one of a center of a cornea and a center of a pupil based on image data of the at least one eye a first number of times, and

during the same time period, using a second different technique based on the glint intensity data independent of the image data of the at least one eye for a second number of times.

2. The system of claim 1 wherein the one or more processors determining the gaze vector using the second different technique further comprises the one or more processors determining a position for each glint with respect to at least one eyeball part based on the glint intensity data and the predetermined positions of the set of illuminators and the at least one respective sensor.

3. The system of claim 2 wherein the values of the glint intensity data are correlated with stored values for specular reflectivities of different eye parts.

4. The system of claim 3 wherein the different eye parts comprise an iris or a pupil or a sclera or a combination of two or more of the iris, the pupil and the sclera.

5. The system of claim 1 further comprising:

wherein the illuminators are infra-red (IR) illuminators which transmit about a predetermined wavelength;

at least one light sensor is an IR camera for generating image data of the glints and the pupil; and

wherein the one or more processors determining the gaze vector using the first technique comprises the one or more processors determining the gaze vector including the center of the cornea and the center of the pupil based on the image data of glints and the pupil and the spatial relationship.

6. The system of claim 1 further comprising:

the one or more processors determining boundaries of a three dimensional gaze detection coordinate system for the at least one eye based on the spatial relationship between the gaze detection elements positioned on the near-eye, mixed reality display device and positions of the glints detected on the at least one eye;

the one or more processors being communicatively coupled to the at least one image generation unit for displaying the at least one virtual image at different locations in a user field of view viewable in the display device;

the one or more processors generating a training gaze data set for each different location by capturing image data of the pupil of the at least one eye by the at least one sensor during display of the at least one virtual image at each of the different locations, identifying a black pupil area in the image data for each location, applying an ellipse fitting algorithm to the black pupil area for identifying pupil position data identifying a position of the eye within an eyeball for each location; and

determining a gaze vector based on the pupil position data and the boundaries of the three dimensional gaze detection coordinate system;

the one or more processors storing training gaze data sets, each training gaze data set storing the pupil position data and the gaze vector determined during display of the virtual image at a respective location of the different locations; and

wherein using a first technique that comprises determining at least one of a center of a cornea and a center of a pupil based on image data of the at least one eye a first number of times further comprises;

based on image data of the pupil in the data representing the captured reflected light, the one or more processors determining current pupil position data for the at least one eye, and

the one or more processors determining the gaze vector based on comparison of the current pupil position data with pupil position data stored in the training gaze data sets and their associated stored gaze vectors.

7. The system of claim 5 further comprising:

wherein the one or more processors determining the gaze vector using the first technique is performed at a first rate; and

the one or more processors determining the gaze vector by the second different technique is performed at a second rate faster than the first rate.

8. The system of claim 5 wherein the IR camera is aligned with an optical axis of a respective display optical system positioned by the display device to be seen through by a respective eye.

9. A method for determining a gaze vector of a user eye based on a gaze detection coordinate system of a near-eye display device comprising:

determining boundaries of the three dimensional gaze detection coordinate system based on positions of glints detected on the user eye, positions on the near-eye display device of illuminators for generating the glints and of at least one sensor for detecting the glints further comprising;

determining a position of a center of a cornea of the user eye with respect to the positions of the illuminators and the at least one sensor based on positions of glints detected on a cornea surface of the user eye,

determining a position of the pupil center from image data generated from the at least one sensor further comprising;

identifying a black pixel area as a black pupil area in a number of image samples,

averaging the black pupil areas to adjust for headshake,

performing an ellipse fitting algorithm on the average black pupil area for determining an ellipse representing the pupil, and

determining the center of the pupil by determining the center of the ellipse representing the pupil, and

determining a position of a fixed center of eyeball rotation of the user eye based on the positions of the cornea center and the pupil center; and

determining a gaze vector for the user eye based on the positions of the center of the cornea, the pupil center and fixed center of eyeball rotation.

10. The method of claim 9, wherein the coordinate system includes a depth axis and changes along the depth axis trigger redetermination of the boundaries.

11. The method of claim 9 wherein the coordinate system includes a depth axis and a change along the depth axis is determined based on a change of size of a facial feature in image data generated from the at least one sensor.

12. The method of claim 9 wherein the coordinate system includes a depth axis and changes along the depth axis trigger re-calibration of training data.

13. The method of claim 12, further comprising:

determining a position from the fixed center of eyeball rotation of the user eye with respect to the illuminators and the at least one sensor for determining movement of the display device with respect to the user eye.

14. The method of claim 9, wherein:

determining the position of the center of the cornea of the user eye with respect to the illuminators and at least one sensor based on positions of glints detected on a cornea surface of the user eye further comprises;

generating a first plane including points including a positions of a first illuminator which generates a first glint, a camera pupil center of a detection area of the at least one sensor, and the first glint on the cornea surface;

generating a second plane including points including a position of a second illuminator which generates a second glint, and the camera pupil center of the detection area of the at least one sensor, and the second glint; and

determining the position of the cornea center based on the intersection of the first and the second planes.

15. The method of claim 14, wherein:

determining the position of the fixed center of eyeball rotation of the user eye based on the positions of the cornea center and the pupil center further comprises estimating the position of the fixed center of eyeball rotation of the user eye based on a vector from the position of the pupil center through the position of the cornea center having a length based on length approximations between the positions of the pupil center and the cornea center and between the positions of the cornea center and the center of eyeball rotation.

16. The method of claim 9, wherein:

determining a gaze vector for the user eye based on the positions of the center of the cornea, the pupil center and fixed center of eyeball rotation further comprises;

modeling an optical axis for the user eye as a ray extending from the fixed center of rotation of the eyeball through the cornea and pupil centers;

applying a correction to the modeled optical axis for estimating a visual axis; and

extending the estimated visual axis from the pupil through a near-eye display supported by the near-eye display in front of the user eye into a user field of view.

17. A gaze vector determination system for use in a mixed reality display system comprising:

at least one image generation unit for generating at least one virtual image for display by a near-eye, mixed reality display device;

a set of infra-red illuminators for producing glints for a user eye, each illuminator having a predetermined position on the display device and generating infra-red radiation about a predetermined wavelength;

at least one sensor for detecting the glints reflected from the user eye and generating glint data including intensity data for the glints, the at least one sensor having a predetermined position in relation to one or more illuminator predetermined positions on the display device to detect infra-red reflected radiation about the predetermined wavelength;

a memory for storing software and data including glint data; and

one or more processors communicatively coupled to the at least one sensor to receive the glint data and having access to the memory for storing the glint data, the one or more processors determining one or more glint positions relative to an eye part based on the glint intensity data, the predetermined position of each illuminator in the set of infra-red illuminators, and the at least one predetermined position of the at least one sensor for detecting the glints reflected from the user eye; and

the one or more processors determining a gaze vector for the user eye based on the one or more glint positions relative to an eye part.

18. The system of claim 17 further comprising:

the one or more processors being communicatively coupled to the at least one image generation unit for controlling the at least one image generation unit for displaying the at least one virtual image at different locations in a user field of view viewable in the display device;

the at least one sensor capturing a calibration glint data set of intensity values for each glint during display of the at least one virtual image at each different location;

the one or more processors storing the calibration glint data sets of intensity values as indicators of calibration gaze vectors; and

the one or more processors determining the gaze vector of the user eye based on comparisons of the captured glint data with the calibration glint data sets.

19. The system of claim 17 further comprising wherein the at least one sensor is a position sensitive detector and the one or more processors turns each illuminator on and off in a predetermined sequence for capturing one or more data sets of glint intensity values.

20. The system of claim 17 wherein the glint intensity data comprises one or more data sets of glint intensity values and the number of glints include at least two glints, and the one or more processors determining a gaze vector for the user eye based on the one or more glint positions relative to an eye part further comprises:

the one or more processors determining one or more glint positions relative to an eye part based on the glint intensity data, the respective predetermined position of each illuminator in the set of infra-red illuminators, and the at least one predetermined position of the at least one sensor for detecting the glints reflected from each eye, and

the one or more processors determining one or more glint positions relative to an eye part based on the glint intensity data further comprises determining a reflectivity data value represented by each glint intensity value, and identifying a glint corresponding to the respective glint intensity value as being located on a sclera, an iris portion or a pupil portion based on the reflectivity data value represented by each glint intensity value.