Systems and methods for position estimation

• US 8,878,749 B1
• Filed: 09/15/2012
• Issued: 11/04/2014
• Est. Priority Date: 01/06/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Wu, Bo, Zhao, Yong, Starner, Thad Eugene, Raffle, Hayes Solos
• Primary Examiner(s): ABDULSELAM, ABBAS I
• Application Number:
  US13/620,904
• Time in Patent Office: 780 Days
• Field of Search: 345/8, 359/13, 359/630, 351/208, 351/209, 351/210
• US Class Current: 345/8
• CPC Class Codes: A61B 3/00 Apparatus for testing the eyes; Instruments for examining the eyes eye inspection using ultrasonic, sonic or infrasonic waves A61B8/10; devices for treatment of the eyes A61F9/00; exercisers for the eyes A61H5/00; optical systems in general G02B
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: G01S 17/06 Systems determining position data of a target
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G03B 2213/025 Sightline detection
• CPC Class Codes: G03B 29/00 Combinations of cameras, projectors or photographic printing apparatus with non-photographic non-optical apparatus, e.g. clocks or weapons; Cameras having the shape of other objects combinations with flash apparatus G03B15/03
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10152 Varying illumination
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 3/003 to produce spatial visual effects
• CPC Class Codes: G09G 5/00 : Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T
• CPC Class Codes: H04N 23/00 : Cameras or camera modules comprising electronic image sensors; Control thereof

Abstract

Systems and methods are described for improving gaze estimation based on glints. An example method may be executable to receive glint-location data indicating a location for each of one or more glints that are reflected from an eye. The glint-location data may be captured by an imaging device that is attached to a head-mountable display (HMD). The method may be further executable to determine a plurality of normalized pupil locations comprising at least a first and a second normalized pupil location. Each normalized pupil location may be determined based on the glint-location data for a different combinatoric subset of the one or more glints. The method may be further executable to determine a regression model to fit the plurality of normalized pupil locations to an expected gaze location, and perform a gaze-tracking process based on the regression model.

Claims

1. A method comprising:

receiving glint-location data indicating a location for each of one or more glints that are reflected from an eye, wherein the glint-location data is captured by an imaging device that is associated with a head-mountable display (HMD);

determining a plurality of normalized pupil locations comprising at least a first and a second normalized pupil location, wherein each normalized pupil location is determined based on the glint-location data for a different combinatoric subset of the one or more glints;

determining a regression model to fit the plurality of normalized pupil locations to an expected gaze location; and

performing, by a computing device, a gaze-tracking process based on the regression model.

2. The method of claim 1, wherein the one or more glints include reflections of an infrared light on a corneal surface.

3. The method of claim 2, further comprising providing the infrared light in a clockwise manner.

4. The method of claim 2, further comprising providing the infrared light in a counterclockwise manner.

5. The method of claim 1, wherein the glint-location data includes location data for four glints that are positioned to form a square.

6. The method of claim 1, wherein determining the plurality of normalized pupil locations further comprises:

determining a relative vector from the first normalized pupil location to the glint-location data using a translation normalization.

7. The method of claim 1, wherein determining the plurality of normalized pupil locations further comprises:

determining a coordinate framework with two glints from the one or more glints; and

determining the first normalized pupil location on the coordinate framework using a similarity normalization of the two glints.

8. The method of claim 1, wherein determining the plurality of normalized pupil locations further comprises:

determining a coordinate framework with three glints from the one or more glints; and

determining the first normalized pupil location on the coordinate framework using an affinity normalization of the three glints.

9. The method of claim 1, wherein determining the plurality of normalized pupil locations further comprises:

determining a coordinate framework with four glints from the one or more glints; and

determining the first normalized pupil location on the coordinate framework using a homography normalization of the four glints.

10. The method of claim 1, wherein the regression model is a polynomial regression model.

11. A system comprising:

at least one processor;

a non-transitory computer-readable medium; and

program instructions stored on the non-transitory computer-readable medium and executable by the at least one processor to cause the system to;

receive glint-location data indicating a location for each of one or more glints that are reflected from an eye, wherein the glint-location data is captured by an imaging device that is associated with a head-mountable display (HMD);

determine a plurality of normalized pupil locations comprising at least a first and a second normalized pupil location, wherein each normalized pupil location is determined based on the glint-location data for a different combinatoric subset of the one or more glints;

determine a regression model to fit the plurality of normalized pupil locations to an expected gaze location; and

perform a gaze-tracking process based on the regression model.

12. The system of claim 11, further comprising program instructions stored on the non-transitory computer-readable medium and executable by the at least one processor to cause the system to:

determine an estimation error associated with at least the first and the second normalized pupil location; and

determine the estimated gaze location by taking a median of at least the first and the second normalized pupil locations having the estimation error less than a threshold.

13. The system of claim 12, wherein determining the estimation error further comprises:

determining an average estimation error associated the first normalized pupil location;

comparing the average estimation error to a threshold; and

eliminating the first normalized pupil location when the average estimation error exceeds the threshold.

14. The system of claim 12, wherein determining the estimation error further comprises:

determining a maximum estimation error associated with the first normalized pupil location;

comparing the maximum estimation error to a threshold; and

eliminating the first normalized pupil location when the maximum estimation error exceeds the threshold.

15. A non-transitory computer-readable medium having stored thereon instructions executable by a computing device having at least one processor to cause the computing device to perform functions comprising:

receiving glint-location data indicating a location for each of one or more glints that are reflected from an eye, wherein the glint-location data is captured by an imaging device that is attached to a head-mountable display (HMD);

determining a plurality of normalized pupil locations comprising at least a first and a second normalized pupil location, wherein each normalized pupil location is determined based on the glint-location data for a different combinatoric subset of the one or more glints;

determining a regression model to fit the plurality of normalized pupil locations to an expected gaze location; and

performing, by a computing device, a gaze-tracking process based on the regression model.

16. The non-transitory computer readable medium of claim 15, wherein the regression model is a polynomial regression model.

17. The non-transitory computer readable medium of claim 15, wherein the glint-location data includes location data for four glints.

18. The non-transitory computer readable medium of claim 15, wherein the instructions are further executable by the computing device to cause the computing device to perform functions comprising:

determining an average estimation error associated with the second normalized pupil location;

comparing the average estimation error to a threshold; and

eliminating the second normalized pupil location when the average estimation error associated with the second normalized pupil location exceeds the threshold.

19. The non-transitory computer readable medium of claim 15, wherein the instructions are further executable by the computing device to cause the computing device to perform functions comprising:

determining an estimation error associated with at least the first and the second normalized pupil location; and

determining the estimated gaze location by taking a median of at least the first and the second normalized pupil locations having the estimation error less than a threshold.

20. The non-transitory computer readable medium of claim 19, wherein the instructions are further executable by the computing device to cause the computing device to perform functions comprising:

determining an average estimation error associated the first normalized pupil location;

comparing the average estimation error to a threshold; and

eliminating the first normalized pupil location when the average estimation error exceeds the threshold.