GEOMETRIC CALIBRATION OF HEAD-WORN MULTI-CAMERA EYE TRACKING SYSTEM

• US 20120050493A1
• Filed: 08/23/2011
• Published: 03/01/2012
• Est. Priority Date: 08/24/2010
• Status: Active Grant

• Current Assignee: Siemens Medical Solutions USA Incorporated (Siemens AG)
• Original Assignee: Siemens Corp. (Siemens AG)
• Inventors: Genc, Yakup, Ernst, Jan
• Granted Patent: US 8,957,948 B2
• Time in Patent Office: Days
• US Class Current: 348/47
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes

Abstract

A system and methods for geometric calibration and use of head-worn multi-camera eye tracking system are provided. The system calibrates the internal and external parameters of the cameras as well as the parameters of a compact geometric model of the human eye. An exo-camera determines a relative pose with regard to a computer monitor and endo-cameras determine the eyeball position and orientation. An endo-exo calibration and an endo-eye calibration each with an optimization step are included. A homography and a joint optimization step are also provided. A single point calibration re-establishes calibration. The system in operational mode determines a user'"'"'s gaze direction to perform a computer input function.

Claims

1. A method for calibrating a frame worn by a wearer containing an exo-camera directed to a display and a first and a second endo-camera, each directed to an eye of the wearer, comprising:

a processor collecting data related to a projection of an image on the display with a known set of 3D coordinates in coordinates of the exo-camera and in coordinates of the endo-camera as an exo-endo pair;

the processor optimizing a relative pose of the exo-endo pair as a set of rotation and translation parameters which minimizes an error criterion;

the processor estimating parameters of a geometric model of the eye while it focuses on a known location in the 3D screen space; and

the processor minimizing a gaze projection error on the display with regard to a known ground truth.

2. The method of claim 1, wherein a first exo-camera and endo-camera pair is formed for the first endo-camera and a second exo-camera and endo-camera pair is formed for the second endo-camera.

3. The method of claim 1, wherein the processor determines a relative transformation between an endo-camera coordinate system and an exo-camera coordinate system.

4. The method of claim 3, wherein the relative transformation is expressed as:

p^x=R^exp^e+t^ex

wherein

R^exε

SO(3) is a rotation matrix, wherein SO(3) is a rotation group,

t^exε

R³ is a translation vector,

p^xε

R³ is a vector of points in the exo camera coordinate system, and

p^eε

R³ is a vector of points in the endo camera coordinate system.

5. The method of claim 4, wherein:

calibration grids G^e, G^x are placed around the endo-exo camera system such that G^x is visible in the exo camera image and G^e is visible in the endo camera image.

6. The method of claim 4, wherein for each image establishing an exo-camera and endo-camera pair external pose matrices T_n^eε

R⁴×

R⁴ and T_n^xε

R⁴×

R⁴ are estimated from marked projected calibration points, wherein;

T_n^e is a endo-camera external pose matrix for image pair n, and

T_n^x is a exo-camera external pose matrix for image pair n.

7. The method of claim 6, wherein the error criterion is determined by an optimization of an expression

8. The method of claim 1, wherein:

parameters of the geometric model of the eye include an orientation of the eye and a position of a center location of the eye.

9. The method of claim 1, wherein

the processor estimates the parameters of the of the geometric model of the eye by minimizing a projection error of an estimated intersection point of the direction of gaze with the display relative to an actual intersection point.

10. The method of claim 1, further comprising:

minimizing a residual error between an estimated location on a display and an actual location on a display by estimating a homography.

11. The method of claim 10, wherein the estimated homography is expressed as:

d=H{tilde over (d)},

wherein;

d is the actual location on the display,

{tilde over (d)} is the estimated location on the display, and

H is a matrix that determines a homography.

12. The method of claim 9, wherein the minimizing of the projection error generates estimated parameters that calibrate the frame relative to the display.

13. The method of claim 12, wherein the estimated parameters are {tilde over (T)}^ex, {tilde over (c)} and {tilde over (r)}, wherein

{tilde over (T)}^ex is an estimation of a transformation matrix for homogeneous coordinates;

{tilde over (c)} is an estimate of a center of the eye; and

{tilde over (r)} is an estimate of a radius of the eye.

14. The method of claim 1, further comprising:

the wearer after calibration looking at the screen and the processor determining from a post calibration exo-camera image and endo-camera image pair a coordinate on the display.

15. The method of claim 14, wherein a gaze of the wearer is applied to provide a command to a computing device.

16. The method of claim 1, further comprising:

the processor determining that re-calibration is needed based on a performance criterion

the processor determining an endo and an exo camera pair of a user'"'"'s gaze to a single calibration point;

the processor determining a reprojection of the user'"'"'s gaze to the single calibration point; and

the processor determining an correction vector based on coordinates of the single calibration point and coordinates of the reprojection of the user'"'"'s gaze to the single calibration point.

17. The method of claim 16, wherein the processor further calibrates the frame in accordance with the correction vector.

18. A system to calibrate a frame worn by a wearer containing an exo-camera directed to a display and a first and a second endo-camera, each directed to an eye of the wearer, comprising:

the frame worn by the wearer;

a processor enabled to execute instructions to perform the steps;

collecting data related to a projection of an image on the display with a known set of 3D coordinates in coordinates of the exo-camera and in coordinates of the endo-camera as an exo-endo pair;

optimizing a relative pose of the exo-endo pair as a set of rotation and translation parameters which minimizes an error criterion;

estimating parameters of a geometric model of the eye while it focuses on a known location in the 3D screen space; and

minimizing a gaze projection error on the display with regard to a known ground truth.

19. The system of claim 18, wherein the processor is enabled to execute instructions to perform a single calibration point alignment.