Calculation method of line-of-sight direction based on analysis and match of iris contour in human eye image

• US 10,082,868 B2
• Filed: 01/23/2017
• Issued: 09/25/2018
• Est. Priority Date: 04/09/2016
• Status: Active Grant

• Current Assignee: Beihang University
• Original Assignee: Beihang University
• Inventors: Lu, Feng, Chen, Xiaowu, Zhao, Qinping
• Primary Examiner(s): Ky, Kevin
• Application Number:
  US15/413,165
• Publication Number:
  US 20170293354A1
• Time in Patent Office: 610 Days
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction

Abstract

The invention provides a calculation method of line-of-sight direction based on analysis and match of iris contour in human eye image, including: a data driven method, for stable calculation of 3D line-of-sight direction via inputting human eye image to be matched with synthetic data of virtual eyeball appearance; two novel optimization matching criterions of eyeball appearance, which effectively reduce effects of uncontrollable factors, such as image scaling and noise on results; a joint optimization method, for the case of continuously shooting multiple human eye images, to further improve calculation accuracy. One application of the invention is virtual reality and human computer interaction which is under the principle that shooting eye images of a user and calculating line-of-sight direction of user to enable interaction with intelligent system interface or virtual realistic object. The invention can be widely used in training, games and entertainment, video surveillance, medical care and other fields.

Claims

1. A calculation method of line-of-sight direction based on analysis and match of iris contour in human eye image, comprising the following steps:

(1) constructing a sphere eyeball model, traversing eyeball orientations with different physical feasibilities, generating 2D virtual eyeball appearances with different orientations by geometric calculation, and storing all the eyeball orientations and corresponding virtual eyeball appearance data in a dataset for use in specific applications;

(2) during application, firstly shooting a facial image of a user, fixing on a left-eye or right-eye region, pre-treating a human eye image, completing brightness correction and extracting pixels on iris edges in the human eye image;

(3) regarding the shot and pre-treated human eye image and the virtual eyeball appearance data in the dataset, matching the human eye image with the virtual eyeball appearance data via a matching optimization algorithm of the human eye image and the virtual eyeball appearance data, wherein a matching result determines an orientation and a position of the eyeball which best match with the human eye image; and

(4) regarding continuously shot human eye images, further conducting joint optimization on the basis of the eyeball appearance matching in step (3), on the condition that a central position of the eyeball remains unchanged, or that the human eye images have been aligned, and precisely and simultaneously calculating 3D line-of-sight direction corresponding to each image;

wherein a method for the joint optimization in step (4), and the method for accurately calculating the 3D line-of-sight direction corresponding to each image simultaneously are as follows, under the assumption that central position of the eyeball remains unchanged or has been aligned when the images shot, conducting the matching of the human eye images with the virtual eyeball appearance in step (3), and calculating the eyeball orientation corresponding to each human eye image and the coordinates of the central position of the eyeball;

excluding the central coordinates of the eyeball with obvious deviation therein, and conducting weighting calculation of standard coordinates of the eyeball center using the remaining coordinates; and

individually carrying on with the optimization in step (3), while adding one optimization constraint, namely, coinciding the matched eyeball central coordinates as much as possible with the standard eyeball central coordinates, and updating the calculation results as the eyeball orientations in the human eye images, namely, the final results of the 3D line-of-sight direction.

2. The calculation method of line-of-sight direction based on analysis and match of iris contour in human eye image according to claim 1, wherein the virtual eyeball appearance data in step (1) are generated as follows, firstly, establishing a 3D sphere model of the eyeball, and adding important elements of a circular iris contour on the surface of the 3D sphere model;

traversing different physically-feasible eyeball orientations, namely, different rotation angles around a horizontal axis and a vertical axis respectively, for each of the rotation angles, projecting a corresponding 3D virtual eyeball appearance to a 2D plane, namely, to a position just in front of the corresponding eyeball, and recording projected iris contour coordinates, eyeball central coordinates and 2D eyeball appearance data of the iris central coordinates; and

storing all the eyeball orientations and the virtual eyeball appearance data corresponding to the eyeball orientations in the dataset.

3. The calculation method of line-of-sight direction based on analysis and match of iris contour in human eye image according to claim 1, wherein the matching optimization algorithm of the human eye image and the virtual eyeball appearance data in step (3) is as follows, matching parameters to be calculated are relative translation amount of the human eye image and the virtual eyeball appearance in a 2D image domain, and the eyeball orientation corresponding to the virtual eyeball appearance, and optimizing matching degree of the human eye image and the virtual eyeball appearance via finding the best values of the two matching parameters, so as to realize matching of the human eye image and the virtual eyeball appearance data.

4. The calculation method of line-of-sight direction based on analysis and match of iris contour in human eye image according to claim 3, wherein the matching degree is calculated by the following functional measurements:

(31) measurement of circular symmetry;

pixels in the proximity of the virtual eyeball iris contour in the human eye image have better circular symmetry, when the matching tends to become ideal; and

(32) measurement of iris contour matching degree;

calculating distances between pixels on the iris edge in the human eye image and the virtual eyeball iris contour, when the matching tends to become ideal, and the distances tend to become equal.

5. The calculation method of line-of-sight direction based on analysis and match of iris contour in human eye image according to claim 4, wherein the measurement of the circular symmetry in step (31) is conducted as follows, coordinates of the iris contour of the virtual eyeball appearance and coordinates of the central position of the iris corresponding to any one group of matching parameter may be determined;

overlapping these coordinates onto the human eye image, via continuously sampling values of pixels on the human eye image, along both the positive and negative directions indicated by lines connecting the iris center of the virtual eyeball appearance and points on the iris contour, while taking the points as reference, so as to obtain 1D pixel column vectors, and the sampling region is proportional to the distances between the points on the iris contour of the virtual eyeball appearance and the iris center of the virtual eyeball appearance;

sampling via traversing the points on the iris contour of the virtual eyeball appearance, and combining all the obtained 1D column vectors into a 2D matrix; and

finally, calculating distribution consistency of each column of numeric of the matrix or a gradient matrix of the matrix in the vertical direction, the consistency may be measured in a matrix kernel function, the correlation coefficient of each column, the concentration of singular values, and the higher the consistency, the better the circular symmetry.

6. The calculation method of line-of-sight direction based on analysis and match of iris contour in human eye image according to claim 4, wherein the measurement of the iris contour matching degree in step (32) is conducted as follows, regarding any one group of matching parameters, determining coordinates of the iris contour of the virtual eyeball appearance corresponding to the matching parameters of the group;

traversing the pixels extracted from the iris edge of the human eye image, and calculating the distances between the edge pixels and the iris contour of the virtual eyeball appearance; and

reviewing distribution of the distances, in which the less the number of the distances which are significantly different from the others, the better the measurement result of the iris contour matching degree.