3D IMAGE ANALYZER FOR DETERMINING THE GAZE DIRECTION

• US 20160335475A1
• Filed: 07/28/2016
• Published: 11/17/2016
• Est. Priority Date: 02/04/2014
• Status: Active Grant

• Current Assignee: Fraunhofer Gesellschaft Zur Foerderung Der Angewandten Forsching E.V.
• Original Assignee: Fraunhofer Gesellschaft Zur Foerderung Der Angewandten Forsching E.V.
• Inventors: KRENZER, Daniel, HESS, Albrecht, KTAI, Andrs
• Granted Patent: US 10,192,135 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 17/145 Square transforms, e.g. Hadamard, Walsh, Haar, Hough, Slant transforms
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/20008 Globally adaptive
• CPC Class Codes: G06T 2207/20061 Hough transform
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 3/60 Rotation of whole images or parts thereof
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/337 involving reference images or patches
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 7/77 using statistical methods
• CPC Class Codes: G06V 10/955 using specific electronic processors
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A 3D image analyzer for the determination of a gaze direction or a line of sight (having a gaze direction vector and a location vector, which e.g. indicates the pupil midpoint and where the gaze direction vector starts) in a 3D room is configured to receive one first set of image data and a further set of image information, wherein the first image contains a pattern, which displays a three-dimensional object from a first perspective into a first image plane, and wherein the further set contains an image having a pattern, which displays the same three-dimensional object from a further perspective into a further image plane, or wherein the further set has an image information and/or a relation between at least two points in the first image and/or at least a position information. The 3D image analyzer has a position calculator and an alignment calculator and calculates therewith a gaze direction in a 3D room.

Claims

1. A 3D image analyzer for determination of a gaze direction, wherein the 3D image analyzer is configured to receive at least one first set of image data, which is determined on the basis of a first image, and a further set of information, which is determined on the basis of the first image or of a further image, wherein the first image comprises a pattern resulting from the display of a three-dimensional object from a first perspective into a first image plane, and wherein the further set comprises an image with a pattern resulting from the display of the same three-dimensional object from a further perspective into a further image plane, or wherein the further set comprises information which describes a relation between at least one point of the three-dimensional object and the first image plane, wherein the 3D image analyzer comprises the following features:

a position calculator which is configured to calculate a position of the pattern within a three-dimensional room based on the first set, a further set, a further set, which is determined on the basis of the further image, and a geometric relation between the perspectives of the first and the further image or to calculate the position of the pattern within a three-dimensional room based on the first set and a statistically determined relation between at least two characterizing features towards each other in the first image, or to calculate the position of the pattern within the three-dimensional room based on the first set and on a position relation between at least one point of the three-dimensional object and the first image plane; and

an alignment calculator which is configured to calculate at least two possible 3D gaze vectors per image and to determine from these two possible 3D gaze vectors the 3D gaze vector according to which the pattern in the three-dimensional room is aligned, wherein the calculation and determination is based on the first set, the further set and on the calculated position of the pattern.

2. The 3D image analyzer according to claim 1, wherein the further set comprises a further image, and wherein the alignment calculator is configured to calculate two further possible 3D gaze vectors and to compare the two further possible 3D gaze vectors and to determine on the basis of the comparison the 3D gaze vector according to which the pattern within the three-dimensional room is aligned.

3. The 3D image analyzer according to claim 1, wherein the further set of image information comprises information how many pixel are scanned from the sclera displayed in first and/or the further image by the projections, which result from the pupil midpoint in the first and/or further image and the display of the two possible 3D gaze vectors into the image.

4. The 3D image analyzer according to claim 2, wherein the alignment calculator is configured to select from the two possible 3D gaze vectors the 3D gaze vector, according to which the pattern is aligned in the three-dimensional room, wherein in this 3D gaze vector its rear projection into the image based on the pupil midpoint scans less pixel then the rear projection of the other 3D gaze vector.

5. The 3D image analyzer according to claim 1, wherein the alignment calculator is configured to determine a distance respectively between the recognized pupil midpoint and the recognized edge of the eye along the two 3D gaze vectors projected into the image and to select the 3D gaze vector, according to which the pattern is aligned in the three-dimensional room from two possible 3D gaze vectors, wherein the 3D gaze vector is selected, the projection of which into the image there scans the smaller distance between the pupil midpoint and the edge of the eye opening.

6. The 3D image analyzer according to claim 1, wherein the further set of image information comprises an information on the relation between a pupil position within the eye recognized in the first image to a reference pupil position and the two possible 3D gaze vectors.

7. The 3D image analyzer according to claim 6, wherein the alignment calculator is configured to determine a reference position of the eye, which corresponds to the focus of the surface of the displayed eye opening with parallel position of the facial plane towards the camera sensor plane or the calculated pupil midpoint with direct gaze to the camera sensor center and to select the 3D gaze vector according to which the pattern is aligned in the three-dimensional room from the two possible 3D gaze vectors, wherein the 3D gaze vector is selected, the display of which in the image based on the pupil midpoint comprises the greater distance to the reference position.

8. The 3D image analyzer according to claim 1, wherein the statistically evaluated relation comprises a distance between two characteristic facial features, a proportion between the two characteristic facial features and/or a proportion between one characteristic facial feature and one image edge.

9. The 3D image analyzer according to claim 1, wherein the position calculator is configured to detect the two or more characteristic features and to compare their position relation with the previously statistically determined and stored data and to determine therefrom the distance and/or the alignment of the pattern towards the camera.

10. The 3D image analyzer according to claim 1, which is configured to receive a plurality of first and further sets of a plurality of samples.

11. The 3D image analyzer according to claim 10, for which the position calculator is configured to calculate the position of the pattern for a plurality of samples, and

wherein the alignment calculator is configured to determine the 3D gaze vector of the pattern for the plurality of samples, in order to, thus, track the 3D gaze vector.

12. The 3D image analyzer according to claim 1, wherein the pattern is a pupil, an iris, or an ellipsis.

13. The 3D image analyzer according to claim 1, wherein the first and the further set originate from a group comprising the coordinates of a pattern, coordinates of a midpoint of the pattern, geometry parameters of the pattern, coordinates of the midpoint of an ellipsis, a first diameter of the ellipsis—

the long axis—

, a second diameter of the ellipsis—

the short axis—

and an inclination angle of an axis of the ellipsis.

14. The 3D image analyzer according to claim 1, wherein the 3D gaze vector is defined as a vector extending through the midpoint of the pattern along a normal direction based on a surface of an object belonging to the pattern.

15. The 3D image analyzer according to claim 1, wherein the calculation of the position and the 3D vector is based on further information originating from the group comprising information on the optical parameters of the camera lens, a position and alignment of the camera lens, a sensor pixel size and an information on the omission or centralization of several sensor pixel.

16. The 3D image analyzer according to claim 15, wherein the alignment calculator is configured to calculate a first virtual projection plane fort eh first image so that a first virtual optical axis, which is defined as perpendicular to the first virtual projection plane, extends through the midpoint of the pattern, and

in order to align the first virtual projection plane based on the first set of image information.

17. The 3D image analyzer according to claim 15, wherein the alignment calculator is configured to calculate a first virtual projection plane for the first image so that a first virtual optical axis, which is defined as perpendicular to the first virtual projection plane, extends through the midpoint of the pattern and to calculate a second virtual projection plane fort he further image so that a second virtual optical axis, which is defined as perpendicular to the second virtual projection plane, extends through the midpoint of the pattern,

wherein the first virtual optical axis extends through the midpoint of the received pattern in the first virtual projection plane and the second virtual optical axis extends through the midpoint of the received pattern in the second virtual projection plane.

18. The 3D image analyzer according to claim 17, wherein the transformation of the first and/or the second image in the first and/or second virtual projection plane occurs on the basis of the specific position of the pattern and/or on the basis of further information originating from a group comprising information on optical parameters of the camera lens, the lens position, the sensor pixel size and an information on the omission or centralization of several sensor pixel

19. The 3D image analyzer according to claim 17, wherein the alignment calculator is configured to display the pattern, which is displayed by a first plurality of intersection beams through the optic onto a first projection plane for the first perspective and by a second plurality of intersection beams through the optic onto a second projection plane for the second perspective, in the first virtual projection plane by a plurality of virtual intersection beams and in the second virtual projection plane by a second plurality of intersection beams.

20. The 3D image analyzer according to claim 19, wherein the pattern is a distorted pupil or iris or an ellipsis, which can be described by a first and a second set of image data comprising at least a first and second axis as well as an inclination angle of one of the axes of the distorted pupil or iris or ellipsis.

21. The 3D image analyzer according to claim 20, wherein the 3D gaze vector can be described by a set of equations, wherein every equation describes a geometric relation of the respective first or respective further virtual projection planes vis-à

-vis the 3D gaze vector.

22. The 3D image analyzer according to claim 21, wherein for the 3D vector with respect to the first virtual projection plane by a first equation on the basis of the image data of the first set, two possible solutions can be calculated, and wherein for the 3D gaze vector with respect to a further virtual projection plane by a further equation on the basis of the image data of the further set, two possible solutions can be calculated.

23. The 3D image analyzer according to claim 22, wherein the difference between the one solution vector of the first equation and the one solution vector of the second equation is less than the difference between other combinations from the solution vectors of the two equations and wherein the described vectors are selected, wherein the 3D image analyzer is calculated by rated averaging of the two selected vectors.

24. The 3D image analyzer according to claim 23, wherein the alignment calculator is configured to calculate an unambiguous result for the 3D gaze vector by means of an equation system comprising the first and second equation.

25. The 3D image analyzer according to claim 1, wherein the 3D image analyzer is implemented into a processing unit.

26. The 3D image analyzer according to claim 25, wherein the processing unit comprises a selective adaptive data processor, which is configured to receive several sets of values, wherein every set is assigned to a respective sample with the following features:

a processor, which is configured to output plausible sets on the basis of the received sets and wherein an implausible set is replaced by a plausible set and wherein values of an implausible set are replaced by internally determined values.

27. An image analyzing system for the determination of a gaze direction based on a previously detected or tracked pupil or iris, comprising the following features:

at least one Hough path for at least one camera of a monoscopic camera assembly or at least two Hough paths for at least two cameras of a stereoscopic or multi-scopic camera assembly, wherein every Hough path comprises a Hough processor with the following features;

a pre-processor which is configured to receive a plurality of samples respectively comprising an image and to rotate and/or to reflect the image of the respective sample and to output a plurality of versions of the image of the respective sample for each sample; and

a Hough transformation unit which is configured to collect a predetermined searched pattern within the plurality of samples on the basis of the plurality of versions, wherein a characteristic of the Hough transformation unit, which depends on the searched pattern, is adjustable;

a unit for analyzing the collected pattern and for outputting a set of image data which describes a position and/or a geometry of the pattern; and

a 3D image analyzer according to claim 1.

28. A method for the determination of a gaze direction, comprising:

receiving of at least one first set of image data, which is determined on the basis of a first image, and a further set of image data, which is determined on the basis of a further image, wherein the first image displays a pattern of a three-dimensional object from a first perspective into a first image plane and wherein the further set comprises a further image or an information, which describes a relation between at least one point of the three-dimensional object and the first image plane;

calculating a position of the pattern in a three-dimensional room based on the first set, a further set, and a geometric relation between the perspectives of the first and the further image or calculating of the position of the pattern in a three-dimensional room based on a first set and a statistically evaluated relation between at least two characteristic features in the first image or calculating the position of the pattern in a three-dimensional room based on the first set and a position relation between at least one point of the three-dimensional object and the first image plane, and

calculating a 3D gaze vector according to which the pattern is aligned in the three-dimensional room based on the first set and the further set.

29. A non-transitory digital storage medium having stored thereon a computer program for performing a method for the determination of a gaze direction, comprising:

receiving of at least one first set of image data, which is determined on the basis of a first image, and a further set of image data, which is determined on the basis of a further image, wherein the first image displays a pattern of a three-dimensional object from a first perspective into a first image plane and wherein the further set comprises a further image or an information, which describes a relation between at least one point of the three-dimensional object and the first image plane;

calculating a position of the pattern in a three-dimensional room based on the first set, a further set, and a geometric relation between the perspectives of the first and the further image or calculating of the position of the pattern in a three-dimensional room based on a first set and a statistically evaluated relation between at least two characteristic features in the first image or calculating the position of the pattern in a three-dimensional room based on the first set and a position relation between at least one point of the three-dimensional object and the first image plane, and

calculating a 3D gaze vector according to which the pattern is aligned in the three-dimensional room based on the first set and the further set,

when said computer program is run by a computer.

30. A 3D image analyzer for determination of a gaze direction, wherein the 3D image analyzer is configured to receive at least one first set of image data, which is determined on the basis of a first image, and a further set of information, which is determined on the basis of the first image or of a further image, wherein the first image comprises a pattern resulting from the display of a three-dimensional object from a first perspective into a first image plane, and wherein the further set comprises an image with a pattern resulting from the display of the same three-dimensional object from a further perspective into a further image plane, or wherein the further set comprises information which describes a relation between at least one point of the three-dimensional object and the first image plane, wherein the 3D image analyzer comprises the following features:

a position calculator which is configured to calculate a position of the pattern within a three-dimensional room based on the first set, a further set, a further set, which is determined on the basis of the further image, and a geometric relation between the perspectives of the first and the further image or to calculate the position of the pattern within a three-dimensional room based on the first set and a statistically determined relation between at least two characterizing features towards each other in the first image, or to calculate the position of the pattern within the three-dimensional room based on the first set and on a position relation between at least one point of the three-dimensional object and the first image plane; and

an alignment calculator which is configured to calculate at least two possible 3D gaze vectors per image and to determine from these two possible 3D gaze vectors the 3D gaze vector according to which the pattern in the three-dimensional room is aligned, wherein the calculation and the determination is based on the first set, the further set and on the calculated position of the pattern,

wherein

the further set of image information comprises information how many pixel are scanned from the sclera displayed in first and/or the further image by the projections, which result from the pupil midpoint in the first and/or further image and the display of the two possible 3D gaze vectors into the image;

or

the further set comprises a further image, and wherein the alignment calculator is configured to calculate two further possible 3D gaze vectors and to compare the two further possible 3D gaze vectors vectors to the two possible 3D gaze vectors and to determine on the basis of the comparison the 3D gaze vector according to which the pattern within the three-dimensional room is aligned;

wherein the alignment calculator is configured to select from the two possible 3D gaze vectors the 3D gaze vector, according to which the pattern is aligned in the three-dimensional room, wherein in this 3D gaze vector its rear projection into the image based on the pupil midpoint scans less sclera pixels than the rear projection of the other 3D gaze vector;

or

the alignment calculator is configured to determine a distance respectively between the recognized pupil midpoint and a recognized edge of the eye along the two possible 3D gaze vectors projected into the image and to select the 3D gaze vector, according to which the pattern is aligned in the three-dimensional room from the two possible 3D gaze vectors, wherein the 3D gaze vector is selected, the projection of which into the image there scans the smaller distance between the pupil midpoint and the edge of the eye opening;

or

the further set of image information comprises an information on the relation between a pupil position within the eye recognized in the first image to a reference pupil position and the two possible 3D gaze vectors;

or

the statistically evaluated relation comprises a distance between two characteristic facial features, a proportion between the two characteristic facial features and/or a proportion between one characteristic facial feature and one image edge;

or

the position calculator is configured to detect the two or more characteristic features and to compare their position relation with the previously statistically determined and stored data and to determine therefrom the distance and/or the alignment of the pattern towards the camera.

31. A method for the determination of a gaze direction, comprising:

receiving of at least one first set of image data, which is determined on the basis of a first image, and a further set of image data, which is determined on the basis of the first image or of a further image, wherein the first image displays a pattern of a three-dimensional object from a first perspective into a first image plane and wherein the further set comprises a further image or an information, which describes a relation between at least one point of the three-dimensional object and the first image plane;

calculating a position of the pattern in a three-dimensional room based on the first set, a further set, and a geometric relation between the perspectives of the first and the further image or calculating of the position of the pattern in the three-dimensional room based on a first set and a statistically evaluated relation between at least two characteristic features in the first image or calculating the position of the pattern in the three-dimensional room based on the first set and a position relation between at least one point of the three-dimensional object and the first image plane, and

calculating a 3D gaze vector according to which the pattern is aligned in the three-dimensional room based on the first set and the further set;

wherein

the further set of image information comprises information how many pixel are scanned from the sclera displayed in first and/or the further image by the projections, which result from the pupil midpoint in the first and/or further image and the display of the two possible 3D gaze vectors into the image;

or

the further set comprises a further image so as to calculate two further possible 3D gaze vectors and to compare the two further possible 3D gaze vectors to the two possible 3D gaze vectors and to determine on the basis of the comparison the 3D gaze vector according to which the pattern within the three-dimensional room is aligned; and

to select from the two possible 3D gaze vectors the 3D gaze vector, according to which the pattern is aligned in the three-dimensional room, wherein in this 3D gaze vector its rear projection into the image based on the pupil midpoint scans less sclera pixels than the rear projection of the other 3D gaze vector;

or

a distance respectively is determined between the recognized pupil midpoint and a recognized edge of the eye along the two possible 3D gaze vectors projected into the image and the 3D gaze vector, according to which the pattern is aligned in the three-dimensional room is selected from the two possible 3D gaze vectors, wherein the 3D gaze vector is selected, the projection of which into the image there scans the smaller distance between the pupil midpoint and the edge of the eye opening;

or

the further set of image information comprises an information on a relation between a pupil position within the eye recognized in the first image to a reference pupil position and the two possible 3D gaze vectors;

or

the statistically evaluated relation comprises a distance between two characteristic facial features, a proportion between the two characteristic facial features and/or a proportion between one characteristic facial feature and one image edge;

or

the two or more characteristic features are detected and their position relations are compared with the previously statistically determined and stored data and therefrom the distance and/or the alignment of the pattern towards the camera is determined.