Method and circuit arrangement for recognising and tracking eyes of several observers in real time

• US 7,950,802 B2
• Filed: 08/16/2006
• Issued: 05/31/2011
• Est. Priority Date: 08/17/2005
• Status: Active Grant

• Current Assignee: SeeReal Technologies GmbH (SeeReal Technologies S.A.)
• Original Assignee: SeeReal Technologies GmbH (SeeReal Technologies S.A.)
• Inventors: Zschau, Enrico, Schwerdtner, Alexander
• Primary Examiner(s): Schwartz; Jordan M.
• Assistant Examiner(s): JONES, JAMES
• Application Number:
  US12/064,078
• Publication Number:
  US 20080231805A1
• Time in Patent Office: 1,749 Days
• Field of Search: 351/210
• US Class Current: 351/210
• CPC Class Codes: G03H 2226/05 Means for tracking the observer
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: H04N 13/368 for two or more viewers
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes

Abstract

The invention relates to a method and to a circuit arrangement for recognising and for tracking, in a contact-free manner, eye positions of several users in real time. The input data comprises a sequence of digital video frames. Said method comprises the following steps: combining a face-finder-instance which is used to examine faces, an eye-finder-instance which is used to examine eye areas, and an eye-tracker-instance which is used to recognise and track eye reference points. The aim of the invention is to convert the eye positions within a hierarchical outlet of the instance to the target, which successively restricts the dataset, which is to be processed, emerging from the dataset of the entire video frame (VF) in order to form a face target area (GZ) and subsequently an eye target area (AZ). Also, an instance or a group of instances, which run in a parallel manner, are carried out, respectively, on a calculating unit thereof.

Claims

1. Method for detecting and tracking eye reference point positions of multiple observers, wherein the eye reference point positions are the positions of the pupils and/or the corners of the eyes, in image data of video frames, which are acquired by at least one image sensor, the method comprising:

face finder instance for detecting face positions in video frames, said face finder instance extracting a much smaller sub-region from the total video frame as target face region for each face in order to transmit information of said target face region to at least one subsequent

eye finder instance for detecting eye positions, said eye finder instance extracting a much smaller sub-region from each target face region as target eye region in order to transmit information of said target eye region to at least one subsequent

eye tracker instance for tracking the eye positions, said eye tracker instance detecting the eye reference point positions in the target eye region in the current video frame and in the subsequent video frames and generating information for a tracking device,

wherein each of the instances, or the face finder instance combined with one of the eye finder instance and the eye tracker instance, or a combination of the face finder instance and the eye finder instance and the eye tracker instance, are executed each in a dedicated computing unit proceeding in parallel with each other.

2. Method according to claim 1, wherein the coordinates of the reference point positions are horizontal and vertical positions.

3. Method according to claim 2, wherein the depth coordinates of the eyes are determined based on two-dimensional reference point positions of multiple image sensors.

4. Method according to claim 3, wherein the face finder instance and a combination of the eye finder instance and the eye tracker instance or a combination of the face finder instance and the eye finder instance and the eye tracker instance are executed in respectively dedicated computing units.

5. Method according to claim 3, wherein for each detected face an eye finder instance and an eye tracker instance are initialised and executed in at least one dedicated computing unit.

6. Method according to claim 3, wherein a valuation order is assigned to the observers and/or the resulting target eye positions, said order being used for the assignment of computing units.

7. Method according to claim 3, wherein one or several face finder instances are executed permanently and initialise the respectively assigned eye finder/eye tracker instances.

8. Method according to claim 1 with a control instance which determines whether a target face region found by the face finder instance results from an already tracked face or from a newly detected face, which assigns the eye tracker instance or a combination of the eye finder instance and the eye tracker instance to the available computing units, initialises them and which synchronises the execution of all instances.

9. Method according to claim 1, wherein image data is acquired by multiple image sensors, wherein an eye finder instance, an eye tracker instance or a combination of the eye finder instance and the eye tracker instance is executed per observer and image sensor in dedicated computing units, and wherein partial and/or final results are exchanged among and processed by the instances respectively related to an observer.

10. Method according to claim 9 wherein, knowing the positions of the image sensors and of the observers as well as parameters of the image sensors, the partial and/or final results of the instances of an image sensor are transformed for the instances of another image sensor and adapted to its perspective.

11. Circuit arrangement for detecting and tracking eye reference point positions of multiple observers, wherein the eye reference point positions are the positions of the pupils and/or the corners of the eyes, in image data of video frames with multiple communicating computing units with a face finder instance, an eye finder instance and an eye tracker instance, wherein the

face finder instance serves to detect the face positions in video frames and extracts a much smaller sub-region from the total video frame as target face region for each face in order to transmit information of said target face region to at least one subsequent

eye finder instance for detecting eye positions, said eye finder instance extracting a much smaller sub-region from each target face region as target eye region in order to transmit information of said target eye region to at least one subsequent

eye tracker instance for tracking the eye positions, said eye finder instance detecting the eye reference point positions in the target eye region in the current video frame and in the subsequent video frames and generating information for a tracking device,

wherein each of the instances, or the face finder instance combined with one of the eye finder instance and the eye tracker instance, or a combination of the face finder instance and the eye finder instance and the eye tracker instance, are executed each in a dedicated computing unit proceeding in parallel with each other.

12. Circuit arrangement according to claim 11 with a separate computing means for the scaling or correction of the image data as regards scaling, gamma correction, brightness control or suchlike.

13. Circuit arrangement according to claim 11 with a computing means which calculates the depth coordinates from the reference points determined from at least two image sensors.

14. Circuit arrangement according to claim 11, wherein at least one of the computing units is implemented as CPU, DSP or suchlike.