Real-time eye detection and tracking under various light conditions

• US 20040005083A1
• Filed: 03/24/2003
• Published: 01/08/2004
• Est. Priority Date: 03/26/2002
• Status: Active Grant

• Current Assignee: Honda Giken Kogyo Kabushiki Kaisha (Honda Motor Company)
• Original Assignee: Honda Giken Kogyo Kabushiki Kaisha (Honda Motor Company)
• Inventors: Fujimura, Kikuo, Zhu, Zhiwei
• Granted Patent: US 7,206,435 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: A61B 5/163 by tracking eye movement, gaze, or pupil change
• CPC Class Codes: A61B 5/18 for vehicle drivers or machine operators
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 7/77 using statistical methods
• CPC Class Codes: G06V 40/19 Sensors therefor

Abstract

System and methods for non-intrusive real-time eye detection and tracking are disclosed. A subject'"'"'s eyes can be detected by using active illumination analysis to generate a difference image of the subject. In the difference image, the bright pupil effect intensifies the appearance of the subject'"'"'s pupils. A component analysis can be used to identify a set of pupil candidates from the difference image. An appearance-based analysis can be applied to the pupil candidates to identify the subject'"'"'s eyes from background noise that can occur under various lighting conditions. After the subject'"'"'s eyes are detected, a multi-stage tracking process is disclosed for detecting real-time eye movements. Further features are disclosed such as an image acquisition apparatus that reduces external illumination interferences.

Claims

1. A method for identifying eyes of a subject, the method comprising the steps of:

generating a difference image of the subject using active illumination analysis;

identifying a set of pupil candidates from the difference image; and

performing an appearance-based analysis with the set of pupil candidates to identify the eyes of the subject from among the pupil candidates.

2. The method of claim 1 wherein the generating step further comprises steps of:

acquiring a first image using an on-axis illuminator; and

acquiring a second image using an off-axis illuminator.

3. The method of claim 2 wherein the first image comprises one of an even and an odd field of a video signal.

4. The method of claim 2 wherein the second image comprises one of an even and an odd field of a video signal.

5. The method of claim 2 wherein at least one of the on-axis illuminator and the off-axis illuminator comprises an infrared emitter having a wavelength band about 40 nm wide.

6. The method of claim 1 wherein the identifying step further comprises the steps of:

removing pixels having an intensity lower than a predetermined threshold from the difference image to identify at least one blob that represents a potential pupil candidate; and

performing a component analysis on the at least one blob to produce the set of pupil candidates.

7. The method of claim 6 wherein the component analysis distinguishes a non-pupil blob from a pupil blob by at least one of size and shape of the at least one blob.

8. The method of claim 1 wherein the performing step further comprises the steps of:

training a support vector machine with training sets having positive and negative images; and

performing pattern matching for a first subject.

9. The method of claim 8 further comprising the steps of:

retraining the support vector machine responsive to the pattern matching performed on the first subject; and

performing pattern matching for a second subject.

10. The method of claim 9 further comprising:

repeating the retraining step and the performing pattern matching for the second subject step until a determined eye identification accuracy is obtained.

11. The method of claim 1 further comprising the steps of:

generating a second difference image of the subject using active illumination; and

performing Kalman filtering on the second difference image to identify eye movement.

12. The method of claim 11 further comprising the steps of:

determining success of the eye movement identification; and

invoking dark pupil image tracking when the eye movement identification is unsuccessful.

13. The method of claim 12 wherein the step of invoking dark pupil image tracking further comprises the steps of:

initializing a target location of the eye movement in a first frame;

deriving weights for each pixel in the target location; and

calculating a predicted location of the eye movement in a second frame.

14. The method of claim 13 further comprising the steps of:

determining whether change between the target location in the first frame and the predicted location in the second frame is smaller than a predetermined threshold.

15. The method of claim 14 further comprising:

repeating the calculating of the predicted location in the second frame when the change exceeds the predetermined threshold.

16. A method for identifying eyes of a subject, the method comprising the steps of:

generating a difference image of the subject using active illumination analysis;

removing pixels having an intensity lower than a predetermined threshold from the difference image to identify at least one blob that represents a potential pupil candidate;

performing a component analysis on the at least one blob to produce a set of pupil candidates; and

performing pattern matching on the set of pupil candidates using a support vector machine with training sets including positive and negative images to identify the eyes of the subject from among the set of pupil candidates.

17. A system for identifying eyes of a subject, the system comprising:

means for generating a difference image of the subject using active illumination analysis;

means for identifying a set of pupil candidates from the difference image; and

means for performing an appearance-based analysis with the set of pupil candidates to identify the eyes of the subject from among the pupil candidates.

18. The system of claim 17 wherein the means for generating further comprises:

means for acquiring a first image using an on-axis illuminator; and

means for acquiring a second image using an off-axis illuminator.

19. The system of claim 18 wherein the first image comprises one of an even and an odd field of a video signal.

20. The system of claim 18 wherein the second image comprises one of an even and an odd field of a video signal.

21. The system of claim 18 wherein at least one of the on-axis illuminator and the off-axis illuminator comprises an infrared emitter having a wavelength band about 40 nm wide.

22. The system of claim 17 wherein the means for identifying further comprises:

means for removing pixels having an intensity lower than a predetermined threshold from the difference image to identify at least one blob that represents a potential pupil candidate; and

means for performing a component analysis on the at least one blob to produce the set of pupil candidates.

23. The system of claim 22 wherein the component analysis distinguishes a non-pupil blob from a pupil blob by at least one of size and shape of the at least one blob.

24. The system of claim 17 wherein the means for performing further comprises:

means for training a support vector machine with training sets having positive and negative images; and

means for performing pattern matching for a first subject.

25. The system of claim 24 further comprising:

means for retraining the support vector machine responsive to the pattern matching performed on the first subject; and

means for performing pattern matching for a second subject.

26. The system of claim 25 further comprising:

means for repeating the retraining step and the performing pattern matching for the second subject step until a determined eye identification accuracy is obtained.

27. The system of claim 17 further comprising:

means for generating a second difference image of the subject using active illumination; and

means for performing Kalman filtering on the second difference image to identify eye movement.

28. The system of claim 27 further comprising:

means for detennining success of the eye movement identification; and

means for invoking dark pupil image tracking when the eye movement identification is unsuccessful.

29. The system of claim 28 wherein the means for invoking dark pupil image tracking further comprises:

means for initializing a target location of the eye movement in a first frame;

means for deriving weights for each pixel in the target location; and

means for calculating a predicted location of the eye movement in a second frame.

30. The system of claim 29 further comprising:

means for determining whether change between the target location in the first frame and the predicted location in the second frame is smaller than a predetermined threshold.

31. The system of claim 30 further comprising:

means for repeating the calculating of the predicted location in the second frame when the change exceeds the predetermined threshold.