Eye gaze tracker system and method

• US 7,783,077 B2
• Filed: 12/01/2006
• Issued: 08/24/2010
• Est. Priority Date: 12/01/2006
• Status: Active Grant

• Current Assignee: The Boeing Co.
• Original Assignee: The Boeing Co.
• Inventors: Miklos, Luke B., Fesenmaier, Eric C., Steyer, Kevin J.
• Primary Examiner(s): AZARIAN, SEYED H
• Application Number:
  US11/565,886
• Publication Number:
  US 20080130950A1
• Time in Patent Office: 1,362 Days
• Field of Search: 382/100, 382/103, 382/104, 382/106, 382/107, 382/115, 382/117, 382/118, 382/123, 382/168, 382/173, 382/181, 382/199, 382/203, 382/209, 382/232, 382/255, 382/257, 382/274, 382/276, 382/286, 382/291, 382/305, 382/312, 701/49, 351/209, 345/7, 345/8, 434/44
• US Class Current: 382/103
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence

Abstract

The invention relates to systems and methods for tracking the movement of an operator'"'"'s eyes in order to determine the real-time gaze angle of the eyes of the operator. In one embodiment, the invention may utilize an eye camera and a processing device. A frame by frame image of the eye may be captured utilizing the camera and the processing device, and a custom template resembling the operator'"'"'s eye may be created utilizing the image produced by the eye camera. The custom template may be substantially the shape and size of the pupil of the eye of the operator, or may also represent additional features of the eye of the operator, such as the iris, sclera, cornea, or eye-lids. A tracking algorithm may determine a substantially real-time gaze angle of the eye.

Claims

1. A method for tracking movement of an eye of an operator comprising:

providing an eye camera, and a processing device;

capturing a frame by frame image of said eye utilizing said camera and said processing device;

creating a custom template of said eye utilizing said eye camera, wherein said custom template resembles at least a portion of said eye of said operator, and said custom template comprises at least one of a group comprising a shape, a size, and an illumination of said portion of said eye; and

utilizing a tracking algorithm to determine a substantially real-time gaze angle of said eye, wherein said tracking algorithm utilizes data pertaining to both said frame by frame image of said eye and to said custom template.

2. The method of claim 1 further comprising the step of providing a head tracking device.

3. The method of claim 2 further comprising the step of determining a focal point of said eye relative to said head tracking device.

4. The method of claim 1 wherein said custom template resembles at least one of a group comprising the operator'"'"'s pupil, iris, sclera, eye-lids, retina, and cornea.

5. The method of claim 1 further comprising providing an LED, wherein said eye camera and said LED operate in an infra-red spectrum.

6. The method of claim 1 further comprising the step of providing a real-time frame capture card, wherein said frame capture card is utilized to capture said frame by frame image of said eye.

7. The method of claim 1 wherein said frame by frame image of said eye captures movement of said eye.

8. The method of claim 1 wherein said camera is infra-red, and said custom template is created using a dark area in said infrared frame by frame image.

9. The method of claim 1 wherein said custom template is created utilizing image processing.

10. The method of claim 1 further comprising the step of having the operator look at a plurality of calibration points, wherein said calibration points are at pre-determined gaze angles.

11. The method of claim 10 wherein a gaze at which said calibration points are displayed utilizes said pre-determined gaze angles and a determined focal point of the eye relative to a head tracking device.

12. The method of claim 10 further comprising the step of determining a position of said portion of said eye at each of said calibration points.

13. The method of claim 10 further comprising the steps of determining a position of said portion of said eye at each of said calibration points, and creating interpolators that map between said position of said portion of said eye at each of said calibration points to said pre-determined gaze angles.

14. The method of claim 1 wherein said tracking algorithm converts both the custom template and eye image into a frequency domain, multiplies the conjugate of the custom template by the eye image to produce a resultant image, and converts the resultant image into a real domain.

15. The method of claim 1 wherein said tracking algorithm does a correlation calculation at a low resolution of the eye image to determine an approximate location of said portion of said eye, said tracking algorithm does multiple correlation calculations at a high resolution of the eye image by splitting the high resolution eye image into multiple pieces and having parallel correlation calculations of each of said multiple pieces conducted by multiple processing devices, and said tracking algorithm determines a location of the portion in the eye image utilizing the low and high resolution correlation calculations.

16. The method of claim 1 wherein said tracking algorithm determines said substantially real-time gaze angle of said eye based on a determined real-time position of said portion of said eye.

17. The method of claim 16 wherein said determined substantially real-time gaze angle is outputted to at least one of a group comprising a device, a display, and a system.

18. A method for tracking movement of an eye of an operator comprising:

providing a head tracking device, an eye camera, and a processing device;

capturing a frame by frame image of said eye utilizing said camera and said processing device;

determining a focal point of said eye relative to said head tracking device;

having the operator look at a plurality of calibration points, wherein said calibration points are each displayed at gazes which are determined utilizing pre-determined gaze angles for each of the calibration points, and the determined focal point; and

determining a position of a portion of said eye of said operator at each of said calibration points.

19. The method of claim 18 further comprising the step of creating a custom template of said portion of said eye utilizing said eye camera, wherein said custom template resembles said portion of said eye of said operator.

20. The method of claim 18 further comprising the step of utilizing a tracking algorithm to determine a substantially real-time gaze angle of said eye, wherein said tracking algorithm utilizes data pertaining to both said frame by frame image of said eye and to said custom template.

21. The method of claim 18 wherein said step of determining a focal point of said eye comprises having the operator look at multiple images in order to align the user to a certain position.

22. The method of claim 18 further comprising the step of mapping said position of said portion of said eye into gaze angles and interpolating between known gaze angles of the calibration points.

23. The method of claim 18 wherein a determined substantially real-time gaze angle is outputted to at least one of a group comprising a device, a display, and a system.

24. A method for tracking movement of an eye of an operator comprising:

providing an eye camera, and a plurality of processing devices;

capturing a frame by frame image of said eye utilizing said camera and at least one of said processing devices; and

utilizing a tracking algorithm to determine a location of a portion of said eye, wherein said tracking algorithm does a correlation calculation at a low resolution of the eye image to determine an approximate location of said portion of said eye, said tracking algorithm does multiple correlation calculations at a high resolution of the eye image by splitting the high resolution eye image into multiple pieces and having parallel correlation calculations of each of said multiple pieces conducted by said processing devices, and said tracking algorithm determines said location of said portion of said eye in the frame by frame eye image utilizing the low and high resolution correlation calculations.

25. The method of claim 24 further comprising the step of creating a custom template of said portion of said eye utilizing said eye camera, wherein said custom template resembles said portion of said eye of said operator.

26. The method of claim 25 wherein said step of utilizing a tracking algorithm to determine said location of said eye in said frame by frame eye image utilizes data pertaining to both said frame by frame image of said eye and to said custom template.

27. The method of claim 25 wherein said tracking algorithm converts both the custom template and eye image into a frequency domain, multiplies the conjugate of the custom template by the eye image to produce a resultant image, and inverts the resultant image into a real domain.

28. The method of claim 24 further comprising the step of determining a focal point of said eye relative to a head tracking device.

29. The method of claim 28 further comprising the step of having the operator look at a plurality of calibration points, wherein said calibration points are each displayed during calibration at gazes which are determined utilizing pre-determined gaze angles for each of the calibration points, and the determined focal point.

30. The method of claim 29 further comprising the step of determining a position of said portion of said eye of said operator at each of said calibration points.

31. The method of claim 30 further comprising the step of storing said position of said portion of said eye and said pre-determined gaze angles at each of said calibration points for use during interpolation.