Eye tracking apparatus and method employing grayscale threshold values

• US 5,481,622 A
• Filed: 03/01/1994
• Issued: 01/02/1996
• Est. Priority Date: 03/01/1994
• Status: Expired due to Fees

• Current Assignee: Rensselaer Polytechnic Institute
• Original Assignee: Rensselaer Polytechnic Institute
• Inventors: Gerhardt, Lester A., Sabolcik, Ross M.
• Primary Examiner(s): Boudreau, Leo H.
• Assistant Examiner(s): Johns, Andrew W.
• Application Number:
  US08/204,008
• Time in Patent Office: 672 Days
• Field of Search: 382/1, 382/9, 382/48, 382/100, 382/103, 382/117, 382/171, 382/173, 382/291, 348/78, 345/8, 345/157, 345/158, 364/709.1, 364/709.11, 351/206, 351/209, 351/210, 351/245
• US Class Current: 382/103
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: A61F 2009/0043 by supporting the instrument on the patient's head, e.g. head bands
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 40/19 Sensors therefor

Abstract

An eye-tracking system determines the position of a user'"'"'s pupil and maps this position into a point of regard of the user on an interface device, such as a display screen, or other real-world object by a system comprising a camera for acquiring a video image of the pupil; a frame grabber coupled to the camera for accepting and converting analog video data from the camera to digital pixel data; a computer coupled to the frame grabber for processing the digital pixel data to substantially determine the position of the pupil; a display screen coupled to the computer; and a support connected to the camera and display screen for fixing the relative physical positions thereof relative to the user'"'"'s pupil. The processing performed by the computer may include the selection of a first pixel intensity threshold for the segmentation of the digital pixel data into first and second groups, where the total pixel area of the first group is selected to be substantially equal to a pre-determined value expected to correspond to the area of a user'"'"'s pupil. The system may be calibrated by the user'"'"'s following a cursor on the display screen while the system measures the pupil position for known locations of the cursor.

Claims

1. A system for determining a point of regard, said system comprising:

camera means for acquiring a video image of a first vision means, said first vision means comprising a pupil, a position of said pupil corresponding to said point of regard, and said video image comprising a pupil image;

frame grabber means, coupled to said camera means, for accepting video data corresponding to said video image from said camera means and converting said video data to digital pixel data of a given dynamic range so as to provide said digital pixel data as a grayscale mapping of the video image;

computer means, coupled to said frame grabber means, for processing said digital pixel data to substantially determine the position of said pupil wherein said computer means includes;

memory means for providing an expected area value for the area of said pupil within said video image;

means for determining a grayscale threshold value in accordance with said expected area value and grayscale distribution of said digital pixel data;

means for segregating said digital pixel data into two separate binary groups as first and second groups using said grayscale threshold value as a discriminator; and

means for substantially determining the position of said pupil in accordance with the segregated data;

feedback means, coupled to said computer means, for accepting data corresponding to the determined position of said pupil from said computer means and feeding back to said first vision means feedback information representative of the determined position; and

support means, connected to said camera means and said feedback means, for fixing the relative physical positions of said camera means and said feedback means.

2. The system of claim 1 wherein said determining means of said computer means further comprises:

means for grouping contiguously related individual pixels from one of said first group or said second group into a first set of representative pixel blobs; and

means for selecting from said first set of pixel blobs a pixel blob corresponding to said pupil image.

3. The system of claim 2 wherein said feedback means is a display screen;

and

said computer means further comprises;

means for determining the position of said pupil image in an image coordinate space; and

means for mapping the position of said pupil image in the image coordinate space into a position in a display screen coordinate space.

4. The system of claim 3 further comprising a calibration means for determining a mapping function between the image coordinate space and the display screen coordinate space, said calibration means comprising:

means for placing a cursor at a pre-determined position on said display screen;

means for orienting said point of regard to substantially coincide with said pre-determined position;

means for determining the position of the pupil image of said vision means in the image coordinate space;

means for repeating said steps of placing a cursor, orienting said point of regard, and determining the pupil image position a plurality of times to generate a plurality of cursor position-pupil image position data pairs; and

means for fitting a curve to said data pairs to determine said mapping function.

5. The system of claim 3 wherein said position of said pupil image in image coordinate space is determined based on the centroid of said pixel blob selected as corresponding to said pupil image.

6. The system of claim 3 wherein said support means substantially fixes the relative physical position of said vision means relative to said camera and display screen.

7. The system of claim 2 wherein said computer means in selecting said pixel blob corresponding to said pupil image further comprises:

means for calculating a statistic for each pixel blob of said first set;

means for comparing said statistic of each pixel blob with an expected value corresponding to said pupil image to calculate a difference error; and

means for selecting said pixel blob corresponding to said pupil image where said difference error is less than a pre-determined tolerance.

8. The system of claim 7 wherein the computer means further comprises:

a counter that is incremented after each failure to select a pixel blob corresponding to said pupil image for the current video image being processed by said computer means; and

adjustment means for relaxing said tolerance by a pre-determined quantity after said counter reaches a pre-determined upper limit.

9. The system of claim 8 wherein:

said counter is decremented after each successful selection of a pixel blob corresponding to said pupil image for the current video image being processed by said computer means; and

said adjustment means tightens said tolerance to a pre-determined baseline value after said counter reaches a pre-determined lower limit.

10. The system of claim 7 wherein said statistic for each blob is an area, length-to-width ratio, or a centroid.

11. The system of claim 7 wherein said expected value corresponding to said pupil image is a running average of a plurality of statistics calculated for previously-selected pixel blobs corresponding to pupil images for previous video images.

12. The system of claim 2 further comprising at least one light source mounted on said support means, each light source illuminating said vision means and creating an associated radiation intensity highlight on said vision means.

13. The system of claim 12 wherein said computer means in processing said digital pixel data further comprises:

means for selecting a second pixel intensity threshold, greater in intensity than said grayscale threshold value, for the segmentation of said pixel data into third and fourth groups, said second pixel intensity threshold selected so that the total pixel area of said fourth group is substantially equal to a pre-determined expected area for all highlights of said light sources illuminating said vision means;

means for grouping contiguously related individual pixels from said fourth group into a second set of respective pixel blobs;

means for selecting from said second set of pixel blobs a pixel blob corresponding to a first highlight; and

means for comparing the relative positions of said pixel blob corresponding to said pupil image and said pixel blob corresponding to said first highlight to determine said point of regard.

14. The system of claim 13 wherein said feedback means is a display screen and said computer means in processing said pixel data further comprises:

means for determining the relative position of said pupil image relative to said first highlight in an image coordinate space; and

means for mapping said relative position of said pupil image in the image coordinate space into a point of regard in a display screen coordinate space.

15. The system of claim 14 further comprising a second light source mounted on said support means, said second light source illuminating said vision means and creating a second radiation intensity highlight on said vision means.

16. The system of claim 15 wherein the position of said pupil image relative to said second highlight is determined by said computer means.

17. The system of claim 1 wherein said computer means processes data corresponding to a plurality of video images to substantially determine the position of said pupil.

18. An eye-tracking system for determining a point of regard on a display, said eye-tracking system comprising:

a display screen providing said display and displaying information corresponding to said point of regard;

a camera for acquiring a video image of an eye having a pupil, the position of said pupil corresponding to said point of regard, and said video image comprising a pupil image;

a frame grabber, coupled to said camera, for accepting video data corresponding to said video image from said camera and converting said video data to digital pixel data of a given dynamic range so as to provide said digital pixel data as a grayscale mapping of said video image;

a computer, coupled between said frame grabber and said display screen, for processing said digital pixel data to substantially determine said point of regard on said display screen wherein said computer includes;

memory means for providing an expected area value for the area of said pupil within said video image;

means for determining a grayscale threshold value in accordance with said expected area value and grayscale distribution of said digital pixel data;

means for segregating said digital pixel data into two separate binary groups as first and second groups using said grayscale threshold value as a discriminator; and

means for substantially determining the position of said pupil in accordance with the segregated data; and

a support, connected to said camera and said display screen, for substantially fixing the physical positions of said camera means and said display screen relative to said eye.

19. The eye-tracking system of claim 18 wherein said computer in processing said digital pixel data further comprises:

means for grouping contiguously related pixels from said first group into a first set of respective pixel blobs;

means for selecting from said first set a pixel blob corresponding to said pupil image;

means for determining the position of said pupil image by a calculated value based on a property of said selected pixel blob; and

means for mapping the position of said pupil image in an image coordinate space into a position in a display screen coordinate space.

20. The eye-tracking system of claim 19 wherein said computer means in selecting the pixel blob from said first set further comprises:

means for calculating a statistic for each pixel blob in said first set;

means for comparing said statistic for each pixel blob in said first set with an expected value corresponding to said pupil image to calculate a difference error; and

means for selecting from said first set the pixel blob corresponding to said pupil image where said difference error is less than a pre-determined tolerance.

21. The eye-tracking system of claim 20 further comprising at least a first light source mounted on said support, said first light source illuminating said eye and creating a radiation intensity highlight on the corneal surface of said eye.

22. The eye-tracking system of claim 21 wherein said computer in processing said digital pixel data further comprises:

means for selecting a second pixel intensity threshold, greater in intensity than said grayscale threshold valve, for the segmentation of said pixel data into third and fourth groups, said second pixel intensity threshold selected so that the total pixel area of said fourth group is substantially equal to a pre-determined expected total image area corresponding to the highlight of said first light source illuminating said eye;

means for grouping contiguously related individual pixels from said fourth group into a second set of respective pixel blobs;

means for selecting from said second set a pixel blob corresponding to a first highlight image of said first light source; and

means for comparing the relative positions of said pixel blob corresponding to said pupil image and said pixel blob corresponding to said first highlight image to determine said point of regard.

23. The system of claim 22 wherein said computer in processing said digital pixel data further comprises:

means for determining the relative position of said pupil image relative to said first highlight image in the image coordinate space; and

means for mapping said relative position of said pupil image in the image coordinate space into a point of regard in the display screen coordinate space.

24. The system of claim 23 further comprising a second light source mounted on said support means, said second light source illuminating said eye and creating a second radiation intensity highlight on said eye, wherein the position of said pupil image relative to a second highlight image corresponding to said second light source is determined by said computer.

25. The system of claim 22 wherein said first light source is a light-emitting diode.

26. The system of claim 22 further comprising a speech synthesizer coupled to said computer.

27. The system of claim 20 wherein all components of said system including a power supply can be carried on one'"'"'s person.

28. The system of claim 20 wherein said camera and said display screen are both directed to the same eye.

29. In a vision-tracking system, a tracking method for determining a point of regard, said method comprising the steps of:

acquiring video data from a camera corresponding to a video image of a vision means, said vision means comprising a pupil, the position of said pupil corresponding to said point of regard, and said video image comprising a pupil image;

converting said video data to digital pixel data corresponding to said video image using an analog-to-digital interface coupled to said camera so as to provide said digital pixel data of a given dynamic range as a grayscale mapping of said video image;

providing an expected area value for said pupil;

determining a grayscale threshold value in accordance with said expected area value and grayscale distribution of said digital pixel data;

segregating said digital pixel data into first and second groups using said grayscale threshold value as a discriminator;

determining the position of said pupil in accordance with the segregated data of one of said first group or said second group; and

providing feedback data corresponding to said pupil position from said computer.

30. The tracking method of claim 29 wherein said step of processing said pixel data further comprises the step of grouping contiguously related individual pixels from one of said first group or said second group into a set of respective pixel blobs.

31. The tracking method of claim 30 wherein said step of processing said pixel data further comprises the step of selecting one of said pixel blobs corresponding to said pupil image.

32. The tracking method of claim 29 wherein said step of providing feedback uses a display.

33. The tracking method of claim 32 further comprising the step of mounting said camera and said display, prior to said step of acquiring video data, to fix the relative physical positions of said camera and said display.

34. The tracking method of claim 29 wherein said feedback data is provided by a display screen, said grayscale threshold value is selected so that the total pixel area of said first group is substantially equal to a pre-determined expected pupil area, and said processing by said computer further comprises the steps of:

grouping contiguously related pixels from said first group into a first set of respective pixel blobs;

selecting from said first set a pixel blob corresponding to said pupil image;

determining the position of said pupil image by a calculated value based on a property of said pixel blob as selected; and

mapping the position of said pupil image in an image coordinate space into a position in a display screen coordinate space.

35. The tracking method of claim 34 wherein said step of selecting the pixel blob from said first set comprises the steps of:

calculating a statistic for each pixel blob in said first set;

comparing said statistic for each pixel blob in said first set with an expected value corresponding to said pupil image to calculate a difference error; and

selecting from said first set the pixel blob corresponding to said pupil image where said difference error is less than a pre-determined tolerance.