Method and apparatus for image-based eye tracking for retinal diagnostic or surgery device

• US 20050270486A1
• Filed: 06/01/2005
• Published: 12/08/2005
• Est. Priority Date: 06/02/2004
• Status: Active Grant

• Current Assignee: OD-OS GmbH
• Original Assignee: SensoMotoric Instruments GmbH (Apple Inc.)
• Inventors: Teiwes, Winfried, Grecu, Horia
• Granted Patent: US 8,811,657 B2
• Time in Patent Office: Days
• US Class Current: 351/209
• CPC Class Codes: A61B 3/102 for optical coherence tomography [OCT]
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: A61B 3/145 by video means
• CPC Class Codes: A61F 2009/00846 Eyetracking
• CPC Class Codes: A61F 2009/00872 Cornea
• CPC Class Codes: A61F 9/008 using laser
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10101 Optical tomography; Optical coherence tomography [OCT]
• CPC Class Codes: G06T 2207/30041 Eye; Retina; Ophthalmic
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G06T 7/0016 involving temporal comparison
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction

Abstract

An eye tracking method for determining a position of an eye or a part of an eye in an image of an image sequence by performing a comparison between said image and a reference image, said process including: aligning a set of images; computing an enhanced reference image based on a combination of said set of aligned images; and determining said position in said image of said image sequence by comparing said image of said image sequence and said enhanced reference image to yield a motion estimation between said reference image and said image of said sequence.

Claims

1. An eye tracking method for determining a position of an eye or a part of an eye in an image of an image sequence by performing a comparison between said image and a reference image, said process including:

aligning a set of images;

computing an enhanced reference image based on a combination of said set of aligned images; and

determining said position in said image of said image sequence by comparing said image of said image sequence and said enhanced reference image to yield a motion estimation between said reference image and said image of said sequence.

2. The method of claim 1, further comprising:

aligning said set of images which are taken before said image sequence to the same retinal position;

combining the aligned set of images by averaging the aligned set of images to generate the enhanced reference image.

3. The method of claim 2, further comprising:

using said retina position determined in said image sequence for tracking and or positional compensation in connection with a diagnosis and/or surgical device.

4. The method of claim 3, wherein said diagnosis and/or surgical device is one of the following:

an OCT apparatus;

a refractive surgery device.

5. The method of claim 1, further comprising the following:

performing OCT spot trace rectification to eliminate a saturated OCT spot trace from an image in said sequence by replacing the saturated area by an interpolation or an average intensity in a corresponding area of unsaturated images and compensating illumination variations using one or more suitable filters.

6. The method of claim 1, comprising:

performing a global motion estimation between said reference image and said image of said sequence;

or

performing a motion estimation based on parts or landmarks of said reference image and said image of said image sequence.

7. The method of claim 6, wherein

said global motion estimation is refined by said motion estimation based on parts or landmarks.

8. The method of claim 1, wherein said enhanced reference image is used for OCT spot rectification or for deriving a spectrally matched filter for selecting one or more retina spectrum components having the highest SNR ratio.

9. The method of claim 1 further comprising:

aligning image sequences taken at different periods of time by aligning an enhanced reference corresponding to one of said sequences with a reference image or an enhanced reference image corresponding to said second sequence.

10. The method of claim 9, wherein said reference image corresponding to said second sequence is a red-free image.

14. The method of claim 13, wherein said tracking information is obtained based on an enhanced reference image as defined in claim 1.

11. A method of enhancing the video display of an image of an eye or a portion of an eye, said method comprising:

aligning a sequence of images which are preceding an actual image of said video stream with respect to said actual image;

combining the resulting aligned set of images into an enhanced image to be used as a basis for an enhanced actual image of said video stream.

12. The method of claim 11, wherein for each actual image of said video stream the display is based on said enhanced image resulting of a combination of aligned images preceding said actual image.

13. The method of claim 11, further comprising:

aligning said images of said preceding sequence based on eye tracking information.

15. A method for operating a diagnostic or surgical device based on eye tracking, said method comprising:

determining whether an anomaly is detected during said eye tracking;

if an anomaly is detected during said eye tracking, putting said diagnostic or surgical device in a HOLD mode and/or

re-performing the operation of said diagnostic or surgical device for those parts of the eye where an anomaly has been detected.

16. The method of claim 15, wherein said anomaly is one of the following:

a change in image quality beyond a certain threshold;

a fast movement of the eye the speed of which is beyond a certain threshold;

a blurring of the image used for eye tracking;

a blurring of the image used for eye tracking in combination with a fast movement;

a cumulated difference between subsequent tracking images lying beyond a certain threshold;

a tracking error which is detected based on one or more images being considered statistically as outliers;

a sequence of images with a highs speed retinal movement;

an image sequence pattern representing a saccade;

an image sequence pattern representing a blink.

17. The method of claim 15, wherein said determination whether an anomaly is detected is performed in realtime, and if an anomaly is detected said diagnostic or surgical device is put into a HOLD mode.

18. The method of claim 15, wherein said anomaly during eye tracking is based on retrospectively evaluating images which have been obtained during eye tracking to identify anomalies, said method further comprising:

reperforming the operation of said surgical or diagnostic device on those locations for which anomalies have been detected.

19. The method of claim 18, wherein said retrospective evaluation comprises:

comparing image sequences obtained during operation of said surgical or diagnostic device with eye movement patterns representing common eye movements to detect said anomalies.

20. An eye tracking system for determining a position of an eye or a part of an eye in an image of an image sequence by performing a comparison between said image and a reference image, said process including:

a module for aligning a set of images;

a module for computing an enhanced reference image based on a combination of said set of aligned images; and

a module for determining said position in said image of said image sequence by comparing said image of said image sequence and said enhanced reference image to yield a motion estimation between said enhanced reference image and said image of said sequence.

21. The system of claim 20, further comprising:

a module for aligning said set of images which are taken before said image sequence to the same retinal position;

a module for combining the aligned set of images by averaging the aligned set of images to generate the enhanced reference image.

22. The system of claim 20, further comprising:

a module for using said retina position determined in said image sequence for tracking and or positional compensation in connection with a diagnosis or surgical device.

23. The system of claim 22, wherein said diagnosis or surgical device is one of the following:

an OCT apparatus;

a refractive surgery device.

24. The system of claim 20, further comprising one or more of the following:

a module for performing OCT spot trace rectification to eliminate a saturated OCT spot trace from an image in said sequence by replacing the saturated area by an interpolation or an average intensity in a corresponding area of unsaturated images;

a module for compensating illumination variations using one or more suitable filters.

25. The system of claim 20, comprising one or more of the following:

a module for performing a global motion estimation between said reference image and said

image of said sequence;

a module for performing a motion estimation based on parts or landmarks of said reference image and said image of said image sequence.

26. The system of claim 25, wherein

said global motion estimation is refined by said motion estimation based on parts or landmarks.

27. The system of claim 20, wherein said enhanced reference image is used for the following:

OCT spot rectification or

deriving a spectrally matched filter for selecting one or more retina spectrum components having the highest SNR ratio.

28. The system of claim 20 further comprising:

a module for aligning image sequences taken at different periods of time by aligning an enhanced reference corresponding to one of said sequences with a reference image or an enhanced reference image corresponding to said second sequence.

29. The system of claim 28, wherein said reference image corresponding to said second sequence is a red-free image.

33. The apparatus of claim 32, wherein said tracking information is obtained based on an enhanced reference image as defined in claim 20.

30. An apparatus for enhancing the video display of an image of an eye or a portion of an eye, said apparatus comprising:

a module for aligning a sequence of images which are preceding an actual image of said video stream with respect to said actual image;

a module for combining the resulting aligned set of images into an enhanced image to be used as a basis for an enhanced actual image of said video stream.

31. The apparatus of claim 30, wherein for each actual image of said video stream the display is based on said enhanced image resulting of a combination of aligned images preceding said actual image.

32. The apparatus of claim 30, further comprising:

a module for aligning said images of said preceding sequence based on eye tracking information.

34. An apparatus to be used in connection with a diagnostic or surgical device which operates based on eye tracking, said apparatus comprising:

a module for determining whether an anomaly is detected during said eye tracking; and

a module for if an anomaly is detected during said eye tracking, putting said diagnostic or surgical device in a HOLD mode and/or

a module for re-performing the operation of said diagnostic or surgical device for those parts of the eye where an anomaly has been detected.

35. The apparatus of claim 34, wherein said anomaly is one or more of the following:

a change in image quality beyond a certain threshold;

a fast movement of the eye the speed of which is beyond a certain threshold;

a blurring of the image used for eye tracking;

a blurring of the image used for eye tracking in combination with a fast movement;

a cumulated difference between subsequent tracking images lying beyond a certain threshold;

a tracking error which is detected based on one or more images being considered statistically as outliers;

a sequence of images with a highs speed retinal movement;

an image sequence pattern representing a saccade;

an image sequence pattern representing a blink.

36. The apparatus of claim 34, wherein said determination whether an anomaly is detected is performed in realtime, and if an anomaly is detected said diagnostic or surgical device is put into a HOLD mode.

37. The apparatus of claim 34, wherein said anomaly during eye tracking is based on retrospectively evaluating images which have been obtained during eye tracking to identify anomalies, said apparatus further comprising:

a module for reperforming the operation of said surgical or diagnostic device on those locations for which anomalies have been detected.

38. The apparatus of claim 37, wherein said retrospective evaluation comprises:

comparing image sequences obtained during operation of said surgical or diagnostic device with eye movement patterns representing common eye movements to detect said anomalies.

39. A computer program product comprising computer program code for determining a position of an eye or a part of an eye in an image of an image sequence by performing a comparison between said image and a reference image, said computer program code including:

computer program code for aligning a set of images;

computer program code for computing an enhanced reference image based on a combination of said set of aligned images; and

computer program code for determining said position in said image of said image sequence by comparing said image of said image sequence and said enhanced reference image to yield a motion estimation between said reference image and said image of said sequence.