Head mounted eye tracking and display system

• US 7,401,920 B1
• Filed: 05/19/2004
• Issued: 07/22/2008
• Est. Priority Date: 05/20/2003
• Status: Active Grant

• Current Assignee: Elbit Systems Limited
• Original Assignee: Elbit Systems Limited
• Inventors: Yaeli, Yosi, Ophir, Yoav, Kranz, Yaron, Shamir, Hanan
• Primary Examiner(s): SCHWARTZ, JORDAN MARC
• Application Number:
  US10/848,824
• Time in Patent Office: 1,525 Days
• Field of Search: 351/206, 351/209, 351/210
• US Class Current: 351/210
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G06V 40/19 Sensors therefor

Abstract

A system for eye tracking that determines the line of sight of a user according to the relative position between the center of the pupil and a reference point, the system including an image detector that captures an image of the eye, a pupil-illuminating light source that illuminates the pupil of the user, a reference light source that illuminates a different portion of the face of the user as a reference point and an imaging processor that analyzes the captured eye image to determine the line of sight.

Claims

1. System for tracking the eye of a user, the system comprising:

an image detector, directed at said eye, for detecting an eye image of at least said eye;

at least one pupil-illuminating light source, each said at least one pupil-illuminating light source emitting a respective pupil-illuminating light beam, each said pupil-illuminating light beam having at least a portion being directed at the pupil of said eye, the portion being aligned with the optical axis of said image detector and at least partially illuminating the pupil of said eye;

at least one reference light source that differs from said pupil-illuminating light source, each said at least one reference light source emitting a respective reference light beam, for illuminating a portion of the face of said user; and

an imaging processor, coupled with said image detector, for analyzing said eye image,

wherein at least a portion of each of said at least one pupil-illuminating light beam is reflected towards said image detector, thereby indicating a pupil region in said eye image, said pupil region being associated with said pupil,

wherein at least a portion of each of said at least one reference light beam is reflected towards said image detector, thereby indicating at least one reference region in said eye image, each said reference region being associated with a recognizable portion of the face,

wherein said imaging processor identifies said pupil region and at least one of said at least one reference region, and

wherein said imaging processor determines the line of sight of said user according to said pupil region and according to at least one of said at least one reference region.

2. The system according to claim 1, wherein said imaging processor determines said line of sight according to the coordinates of the center of said pupil region.

3. The system according to claim 1, further comprising an optical relay assembly,

wherein said optical relay assembly receives at least one selected light beam of said at least one pupil-illuminating light beam and said at least one reference light beam, and

wherein said optical relay assembly directs at least a portion of said at least one selected light beam toward said eye.

4. The system according to claim 3, wherein said imaging processor controls said at least selected pupil-illuminating light source to produce said respective pupil-illuminating light beam, characterized by an increased intensity, when said intensity of ambient light increases,

wherein said imaging processor controls said at least selected pupil-illuminating light source to produce said respective pupil-illuminating light beam, characterized by a decreased intensity, when said intensity of ambient light decreases,

wherein said imaging processor identifies said pupil region as a brighter region in said eye image, when said intensity of ambient light increases, and

wherein said imaging processor identifies said pupil region as a darker region in said eye image, when said intensity of ambient light decreases.

5. The system according to claim 1, further comprising at least one ambient light detector, for determining the intensity of ambient light in said eye image,

wherein said imaging processor is further coupled with at least one of said at least one pupil-illuminating light source, and

wherein said imaging processor controls at least a selected one of said at least one pupil-illuminating light source to produce said respective pupil-illuminating light beam, characterized according to said intensity of ambient light.

6. The system according to claim 1, wherein said imaging processor is further coupled with at least one selected light source of said at least one pupil-illuminating light source and said reference light source, and

wherein said imaging processor controls said at least one selected light source according to said eye image.

7. The system according to claim 1, further comprising a beam splitter,

wherein said beam splitter receives said pupil-illuminating light beam from said pupil-illuminating light source,

wherein said beam splitter reflects at least a portion of said pupil-illuminating light beam toward said eye, and

wherein said beam splitter transmits at least a portion of said eye image toward said image detector.

8. The system according to claim 1, further being mounted on a helmet.

9. The system according to claim 8, further comprising a helmet visor,

wherein said helmet visor receives at least one selected light beam of said at least one pupil-illuminating light beam and said at least one reference light beam, and

wherein said helmet visor at least partially reflects said at least one selected light beam toward said eye of said user.

10. The system according to claim 9, wherein said helmet visor at least partially transmits light from a scene, thereby allowing said user to view said scene.

11. The system according to claim 10, wherein said imaging processor selects a target from said scene according to said line of sight.

12. The system according to claim 11, further comprising at least one display light source, each said at least one display light source emitting a respective display light beam, each said display light beam carrying at least a portion of a display image,

wherein said target is outside the field of display of said display image.

13. The system according to claim 9, wherein the shape of said visor is selected from the list consisting of:

spherical;

aspherical; and

planar.

14. The system according to claim 1, further comprising at least one display light source, each said at least one display light source emitting a respective display light beam, each said display light beam carrying at least a portion of a display image.

15. The system according to claim 14, wherein said at least one display light source displays a target marking respective of a selected target, around said selected target, and

wherein a weapon system is actuated when said target marking substantially matches said line of sight.

16. The system according to claim 14, wherein said at least one display light source further projects at least one stimulated light beam toward said eye, and

wherein said imaging processor further analyzes the movement of said eye in response to said at least one stimulating light beam, thereby examining user reflexes.

17. The system according to claim 14, wherein said imaging processor registers a logical display element from said display image with the field of view of said user.

18. The system according to claim 14, wherein said display image is collimated.

19. The system according to claim 18, further comprising a collimator, for collimating said at least one display light beam.

20. The system according to claim 1, said imaging processor being further coupled with at least one display light source,

wherein said imaging processor controls said at least one display light source to produce a display image according to said eye image.

21. The system according to claim 14, wherein at least a portion of at least one of said reference light sources is aligned with at least a portion of the optical axis of said display light source.

22. The system according to claim 1, wherein said recognizable portion of the face is the cornea.

23. The system according to claim 1, wherein said recognizable portion of the face is the eyelids.

24. The system according to claim 1, wherein said imaging processor further determines a physiological state of said user according to said eye image.

25. The system according to claim 24, wherein said imaging processor further determines said physiological state by comparing said eye image with a reference eye model, thereby detecting anomalies in said eye image.

26. The system according to claim 24, wherein said imaging processor further statistically analyzes said eye image, thereby determining temporal features of said eye.

27. The system according to claim 24, wherein said imaging processor further statistically analyzes said eye image, thereby determining temporal features of said eye, and

wherein said imaging processor further compares said temporal features with respective features associated with a reference eye model, thereby detecting anomalies in said eye.

28. The system according to claim 24, wherein said physiological state is selected from the list consisting of:

fatigue;

loss of consciousness;

cross-eye;

astigmatism;

eye damage; and

vertigo.

29. The system according to claim 24, wherein said imaging processor further initiates an alarm signal according to said physiological state.

30. The system according to claim 1, further comprising a stimulus light source, wherein said stimulus light source projects at least one stimulating light beam toward said eye, and

wherein said imaging processor further analyzes the movement of said eye in response to said at least one stimulating light beam, thereby examining user reflexes.

31. The system according to claim 1, wherein at least a selected one of said reference light beams is collimated.

32. The system according to claim 31, further comprising a collimator, for collimating said selected reference light beam.

33. The system according to claim 32, further comprising at least one display light source, each said at least one display light source emitting a respective display light beam, each said at least one display light beam carrying at least a portion of a display image,

wherein said collimator further collimates at least one of said at least one display light beam.

34. Method for tracking the eye of a user, the method comprising the procedures of:

emitting at least one pupil-illuminating light beam, each said at least one pupil-illuminating light beam having at least a portion being directed at the eye, the portion being aligned with an optical axis of an image detector and at least partially illuminating the pupil of said eye;

emitting at least one reference light beam differing from said pupil-illuminating light beam, for illuminating a portion of the face of said user;

detecting an eye image of at least said eye, around the optical axis of the image detector;

identifying in said eye image a pupil region and at least one reference region associated with a recognizable portion of the face; and

determining the line of sight of said user according to said pupil region and according to at least one of said at least one reference region,

wherein at least a portion of said at least one pupil-illuminating light beam indicates said pupil region in said eye image,

wherein at least a portion of each of said at least one reference light beam indicates said at least one reference region in said eye image, and

wherein said line of sight is determined according to said pupil region and according to at least one of said at least one reference region.

35. The method according to claim 34, wherein said line of sight is further determined according to the coordinates of the center of said pupil region.

36. The method according to claim 34, further comprising the procedure of emitting at least one display light beam, each said display light beam carrying at least a portion of a display image.

37. The method according to claim 36, further comprising the procedure of collimating said at least one display light beam.

38. The method according to claim 36, further comprising the procedure of controlling at least said display light beam according to said line of sight, after said procedure of determining said line of sight.

39. The method according to claim 34, further comprising the preliminary procedure of detecting ambient light,

wherein at least a selected one of said at least one pupil-illuminating light beam and said at least one reference light beam is produced according to said detected ambient light.

40. The method according to claim 34, wherein said recognizable portion of the face is the cornea.

41. The method according to claim 34, wherein said recognizable portion of the face is the eyelids.

42. The method according to claim 34, further comprising the procedure of directing at least one of said at least one pupil-illuminating light beam toward said eye, performed after said procedure of emitting said at least one pupil-illuminating light beam.

43. The method according to claim 42, further comprising the procedure of directing at least a portion of said pupil-illuminating light beam toward an image detector, performed after said procedure of directing said at least one of said at least one pupil-illuminating light beam toward said eye.

44. The method according to claim 34, further comprising the procedure of directing at least one of said at least one reference light beam toward the face of said user, performed after said procedure of emitting said at least one reference light beam.

45. The method according to claim 44, further comprising the procedure of directing at least a portion of said reference light beam toward an image detector, performed after said procedure of directing at least one of said at least one reference light beam toward said face.

46. The method according to claim 34, further comprising the procedure of collimating said at least one reference light beam.

47. The method according to claim 34, further comprising the procedure of determining a physiological state of said user according to said eye image.

48. The method according to claim 47, wherein said physiological state is determined by comparing said eye image with a reference eye model, thereby detecting anomalies in said eye image.

49. The method according to claim 47, further comprising the procedures of:

statistically analyzing said eye image, thereby determining statistical features of said eye; and

comparing said statistical features with respective features associated with a reference eye model, thereby determining anomalies in said eye,

wherein said physiological state is further determined according to said anomalies.

50. The method according to claim 34, further comprising the procedure of statistically analyzing said eye image, thereby determining statistical features of said eye.

51. The method according to claim 34, further comprising the procedures of:

identifying a target along said determined line of sight;

prompting user confirmation for target acquisition;

receiving user confirmation of said target; and

directing a weapon toward said target.

52. System for directing a weapon toward a target, the system comprising:

an eye tracker for tracking the eye of a user;

a head position tracker for monitoring at least the position of the head of said user;

a vehicle position and orientation tracker for monitoring the position and orientation of a vehicle; and

a processor coupled with said eye tracker, said head position tracker and with said vehicle position and orientation tracker, wherein said eye tracker comprises;

an image detector directed at said eye, for detecting an eye image of at least said eye;

at least one pupil-illuminating light source, each said at least one pupil-illuminating light source emitting a respective pupil-illuminating light beam, at least a portion of each said pupil-illuminating light beam being aligned with at least a portion of the optical axis of said image detector, each said pupil-illuminating light beam at least partially illuminating the pupil of said eye;

at least one reference light source, each said at least one reference light source emitting a respective reference light beam, for illuminating a portion of the face of said user; and

an imaging processor, coupled with said image detector, for analyzing said eye image,

wherein at least a portion of each of said at least one pupil-illuminating light beam is reflected towards said image detector, thereby indicating a pupil region in said eye image, said pupil region being associated with said pupil,

wherein at least a portion of each of said at least one reference light beam is reflected towards said image detector, thereby indicating at least one reference region in said eye image, each said reference region being associated with a recognizable portion of said face,

wherein said imaging processor identifies said pupil region and at least one of said at least one reference region,

wherein said imaging processor determines the line of sight of said user according to said pupil region and according to at least one of said at least one reference region, and

wherein said processor directs said weapon toward said target according to said line of sight, said at least position of said head and according to said position and orientation of said vehicle.

53. The system according to claim 52, further comprising an audio assembly coupled with said processor, said audio assembly sounding audible signals to said user indicating predetermined states or operation modes of said weapon.