System and methods for controlling automatic scrolling of information on a display or screen

• US 20020105482A1
• Filed: 02/22/2002
• Published: 08/08/2002
• Est. Priority Date: 05/26/2000
• Status: Active Grant

• Current Assignee: Jerome H. Lemelson, John H. Hiett
• Original Assignee: Jerome H. Lemelson, John H. Hiett
• Inventors: Lemelson, Jerome H., Hiett, John H.
• Granted Patent: US 6,603,491 B2
• Time in Patent Office: Days
• US Class Current: 345/7
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0485 Scrolling or panning

Abstract

A system 10 for controlling the automatic scrolling of information on a computer display or screen 12 is disclosed. The system 10 generally includes a computer display or screen12, a computer system 14, gimbaled sensor system 16 for following and tracking the position and movement of the user'"'"'s head 18 and user'"'"'s eye 20, and a scroll activating interface algorithm using a neural network to find screen gaze coordinates implemented by the computer system 14 so that corresponding scrolling function is performed based upon the screen gaze coordinates of the user'"'"'s eye 20 relative to a certain activation area(s) on the display or screen 12. The gimbaled sensor system 16 contains a gimbaled platform 24 mounted at the top of the display or screen 12. The gimbaled sensor system 16 tracks the user'"'"'s 22 head 18 and eye 20, allows the user to be free from any attachments while the gimbaled sensor system 16 is eye tracking, and still allows the user to freely move his or her head when the system 10 is in use. A method of controlling automatic scrolling of information on a display or screen 12 by a user 22 is also disclosed. The method generally includes the steps of finding a screen gaze coordinate 146 on the display or screen 12 of the user 22, determining whether the screen gaze coordinate 146 is within at least one activated control region, and activating scrolling to provide a desired display of information when the gaze direction is within at least one activated control region. In one embodiment, the control regions are defined as upper control region 208, lower region 210, right region 212, and left region 214 for controlling the scrolling respectively in the downward, upward, leftward, and rightward directions. In another embodiment, the control regions are defined by concentric rings 306, 308, and 310 for maintaining the stationary position of the information or controlling the scrolling of the information towards the center of the display or screen 12.

Claims

1. A system for controlling automatic scrolling of information on a display or a screen comprising:

a computer system coupled to the display or the screen,

an eye scroll control sensor coupled to the computer system for tracking and determining a position of the eye of a user relative to the display or the screen, and

a scroll activating interface system coupled to the computer system and interfaced with the eye scroll control sensor for implementing automatic scrolling based upon the position of the eye of the user relative to an activation area on the display or the screen.

2. The system for controlling automatic scrolling of information according to claim 1 wherein the eye scroll control sensor further comprises:

an eye tracking system for tracking the eye of the user, and

an eye position determining system for determining the position of the eye of the user relative to the display or the screen.

3. The system for controlling automatic scrolling of information according to claim 1 wherein the eye scroll control sensor is an eye and head scroll control sensor coupled to the computer system for tracking and determining the position of the eye of a user and a position of the head of the user relative to the display or the screen.

4. The system for controlling automatic scrolling of information according to claim 3 wherein the eye and head scroll control sensor further comprises:

a gimbaled head sensor system for tracking the position of the head of the user relative to the display or the screen.

5. The system for controlling automatic scrolling of information according to claim 4 wherein the gimbaled head sensor system further comprises:

a camera or photo sensor for providing images of the head and the eye of the user,

a zoom lens coupled to the camera or photo sensor for focusing the camera or photo sensor at the user, and

optics coupled to the camera or photo sensor for aiding the camera or photo sensor in detecting or providing images of the head and the eye of the user so that the images can be processed by the computer system.

6. The system for controlling automatic scrolling of information according to claim 5 wherein:

the computer system further comprises a computer interface card, and

the computer interface card further comprises a digital to analog converter coupled to the zoom lens for converting a digital control signal from the computer system to an analog control signal for focusing the camera or photo sensor at the user.

7. The system for controlling automatic scrolling of information according to claim 4 wherein the gimbaled head sensor system further comprises:

an LED or laser coupled to optics of the gimbaled targeted at the head and eye of the user to provide a glint on the eye of the user to enhance focus of the gimbaled onto a center of the eye of the user.

8. The system for controlling automatic scrolling of information according to claim 7 wherein:

the computer system further comprises a computer interface card, and

the computer interface card has a light source driver for driving the LED or laser that is coupled to optics of the gibaled head sensor system.

9. The system for controlling automatic scrolling of information according to claim 4 wherein the gimbaled head sensor system further comprises:

a distance range finder for determining a distance between the display or the screen and the user.

10. The system for controlling automatic scrolling of information according to claim 9 wherein the distance range finder is an ultrasonic range finder for determining the distance between the display or the screen and the user.

11. The system for controlling automatic scrolling of information according to claim 9 wherein the distance range finder is an LED or laser range finder for determining the distance between the display or the screen and the user.

12. The system for controlling automatic scrolling of information according to claim 9 wherein

the computer system further comprises a computer interface card, and

the computer interface card has a range driver that drives the distance range finder.

13. The system for controlling automatic scrolling of information according to claim 4 wherein:

the gimbaled head sensor system further comprises a directional microphone for detecting or sensing audio signals from the user to provide voice or sound control to the user of the automatic scrolling,

the computer system further comprises a computer interface card, and

the computer interface card further comprises;

an audio amplifier coupled to the directional microphone for amplifying the detected or sensed voice or sound signals, and

an audio filter coupled to the audio amplifier for filtering out any noise or interference that is detected or sensed by the directional microphone.

14. The system for controlling automatic scrolling of information according to claim 13 wherein:

the computer interface card further comprises an analog to digital converter coupled to the audio filter, for converting the audio signals that are audio analog signals to audio digital signals so that the audio digital signals are capable of being processed by the computer system.

15. The system for controlling automatic scrolling of information according to claim 4 wherein:

the gimbaled head sensor system further comprises a camera or photo sensor for detecting or providing video images of the head and the eye of the user to provide video signals to be image processed by the computer system so that the head sensor camera or photo sensor is focused on a center of the eye of the user,

the computer system further comprises a computer interface card, and

the computer interface card further comprises;

a video amplifier coupled to the camera or photo sensor for amplifying the video signals, and

a video filter coupled to the video amplifier for filtering out any video noise or video interference that is picked up or sensed by the camera or photo sensor.

16. The system for controlling automatic scrolling of information according to claim 15 wherein:

the computer interface card further comprises an analog to digital converter coupled to the video filter for converting the video signals that are analog video signals to digital video signal so that the digital video signals are capable of being processed by the computer system.

17. The system for controlling automatic scrolling of information according to claim 4 further comprising a pan servo and a tilt servo coupled to the gimbaled head sensor system for respectively controlling the horizontal and vertical pivotal movements of the gimbaled head sensor system.

18. The system for controlling automatic scrolling of information according to claim 17 wherein:

the computer system further comprises a computer interface card, and

the computer interface card further comprises;

an analog to digital converter coupled to the pan servo for converting analog horizontal position signals of the gimbaled head sensor system to digital horizontal position signals, and

an analog to digital converter coupled to the tilt servo for converting analog vertical position signals of the gimbaled head sensor system to digital vertical position signals.

19. The system for controlling automatic scrolling of information according to claim 17 wherein:

the computer system further comprises a computer interface card, and

the computer interface card further comprises;

a pan driver coupled to the pan servo for driving the pan servo and for controlling the horizontal movement of the gimbaled head sensor system, and

a tilt driver coupled to the tilt servo for driving the tilt servo and for controlling the vertical movement of the gimbaled head sensor system.

20. The system for controlling automatic scrolling of information according to claim 19 wherein:

the computer interface card further comprises;

a digital to analog converter coupled to the pan driver for converting a digital control signal from the computer system to an analog control signal for controlling the pan servo and horizontal position of the gimbaled head sensor system.

21. The system for controlling automatic scrolling of information according to claim 19 wherein:

the computer interface card further comprises;

a digital to analog converter coupled to the tilt driver for converting a digital control signal from the computer system to an analog control signal for controlling the tilt servo and vertical position of the gimbaled head sensor system.

22. The system for controlling automatic scrolling of information according to claim 4 wherein the computer system further comprises:

a computer interface bus,

a microprocessor coupled to the computer interface bus,

memory coupled to the microprocessor,

a computer interface card interfaced with the computer interface bus, and

a display driver coupled to the computer interface bus and is adapted to be coupled to the display or the screen wherein the display driver drives the display or the monitor.

23. The system for controlling automatic scrolling of information according to claim 22 wherein the computer interface card further comprises:

buffer memory that couples to the computer interface bus and that stores and allows organized providing of digital information between various components of the system.

25. The method of controlling automatic scrolling of information on a display or a screen according to claim 24 wherein the step of finding a gaze direction on the display or the screen further comprises the steps of:

detecting a user that is closest to the display or the screen,

focusing and magnifying a field of view of a camera on an eye of the user to provide a magnified image of the eye,

inputting the magnified image into a computer system,

determining physical coordinates of a center of a cornea of the eye and a glint center of the eye,

determining a vector between the center of the cornea and a glint center on the display or the screen,

calculating gaze coordinates of the user on the display or the screen, and

sending the gaze coordinates to the computer system for processing by an application program for controlling the scrolling of information on the display or the screen.

26. The method of controlling automatic scrolling of information on a display or a screen according to claim 25 wherein the step of focusing and magnifying a field of view of a camera on an eye of the user further comprises the steps of:

determining a correction vector for re-centering the camera to adjust the field of view, and

zooming the field of view of the camera onto the center of the eye of the user.

27. The method of controlling automatic scrolling of information on a display or a screen according to claim 25 wherein the step of detecting a user that is closest to the display or the screen further comprises:

reading raw image data of an image of the user into the computer system,

filtering the image of the user,

segmenting the raw image data,

forming clusters of the raw image data,

determining head clusters from the clusters of the raw image data,

determining eye regions of the head clusters,

determining an eye center of gravity for each of the eye regions,

adjusting a pan and a tilt of a camera for bringing an image center to the eye center,

calculating an area for the eye cluster, and

zooming the camera so that a large portion of an eye cluster is within a field of view of the camera.

28. The method of controlling automatic scrolling of information on a display or a screen according to claim 24 wherein the step of determining whether the gaze direction is within at least one activated control region further comprises the steps of:

determining whether the gaze direction is within an upper horizontal region, and

scrolling information downwards at a rate based on the screen gaze coordinates that are within the lower horizontal region.

29. The method of controlling automatic scrolling of information on a display or a screen according to claim 24 wherein the step of determining whether the gaze direction is within at least one activated control region further comprises the steps of:

determining whether the gaze direction is within a lower horizontal region, and

scrolling information upwards at a rate based on the screen gaze coordinates that are within the lower horizontal region.

30. The method of controlling automatic scrolling of information on a display or a screen according to claim 24 wherein the step of determining whether the gaze direction is within at least one activated control region further comprises the steps of:

determining whether the gaze direction is within a right vertical region, and

scrolling information leftwards at a rate based on the screen gaze coordinates that are within the right vertical region.

31. The method of controlling automatic scrolling of information on a display or a screen according to claim 24 wherein the step of determining whether the gaze direction is within at least one activated control region further comprises the steps of:

determining whether the gaze direction is within a left vertical region, and

scrolling information rightwards at a rate based on the screen gaze coordinates that are within the left vertical region.

32. The method of controlling automatic scrolling of information on a display or a screen according to claim 24 wherein the step of determining whether the gaze direction is within at least one activated control region further comprises the steps of:

determining whether the gaze direction is within an upper horizontal region and a right vertical region, and

scrolling information respectively downwards and leftwards at a rate based on the screen gaze coordinates that are within the upper horizontal region and the right vertical region.

33. The method of controlling automatic scrolling of information on a display or a screen according to claim 24 wherein the step of determining whether the gaze direction is within at least one activated control region further comprises the steps of:

determining whether the gaze direction is within an upper horizontal region and a left vertical region, and

scrolling information respectively downwards and rightwards at a rate based on the screen gaze coordinates that are within the upper horizontal region and the left vertical region.

34. The method of controlling automatic scrolling of information on a display or a screen according to claim 24 wherein the step of determining whether the gaze direction is within at least one activated control region further comprises the steps of:

determining whether the gaze direction is within a lower horizontal region and a right vertical region, and

scrolling information respectively upwards and leftwards at a rate based on the screen gaze coordinates that are within the upper horizontal region and the right vertical region.

35. The method of controlling automatic scrolling of information on a display or a screen according to claim 24 wherein the step of determining whether the gaze direction is within at least one activated control region further comprises the steps of:

determining whether the gaze direction is within a lower horizontal region and a left vertical region, and

scrolling information respectively upwards and rightwards at a rate based on the screen gaze coordinates that are within the lower horizontal region and the left vertical region.

36. The method of controlling automatic scrolling of information on a display or a screen according to claim 24 wherein the step of determining whether the gaze direction is within at least one activated control region further comprises the steps of:

determining whether the gaze direction is within an upper horizontal region and a right vertical region, and

allowing information on the display or the screen to remain stationary when the screen gaze coordinates are within the upper horizontal region and the right vertical region.

37. The method of controlling automatic scrolling of information on a display or a screen according to claim 24 wherein the step of determining whether the gaze direction is within at least one activated control region further comprises the steps of:

determining whether the gaze direction is within an upper horizontal region and a left vertical region, and

allowing information on the display or the screen to remain stationary when the screen gaze coordinates are within the upper horizontal region and the left vertical region.

38. The method of controlling automatic scrolling of information on a display or a screen according to claim 24 wherein the step of determining whether the gaze direction is within at least one activated control region further comprises the steps of.

determining whether the gaze direction is within a lower horizontal region and a right vertical region, and

allowing information on the display or the screen to remain stationary when the screen gaze coordinates are within the upper horizontal region and the right vertical region.

39. The method of controlling automatic scrolling of information on a display or a screen according to claim 24 wherein the step of determining whether the gaze direction is within at least one activated control region further comprises the steps of:

determining whether the gaze direction is within a lower horizontal region and a left vertical region, and

allowing information on the display or the screen to remain stationary when the screen gaze coordinates are within the lower horizontal region and the left vertical region.

40. The method of controlling automatic scrolling of information on a display or a screen according to claim 24 wherein the step of determining whether the gaze direction is within at least one activated control region further comprises the steps of:

determining whether the gaze direction is within a static region defined by at least one of a number of concentric circles,

determining whether the gaze direction is within an activated control region defined by another of the number of concentric circles, and

activating scrolling according to the gaze direction within the activated control region to provide a desired display of information so that the region at which the gaze direction of the user is directed to the center of the display or the screen at a rate that is defined for the concentric circle at which the gaze direction of the user is directed.

41. The method of controlling automatic scrolling of information on a display or a screen according to claim 40 wherein the step of determining whether the gaze direction is within an activated control region defined by another of the number of concentric circles further comprises the steps of:

calculating a radius and an angle with respect to a center of the number of concentric circles to define a gaze vector,

calculating horizontal and vertical scroll rates based on the gaze vector, and

scrolling the information on the display or the screen in the horizontal and the vertical directions based on the calculated horizontal and vertical scroll rates.

42. The method of controlling automatic scrolling of information on a display or a screen according to claim 40 wherein the step of determining whether the gaze direction is within an activated control region defined by another of the number of concentric circles further comprises the step of:

determining whether the gaze direction is within an activated control quadrant wherein the information on the display or the screen is moved downward and leftward.

43. The method of controlling automatic scrolling of information on a display or a screen according to claim 40 wherein the step of determining whether the gaze direction is within an activated control region defined by another of the number of concentric circles further comprises the step of:

determining whether the gaze direction is within an activated control quadrant wherein the information on the display or the screen is moved upward and leftward.

44. The method of controlling automatic scrolling of information on a display or a screen according to claim 40 wherein the step of determining whether the gaze direction is within an activated control region defined by another of the number of concentric circles further comprises the step of:

determining whether the gaze direction is within an activated control quadrant wherein the information on the display or the screen is moved upward and rightward.

45. The method of controlling automatic scrolling of information on a display or a screen according to claim 40 wherein the step of determining whether the gaze direction is within an activated control region defined by another of the number of concentric circles further comprises the step of:

determining whether the gaze direction is within an activated control quadrant wherein the information on the display or the screen is moved downward and rightward.

46. The method of controlling automatic scrolling of information on a display or a screen according to claim 24 further comprising the steps of:

allowing the user to focus on one line of the information and read an entire document by shifting the information one line up as the user scrolls the information respectively to the right and by shifting the information one line down as the user scrolls the information to the left.

47. The method of controlling automatic scrolling of information on a display or a screen according to claim 24 further comprising the steps of:

determining whether the gaze direction is within a pop-up menu region, and

activating a pop-up menu algorithm when the gaze direction is within the pop-up menu region.

48. The method of controlling automatic scrolling of information on a display or a screen according to claim 24 further comprising the steps of:

determining whether the gaze direction is within a hypertext location, and

activating a hypertext switching algorithm for jumping to a selected hypertext location when the gaze direction is within a desired hypertext area.

50. The method of making a system for controlling automatic scrolling of information according to claim 49 wherein the step of coupling an eye scroll control sensor further comprises the steps of:

coupling an eye tracking system to the computer system for tracking the eye of the user, and

coupling an eye position determining system to the computer system for determining the position of the eye of the user relative to the display or the screen.

51. The method of making a system for controlling automatic scrolling of information according to claim 49 wherein the step of coupling an eye scroll control sensor further comprises the step of:

coupling an eye and head scroll control sensor to the computer system for tracking and determining the position of the eye of a user and a position of the head of the user relative to the display or the screen.

52. The method of making a system for controlling automatic scrolling of information according to claim 51 wherein the step of coupling an eye and head scroll control sensor further comprises the step of:

coupling a gimbaled head sensor system to the computer system for tracking the position of the head of the user relative to the display or the screen.

53. The method of making a system for controlling automatic scrolling of information according to claim 52 wherein the step of coupling a gimbaled head sensor system further comprises the step of:

coupling an LED or laser to optics of the gimbaled head sensor system targeted at the head and eye of the user to provide a glint on the eye of the user to enhance focus of the gimbaled head sensor system onto a center of the eye of the user.

54. The method of making a system for controlling automatic scrolling of information according to claim 52 wherein the step of coupling a gimbaled head sensor system further comprises the step of:

coupling a distance range finder to the computer system for determining a distance between the display or the screen and the user.

55. The method of making a system for controlling automatic scrolling of information according to claim 52 wherein the step of coupling the gimbaled head sensor system further comprises the step of:

coupling a directional microphone to the computer system for detecting or sensing audio signals from the user to provide voice or sound control to the user of the automatic scrolling.

56. The method of making a system for controlling automatic scrolling of information according to claim 52 wherein the step of coupling the gimbaled head sensor system further comprises the step of:

coupling a camera or photo sensor to the computer system for detecting or providing video images of the head and the eye of the user to provide video signals to be image processed by the computer system so that the head sensor camera or photo sensor is focused on a center of the eye of the user.

57. The method of making a system for controlling automatic scrolling of information according to claim 52 further comprising the steps of:

coupling a pan servo and a tilt servo to the gimbaled head sensor system for respectively controlling the horizontal and vertical pivotal movements of the gimbaled head sensor system.

58. The method of making a system for controlling automatic scrolling of information according to claim 57 further comprising the steps of:

coupling a pan driver to the pan servo for driving the pan servo and for controlling the horizontal movement of the gimbaled head sensor system, and

coupling a tilt driver to the tilt servo for driving the tilt servo and for controlling the vertical movement of the gimbaled head sensor system.

59. The system for controlling automatic scrolling of information according to claim 1 wherein the computer system provides medical heads-up information on the display or the screen and the eye of the user controls the automatic scrolling of the information.

60. The system for controlling automatic scrolling of information according to claim 1 wherein the computer system provides information to a technician on the display or the screen and the eye of the technician controls the automatic scrolling of the information.

61. The system for controlling automatic scrolling of information according to claim 1 wherein the computer system provides information to an assembly line worker on the display or the screen and the eye of the worker controls the automatic scrolling of the information.

62. The system for controlling automatic scrolling of information according to claim 1 wherein the computer system provides information to an air traffic controller on the display or the screen and the eye of the controller controls the automatic scrolling of the information.

63. The system for controlling automatic scrolling of information according to claim 1 wherein the computer system provides information to an assembly line worker on the display or the screen and the eye of the worker controls the automatic scrolling of the information.

64. The system for controlling automatic scrolling of information according to claim 1 wherein the computer system provides information for an weather person on the display or the screen and the eye of the weather person controls the automatic scrolling of the information.

65. The method of controlling automatic scrolling of information on a display or a screen according to claim 25 wherein the step of calculating gaze coordinates further comprises the step of:

using a neural network to calculate the gaze coordinates on the display or the screen.

24. A method of controlling automatic scrolling of information on a display or a screen by a user comprising the steps of:

finding a screen gaze direction on the display or the screen of the user,

determining whether the gaze direction is within at least one activated control region, and

activating scrolling to provide a desired display of information when the gaze direction is within the at least one activated control region.

49. A method of making a system for controlling automatic scrolling of information on a display or a screen comprising the steps of:

providing a computer system coupled to the display or the screen,

coupling an eye scroll control sensor to the computer system for tracking and determining a position of the eye of a user relative to the display or the screen, and

coupling a scroll activating interface system to the computer system and interfacing the scroll activating interface system with the eye scroll control sensor for implementing automatic scrolling based upon the position of the eye of the user relative to an activation area on the display or the screen.