Near-to-eye tracking for adaptive operation

• US 8,552,850 B2
• Filed: 02/17/2010
• Issued: 10/08/2013
• Est. Priority Date: 02/17/2010
• Status: Active Grant

• Current Assignee: Honeywell International Inc.
• Original Assignee: Honeywell International Inc.
• Inventors: De Mers, Robert E., Best, Carl A., Cupero, Frank
• Primary Examiner(s): Lim, Steven
• Assistant Examiner(s): Knox, Kaleria
• Application Number:
  US12/707,293
• Publication Number:
  US 20110199202A1
• Time in Patent Office: 1,329 Days
• Field of Search: 340/945, 340/576, 340/439, 340/546, 340/156, 345/7, 341/121, 351/210, 351/246, 359/630
• US Class Current: 340/439
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: A61B 5/163 by tracking eye movement, gaze, or pupil change
• CPC Class Codes: A61B 5/18 for vehicle drivers or machine operators
• CPC Class Codes: G01C 23/00 Combined instruments indicating more than one navigational value, e.g. for aircraft; Combined measuring devices for measuring two or more variables of movement, e.g. distance, speed or acceleration
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G08B 21/06 indicating a condition of sleep, e.g. anti-dozing alarms
• CPC Class Codes: G08G 5/0021 located in the aircraft

Abstract

A method and a system are provided for adaptive vehicle operation. The method includes the steps of sensing a position and an orientation of a near-to-eye display device worn by the vehicle operator, determining a direction of gaze of the vehicle operator in response to the position and the orientation of the near-to-eye display device, and selectively generating an alert signal in response to the direction of gaze of the vehicle operator.

Claims

1. A method of alerting a vehicle operator comprising:

sensing a position and an orientation of a near-to-eye display device worn by the vehicle operator;

determining a direction of gaze of the vehicle operator in response to the position and the orientation of the near-to-eye display device;

selectively generating an alert signal in response to the direction of gaze of the vehicle operator; and

estimating an attention level of the vehicle operator in response to the determined direction of gaze and an amount of change of the direction of gaze during a predetermined time period, and wherein the step of selectively generating the alert comprises generating an inattention alert signal in response to the estimated attention level being below a threshold attention level.

2. The method in accordance with claim 1 wherein the alert signal is an inattention alert signal, and wherein the step of selectively generating the alert signal comprises selectively generating the inattention alert signal further in response to measurement of passing of a selected one of a plurality of predetermined time periods, each of the plurality of predetermined time periods associated with one of a plurality of vehicle operation modes.

3. The method in accordance with claim 1 further comprising storing information corresponding to the direction of gaze of the vehicle operator.

4. The method in accordance with claim 1 wherein the step of sensing the position and the orientation of the near-to-eye display device comprises sensing using at least one sensor.

5. The method in accordance with claim 1 wherein the step of sensing the orientation of the near-to-eye display device comprises sensing an output of at least one infrared light emitting diode affixed to the near-to-eye display device.

6. The method in accordance with claim 1 further comprising the step of initiating a change in system operation in response to the determined direction of gaze of the vehicle operator.

7. The method in accordance with claim 1 wherein the step of selectively generating the alert signal comprises initiating an alert with a selected one of a plurality of alerting devices, the selected one of the plurality of alerting devices selected at least in part in response to the determined direction of gaze of the vehicle operator.

8. The method in accordance with claim 7 wherein the plurality of alerting devices includes at least one visual alerting device and at least one audible alerting device, and wherein the step of selectively generating the alert signal comprises initiating the alert with a selected one of the at least one visual alerting device or a selected one of the at least one audible alerting device in response to the determined direction of gaze of the vehicle operator.

9. The method in accordance with claim 1 wherein the step of selectively generating the alert signal comprises:

initiating an alert with an alerting device when the direction of gaze of the vehicle operator is determined to be away from a display monitor; and

initiating provision of alert information on the display monitor when the direction of gaze of the vehicle operator is determined to be towards the display monitor.

10. An attention monitoring system coupled to a near-to-eye display system which senses a position and orientation of a near-to-eye display device and generates focus data corresponding to a direction of gaze of a vehicle operator in response to the position and the orientation of the near-to-eye display device, the vehicle monitoring system comprising:

a plurality of alerting devices for providing sensible alerts to the vehicle operator; and

a controller coupled to the near-to-eye display system for receiving the focus data therefrom and coupled to the plurality of alerting devices, providing an alert signal to a selected one of the plurality of alerting devices in response to the focus data, and estimating an attention level of the vehicle operator in response to the determined direction of gaze and an amount of change of the direction of gaze during a predetermined time period, and wherein the step of selectively generating the alert comprises generating an inattention alert signal in response to the estimated attention level being below a threshold attention level.

11. The attention monitoring system in accordance with claim 10 further comprising a memory device for storing a plurality of predetermined time periods, wherein each of the plurality of predetermined time periods is associated with one of a plurality of vehicle operation modes, and wherein the controller is further coupled to the memory device for generating an inattention alert signal and providing the inattention alert signal to one of the plurality of alerting devices in response to measurement of passing of a selected one of the plurality of predetermined time periods, the selected one of the plurality of predetermined time periods selected in response to a present vehicle operation mode.

12. The attention monitoring system in accordance with claim 10 further comprising a black box storage device, wherein the controller is coupled to the black box storage device and stores the focus data in the black box storage device as it is received from the near-to-eye-display system.

13. A monitoring system comprising:

a near-to-eye display device worn by an operator of a vehicle;

one or more sensors for monitoring a position and an orientation of the near-to-eye display device; and

a controller coupled to the one or more sensors for determining a direction of gaze of the operator of the vehicle in response to the position and the orientation of the near-to-eye display device, for selectively generating an alert signal in response to the direction of gaze of the operator of the vehicle, and for estimating an attention level of the vehicle operator in response to the determined direction of gaze and an amount of change of the direction of gaze during a predetermined time period, and wherein the step of selectively generating the alert comprises generating an inattention alert signal in response to the estimated attention level being below a threshold attention level.

14. The monitoring system in accordance with claim 13 further comprising a data storage device coupled to the controller for storing information corresponding to the direction of gaze of the operator.

15. The monitoring system in accordance with claim 13 further comprising:

a memory device coupled to the controller for storing a plurality of predetermined monitoring times, each of the plurality of predetermined monitoring times corresponding to one of a plurality of vehicle operation modes of the vehicle; and

an inattention alerting device for presenting an operator sensible alert in response to receiving an inattention signal from the controller,

wherein the controller determines inattention of the operator of the vehicle and provides the inattention signal to the inattention alerting device in response to the direction of gaze of the operator determined while monitoring the position and the orientation of the near-to-eye display device during a selected one of the plurality of predetermined monitoring times, the controller selecting the one of the plurality of predetermined monitoring times in response to a present vehicle operation mode of the vehicle.

16. The monitoring system in accordance with claim 13 further comprising a plurality of alerting devices for presenting operator sensible alerts when activated by an alert signal from the controller, wherein the controller selects which of the plurality of alerting devices to provide the alert signal to in response to the direction of gaze of the operator of the vehicle.

17. The monitoring system in accordance with claim 16 wherein the plurality of operator alerting devices comprise at least one visual alerting device and at least one audible alerting device.

18. The monitoring system in accordance with claim 16 further comprising a display for displaying visible information to the operator of the vehicle, wherein the controller is coupled to the display and provides an alert to the operator by providing the alert signal to one of the operator alerting devices in response to determining that the direction of gaze of the operator is away from the display, the controller providing the alert to the user by providing signals to the display to present an alert on the display in response to determining that the direction of gaze of the operator is toward the display.

19. The monitoring system in accordance with claim 13 further comprising a black box storage device, wherein the controller is coupled to the black box storage device and stores data in the black box storage device corresponding to the direction of gaze of the operator of the vehicle.