Methods and apparatus for determining pilot awareness of a system-initiated change based on scanning behavior

• US 9,227,736 B2
• Filed: 03/07/2014
• Issued: 01/05/2016
• Est. Priority Date: 03/07/2014
• Status: Active Grant

• Current Assignee: Honeywell International Inc.
• Original Assignee: Honeywell International Inc.
• Inventors: Whitlow, Stephen, De Mers, Robert E.
• Primary Examiner(s): ARTHUR JEANGLAUDE, GERTRUDE
• Application Number:
  US14/200,179
• Publication Number:
  US 20150251771A1
• Time in Patent Office: 669 Days
• Field of Search: 701/3, 701/4, 701/14, 340/5.82, 340/945
• US Class Current: 1/1
• CPC Class Codes: B64D 45/00 Aircraft indicators or protectors not otherwise provided for camouflage F41H3/00
• CPC Class Codes: B64D 45/0056 detecting passenger or crew behavior by sensors, e.g. biometrics
• CPC Class Codes: B64D 45/0059 by communicating emergency situations to ground control or between crew members
• CPC Class Codes: G01C 23/00 Combined instruments indicating more than one navigational value, e.g. for aircraft; Combined measuring devices for measuring two or more variables of movement, e.g. distance, speed or acceleration
• CPC Class Codes: G06F 21/32 using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G08B 21/06 indicating a condition of sleep, e.g. anti-dozing alarms
• CPC Class Codes: G08G 5/0021 located in the aircraft
• CPC Class Codes: G08G 5/0052 for cruising combined instruments indicating more than one navigational value G01C23/00

Abstract

A method for assessing and responding to a level of awareness of a flight crew member onboard an aircraft is provided. The method assesses an eye-scanning behavior of the flight crew member to determine whether the flight crew member is aware of a system-initiated change to a flight parameter; and when the flight crew member is not aware of the system-initiated change, orients the flight crew member to the system-initiated change.

Claims

1. A method for assessing and responding to a level of awareness of a flight crew member onboard an aircraft, the method comprising:

assessing an eye-scanning behavior of the flight crew member to determine whether the flight crew member is aware of a system-initiated change to a flight parameter, the eye-scanning behavior comprising eye position, gaze direction, and eye movements of the flight crew member; and

when the flight crew member is not aware of the system-initiated change, orienting the flight crew member to the system-initiated change.

2. The method claim 1, wherein the method further comprises:

monitoring an avionics communication bus onboard the aircraft;

identifying a modification to an existing flight mode, via the monitored avionics communication bus; and

determining that the modification was automatically executed;

wherein the automatically executed modification comprises the system-initiated change; and

wherein the flight mode comprises a set of control settings for an automatic flight system (AFS) which controls the aircraft in pitch, roll and yaw axes as well as thrust settings.

3. The method of claim 2, further comprising:

when the flight crew member is aware of the system-initiated change, continuing to monitor the avionics communications bus onboard the aircraft.

4. The method of claim 1, wherein the assessing step further comprises:

identifying an eye-scanning region, onboard the aircraft, related to the system-initiated change to a flight parameter;

calculating a minimum amount of time that indicates awareness of the flight crew member regarding the system-initiated change; and

assessing the eye-scanning behavior of the flight crew member to determine whether the flight crew member fixated on the eye-scanning region for at least the minimum amount of time.

5. The method of claim 4, wherein the assessing step further comprises:

assessing the eye-scanning behavior for a time period of interest, to determine whether the flight crew member fixated on the eye-scanning region for the required eye-fixation time during the time period of interest;

wherein the time period of interest begins upon occurrence of the system-initiated change and the end point comprises a dynamic value that is function of time sensitivity of detected change, current estimate of pilot workload, and complexity of display elements that depicts the change.

6. The method of claim 5, wherein the assessing step further comprises:

identifying a plurality of discrete periods of time, during the time period of interest, in which the flight crew member fixates on the eye-scanning region;

summing the plurality of discrete periods of time to obtain a total eye-fixation time; and

evaluating the total-eye fixation time to determine whether the flight crew member fixated on the eye-scanning region for the required eye-fixation time during the time period of interest.

7. The method of claim 4, wherein the calculating step further comprises:

obtaining a baseline eye-fixation time; and

adjusting the baseline eye-fixation time to determine the required eye-fixation time, based on at least one factor;

wherein the at least one factor indicates an ability of the flight crew member to become aware of the system-initiated change.

8. The method of claim 7, wherein the adjusting step further comprises:

determining a current phase of flight, wherein the at least one factor comprises the determined current phase of flight; and

adjusting the baseline eye-fixation time based on the determined current phase of flight.

9. The method of claim 7, wherein the eye-scanning region comprises a display element;

and

wherein the adjusting step further comprises;

determining complexity of the display element, wherein the at least one factor comprises the determined complexity of the display element; and

adjusting the baseline eye-fixation time based on the determined complexity of the display element.

10. The method of claim 7, wherein the adjusting step further comprises:

estimating fatigue of the flight crew member, wherein the at least one factor comprises the estimated fatigue of the flight crew member; and

adjusting the baseline eye-fixation time based on the estimated fatigue of the flight crew member.

11. A system for determining and responding to a level of awareness of a flight crew member with regard to system-initiated changes onboard an aircraft, the system comprising:

a flight parameter analysis module, configured to identify an automatically executed conversion of an existing flight mode of the aircraft;

an eye-tracking module, configured to sense an eye-scanning behavior of the flight crew member inside the aircraft, the eye-scanning behavior comprising eye position, gaze direction and eye movements of the flight crew member;

a data analysis module, configured to evaluate the eye-scanning behavior of the flight crew member to determine awareness of the flight crew member of the automatically executed conversion of the existing flight mode; and

a notification module, configured to orient the flight crew member to the automatically executed conversion of the existing flight mode, when the data analysis module determines that the flight crew member is unaware of the automatically executed conversion.

12. The system of claim 11, wherein the data analysis module is further configured to:

identify a display element related to the automatically executed conversion of an existing flight mode;

calculate a required eye-fixation time associated with the identified display element; and

evaluate the eye-scanning behavior of the flight crew member to determine whether the flight crew member fixated on the identified display element for the required eye-fixation time;

wherein the required eye-fixation time comprises a minimum amount of time that indicates awareness of the flight crew member regarding the system-initiated change.

13. The system of claim 11, wherein the notification module is further configured to orient the flight crew member to the automatically executed conversion of an existing flight mode using a visual or auditory alert.

14. A non-transitory, computer-readable medium containing instructions thereon, which, when executed by a processor, perform a method comprising:

determining, with an onboard monitoring subsystem of an aircraft, that a flight crew member is unaware of a system-initiated change to an operating parameter by analyzing eye-scanning behavior of the flight crew member, wherein the eye-scanning behavior comprises eye position, gaze direction and eye movements of the flight crew member; and

presenting a notification of the system-initiated change to the flight crew member.

15. The non-transitory, computer-readable medium of claim 14, wherein the determining step further comprises:

detecting occurrence of the system-initiated change;

identifying a region in the aircraft associated with the system-initiated change, wherein awareness of the system-initiated change requires visual attention to the region; and

sensing a lack of user visual attention toward the region.

16. The non-transitory, computer-readable medium of claim 14, wherein the determining step further comprises:

identifying a display element related to the system-initiated change to an operating parameter;

calculating a required eye-fixation time associated with the identified display element; and

assessing the eye-scanning behavior of the flight crew member to determine whether the flight crew member fixated on the identified display element for the required eye-fixation time;

wherein the required eye-fixation time comprises a minimum amount of time that indicates awareness of the flight crew member regarding the system-initiated change.

17. The non-transitory, computer-readable medium of claim 16, wherein the calculating step further comprises:

obtaining a baseline eye-fixation time; and

adjusting the baseline eye-fixation time to determine the required eye-fixation time, based on at least one factor;

wherein the at least one factor indicates an ability of the flight crew member to become aware of the system-initiated change.

18. The non-transitory, computer-readable medium of claim 17, wherein the adjusting step further comprises:

determining a current phase of flight, wherein the at least one factor comprises the determined current phase of flight; and

adjusting the baseline eye-fixation time based on the determined current phase of flight.

19. The non-transitory, computer-readable medium of claim 17, wherein the adjusting step further comprises:

determining complexity of the identified display element, wherein the at least one factor comprises the determined complexity of the identified display element; and

adjusting the baseline eye-fixation time based on the determined complexity of the identified display element.

20. The non-transitory, computer-readable medium of claim 17, wherein the adjusting step further comprises:

estimating fatigue of the flight crew member, wherein the at least one factor comprises the estimated fatigue of the flight crew member; and

adjusting the baseline eye-fixation time based on the estimated fatigue of the flight crew member.