Avionics text entry, cursor control, and display format selection via voice recognition

• US 9,922,651 B1
• Filed: 08/13/2014
• Issued: 03/20/2018
• Est. Priority Date: 08/13/2014
• Status: Active Grant

• Current Assignee: Rockwell Collins, Inc. (Raytheon Technologies Corporation d/b/a RTX)
• Original Assignee: Rockwell Collins, Inc. (Raytheon Technologies Corporation d/b/a RTX)
• Inventors: Nelson, Brent J., Gribble, David A.
• Primary Examiner(s): Thomas-Homescu, Anne L
• Application Number:
  US14/458,838
• Time in Patent Office: 1,315 Days
• CPC Class Codes: B64D 43/00 Arrangements or adaptations of instruments arrangements of cameras B64D47/08; aeronautical measuring instruments per se G01C
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0236 using selection techniques to select from displayed items
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04892 Arrangements for controlling cursor position based on codes indicative of cursor displacements from one discrete location to another, e.g. using cursor control keys associated to different directions or using the tab key arrangements for controlling cursor position based on coordinate signals G06F3/038
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command

Abstract

A method includes detecting an initiation command to begin outputting text data received from a voice recognition system to a text buffer of an aircraft. The method includes recognizing user speech as the text data and outputting the text data to the text buffer for presentation to a user. The method further includes detecting an end command to end outputting the text data received from the voice recognition system to the text buffer. The method also includes receiving a confirmation input from the user, the confirmation input indicating that the user has confirmed accuracy of the text data of the text buffer. The method additionally includes receiving a destination selection from the user, the destination selection indicating a destination selected by the user for the text data of the text buffer. The method further includes outputting the text data of the text buffer to the destination selected by the user.

Claims

1. A method, comprising:

recognizing, via an eye tracking system comprising an eye tracking system processor, a beginning eye tracking command to begin outputting text data received from a voice recognition system implemented in an aircraft to a text buffer on a display based on a current location of a pilot'"'"'s gaze, wherein the text buffer is implemented as at least one of a portion of a cockpit display of the aircraft or a display implemented on a keypad of the aircraft;

recognizing, via the voice recognition system, pilot speech as the text data;

outputting, via the eye tracking system, the text data to the text buffer for graphical presentation to a pilot of the aircraft, wherein the text buffer is configured to temporarily display the text data as textual content to the pilot at least until a destination computing device is selected by the pilot;

detecting, via the voice recognition system, an end command to end outputting the text data received from the voice recognition system to the text buffer;

receiving a confirmation input from the pilot, the confirmation input indicating that the pilot has confirmed accuracy of the text data of the text buffer;

receiving a destination selection from the pilot, the destination selection indicating the destination computing device selected by the pilot for the text data of the text buffer;

outputting the text data of the text buffer to the destination computing device selected by the pilot, the destination computing device including a processor configured to utilize the text data to execute instructions of aircraft function software to perform one or more aircraft function operations, wherein the destination computing device selected by the pilot is a flight management system (FMS) implemented in the aircraft, a weather radar system implemented in the aircraft, an attitude and heading reference system (AHRS) implemented in the aircraft, or a traffic collision avoidance system (TCAS) implemented in the aircraft;

utilizing, by the processor of the destination computing device, the text data to execute the instructions of the aircraft function software to perform the one or more aircraft function operations; and

performing a cursor control operation on a graphical user interface (GUI) of at least one cockpit display of the aircraft, wherein performing the cursor control operation comprises;

recognizing, via the voice recognition system, a cursor control keyword configured to indicate that a voice recognition sequence is for performing the cursor control operation to move a cursor from a current position to a new position;

recognizing, via the voice recognition system, at least two move keywords configured to indicate where to move the cursor; and

performing one or more cursor control operations to move the cursor of the GUI from the current position to a second position based at least on the at least two move keywords.

2. The method of claim 1, further comprising:

recognizing, via the voice recognition system, one or more control keywords;

upon recognizing the one or more control keywords, recognizing pilot speech as a portion of contents of the text buffer to be edited; and

editing the portion of the contents of the text buffer.

3. The method of claim 1, further comprising:

detecting a copy command to perform a copy operation;

recognizing a paste destination for the copy operation; and

copying contents of the text buffer to the paste destination.

4. The method of claim 1, further comprising:

detecting a paste command to perform a paste operation;

recognizing a copy source for the paste operation; and

receiving text data of the copy source to be presented in the text buffer.

5. The method of claim 1, wherein the text buffer comprises the display implemented on the keypad of the aircraft.

6. The method of claim 1, further comprising:

performing operations for selecting a cockpit display format of a graphical user interface (GUI) of at least one cockpit display of the aircraft, wherein the operations include;

recognizing, via the voice recognition system, a cockpit display format keyword configured to indicate that a voice recognition sequence is for selecting the cockpit display format;

recognizing, via the voice recognition system, a content keyword configured to indicate content to display;

receiving a command to conclude a voice recognition sequence for selecting the cockpit display format; and

performing one or more cockpit display format operations to generate and output GUI data to the at least one cockpit display, wherein the GUI data is generated based at least on the content to display, wherein the content to display includes at least one of flight management system (FMS) content, map content, or flight checklist content.

8. The method of claim 1, wherein recognizing, via the voice recognition system, the at least two move keywords configured to indicate where to move the cursor relative to the current position of the cursor further comprises:

recognizing, via the voice recognition system, the at least two move keywords configured to indicate to move the cursor relative to the current position of the cursor in a specified direction by a predetermined increment.

9. The method of claim 1, wherein recognizing, via the voice recognition system, the at least two move keywords configured to indicate where to move the cursor relative to the current position of the cursor further comprises:

recognizing, via the voice recognition system, the at least two move keywords configured to indicate to move the cursor relative to the current position of the cursor from the current position to a second position corresponding to a selectable object.

10. The method of claim 1, further comprising:

performing operations for selecting a cockpit display format of a graphical user interface (GUI) of at least two cockpit displays of the aircraft, wherein the operations include;

recognizing, via a voice recognition system comprising a processor, a cockpit display format keyword configured to indicate that the voice recognition sequence is for selecting the cockpit display format of a GUI of at least two cockpit displays of an aircraft, wherein the voice recognition system is implemented in the aircraft;

recognizing a targeted cockpit display of the at least two cockpit displays for the cockpit display format;

recognizing, via the voice recognition system, a size keyword configured to indicate a size to make a window to display on the targeted cockpit display;

recognizing, via the voice recognition system, a content keyword configured to indicate content to display in the window on the targeted cockpit display;

receiving a command to conclude the voice recognition sequence for selecting the cockpit display format; and

performing one or more cockpit display format operations to generate and output GUI data to the targeted cockpit display of the at least two cockpit displays of the aircraft, wherein the GUI data is generated based at least on the content to display, the size to make the window, and the targeted cockpit display.

11. The method of claim 1, wherein the destination computing device including the processor is configured to utilize the text data to execute aircraft function software to perform one or more aircraft function operations and then output data to memory to be used by the processor for executing instructions of another aircraft function software.

12. The method of claim 1, wherein the destination computing device selected by the pilot is the weather radar system implemented in the aircraft, the attitude and heading reference system (AHRS) implemented in the aircraft, or the traffic collision avoidance system (TCAS) implemented in the aircraft.

7. A method for selecting a cockpit display format of a graphical user interface (GUI) of at least two cockpit displays of an aircraft, comprising:

receiving a command to initiate a voice recognition sequence;

recognizing, via a voice recognition system implemented in an aircraft, a cockpit display format keyword configured to indicate that the voice recognition sequence is for selecting the cockpit display format of a GUI of at least two cockpit displays of the aircraft;

recognizing a targeted cockpit display of the at least two cockpit displays for the cockpit display format, wherein recognizing the targeted cockpit display of the at least two cockpit displays for the cockpit display format further comprises;

recognizing, via the voice recognition system, a cockpit display target keyword configured to indicate the targeted cockpit display of the at least two cockpit displays for the cockpit display format; and

recognizing, via an eye tracking system, an eye tracking cockpit display target command configured to indicate the targeted cockpit display of the at least two cockpit displays for the display format;

recognizing, via the voice recognition system, a size keyword configured to indicate a size to make a window to display on the targeted cockpit display;

recognizing, via the voice recognition system, a content keyword configured to indicate content to display in the window on the targeted cockpit display;

performing a cursor control operation on the GUI, wherein performing the cursor control operation comprises;

recognizing, via the voice recognition system, a cursor control keyword configured to indicate that a voice recognition sequence is for performing the cursor control operation to move a cursor from a current position to a new position;

recognizing, via the voice recognition system, at least one move keyword configured to indicate where to move the cursor; and

performing one or more cursor control operations to move the cursor of the GUI from the current position to a second position based at least on the at least one move keyword;

receiving a command to conclude the voice recognition sequence for selecting the cockpit display format; and

performing one or more cockpit display format operations to generate and output GUI data to the targeted cockpit display of the at least two cockpit displays of the aircraft, wherein the GUI data is generated based at least on the content to display, the size to make the window, and the targeted cockpit display, wherein the content to display includes at least one of map content or flight checklist content.