Bimodal user interface system, device, and method for streamlining a user's interface with an aircraft display unit

• US 10,672,280 B1
• Filed: 09/29/2011
• Issued: 06/02/2020
• Est. Priority Date: 09/29/2011
• Status: Active Grant

• Current Assignee: Rockwell Collins, Inc. (Raytheon Technologies Corporation d/b/a RTX)
• Original Assignee: Rockwell Collins, Inc. (Raytheon Technologies Corporation d/b/a RTX)
• Inventors: Barber, Sarah
• Primary Examiner(s): Olshannikov, Alex
• Application Number:
  US13/248,814
• Time in Patent Office: 3,169 Days
• Field of Search: 715702, 715251, 715825, 248702
• CPC Class Codes: G08G 5/0021 located in the aircraft
• CPC Class Codes: G08G 5/0034 Assembly of a flight plan
• CPC Class Codes: G08G 5/0039 Modification of a flight plan

Abstract

Present novel and non-trivial system, device, and method for streamlining a user'"'"'s interface with an aircraft display unit. The system is comprised of a tactile interface device, a voice recognition device, a display unit, and a bimodal interface processor (“BIP”). Both the tactile interface device and the voice recognition device are configured to provide tactile and voice input data to the BIP, and the display unit is configured with at least one page comprised of user-selectable widget(s) and user-enterable widget(s). The BIP is configured to receive the tactile input data corresponding to selections of each user-selectable widget and each user-enterable widget unless the latter has been inhibited by an activation of the user-enterable widget. The BIP is further configured to receive voice input data corresponding to each user-enterable widget unless the user-enterable widget has not been activated. The activation of each user-enterable widget is controlled through tactile input data.

Claims

1. A bimodal user interface system employed to streamline a pilot'"'"'s interface with a display unit by selectively restricting the availability and use of tactile and voice modes, comprising:

a display unit configured to present an image comprised of at least one enterable widget and at least one selectable widget and configured for bimodal entering of a flight plan by a pilot, where

each enterable widget and each selectable widget are graphical user interfaces for facilitating a pilot'"'"'s interaction,

each enterable widget and each selectable widget include a tactile mode and a voice mode, and

each enterable widget is either an inactive enterable widget or an active enterable widget, where

an inactive enterable widget is a widget with its tactile mode activated and voice mode deactivated, such that the inactive enterable widget is responsive to pilot input received via a tactile input device only, and

an active enterable widget is a widget with its tactile mode deactivated and voice mode activated, such that the active enterable widget is responsive to pilot input received via a voice input device only; and

a bimodal interface processor including at least one processor coupled to a non-transitory processor-readable medium storing processor-executable code and configured to;

generate image data representative of the image presented by the display unit;

receive, via an inactive enterable widget included in the image, selection data representative of its selection by the pilot to begin the entering of at least first and final waypoints of the flight plan, whereupon

the selected inactive enterable widget changes to a first active enterable widget;

receive, via the first active enterable widget only, first input data representative of the first waypoint of the flight plan being entered, whereupon

the entering of the first waypoint is presented to the pilot;

receive second input data representative of a completion of the first waypoint being entered, whereupon

the first waypoint is entered into the flight plan and the voice mode of the first active enterable widget is deactivated, where

the second input data is received via the first active enterable widget in response to a predefined voice command separate from the first waypoint being entered, via an inactive enterable widget in response to a tactile selection and into which no waypoint has been entered, or via a selectable widget in response to a tactile selection only of an auto-completion entry in a pop-up widget;

receive, via a second active enterable widget only, third input data representative of the final waypoint of the flight plan being entered, whereupon the entering of the final waypoint is presented to the pilot;

receive fourth input data representative of a completion of the entering of the final waypoint, whereupon

the final waypoint is entered into the flight plan and the voice mode of the second active enterable widget is deactivated, where

the fourth input data is received via the second active enterable widget in response to a predefined voice command separate from the first waypoint being entered, via an inactive enterable widget in response to a tactile selection and into which no waypoint has been entered, or via a selectable widget in response to a tactile selection only of an auto-completion entry in a pop-up widget; and

receive, via a selectable widget in response to a tactile selection only, fifth input data representative of a completion of the entering of the flight plan, whereby

a user system of the flight plan is notified of the completion.

2. The system of claim 1, wherein the tactile input device is a screen of the display unit.

3. The system of claim 1, wherein the voice input device employs a voice recognition system.

4. The system of claim 1, wherein

the bimodal interface processor is further configured to:

receive, via at least one third active enterable widget only and prior to the third input data being received, sixth input data representative of at least one waypoint in between the first and final waypoints being entered, whereupon the entering of each waypoint of the at least one waypoint is presented to the pilot; and

receive seventh input data representative of a completion of each waypoint of the at least one waypoint being entered, whereupon

each waypoint of the at least one waypoint is entered into the flight plan and the voice mode of the its active enterable widget is deactivated, where

the seventh input data is received via the at least one third active enterable widget in response to a predefined voice command separate from its waypoint being entered, via an inactive enterable widget in response to a tactile selection and into which no waypoint has been entered, or via a selectable widget in response to a tactile selection only of an auto-completion entry in a pop-up widget.

5. The system of claim 1, wherein

the display unit is further configured to present a second image comprised of at least one second selectable widget, a third selectable widget for each second selectable widget, and at least one third active enterable widget for the third selectable widget and configured for revising the flight plan, where

each second selectable widget is responsive to pilot input received via the tactile input device only, and

each third selectable widget is responsive to pilot input received via the tactile input device or the voice input device;

and

the bimodal interface processor is further configured to;

generate image data representative of the second image presented by the display unit;

receive, via a second selectable widget only, sixth input data representative of a symbol being selected, whereupon

at least one predefined waypoint command is presented to the pilot in a third selectable widget;

receive, via the third selectable widget only, seventh input data representative of one predefined waypoint command for the selected symbol, whereupon

flight plan revision information is presented to the pilot in a third active enterable widget;

receive, via the third active enterable widget only, eighth input data representative of flight plan revision information being entered, whereupon

the entering of the flight plan revision information is presented to the pilot; and

receive ninth input data representative of a completion of the entering of the flight plan revision information, whereby

the user system of the flight plan is notified of the completion of the entering of the flight plan revision information.

6. A bimodal user interface device employed to streamline a pilot'"'"'s interface with a display unit by selectively restricting the availability and use of tactile and voice modes, comprising:

a bimodal interface processor including at least one processor coupled to a non-transitory processor-readable medium storing processor-executable code and configured to;

generate image data representative of an image comprised of at least one enterable widget and at least one selectable widget presented by a display unit and configured for bimodal entering of a flight plan by a pilot, where

each enterable widget and each selectable widget are graphical user interfaces for facilitating a pilot'"'"'s interaction,

each enterable widget and each selectable widget include a tactile mode and a voice mode, and

each enterable widget is either an inactive enterable widget or an active enterable widget, where

an inactive enterable widget is a widget with its tactile mode activated and voice mode deactivated, such that the inactive enterable widget is responsive to pilot input received via a tactile input device only, and

an active enterable widget is a widget with its tactile mode deactivated and voice mode activated, such that the active enterable widget is responsive to pilot input received via a voice input device only;

receive, via an inactive enterable widget included in the image, selection data representative of its selection by the pilot to begin the entering of at least first and final waypoints of the flight plan, whereupon

the selected inactive enterable widget changes to a first active enterable widget;

receive, via the first active enterable widget only, first input data representative of the first waypoint of a flight plan being entered, whereupon

the entering of the first waypoint is presented to the pilot;

receive second input data representative of a completion of the first waypoint being entered, whereupon

the first waypoint is entered into the flight plan and the voice mode of the first active enterable widget is deactivated, where

the second input data is received via the first active enterable widget in response to a predefined voice command separate from the first waypoint being entered, via an inactive enterable widget in response to a tactile selection and into which no waypoint has been entered, or via a selectable widget in response to a tactile selection only of an auto-completion entry in a pop-up widget;

receive, via a second active enterable widget only, third input data representative of the final waypoint of the flight plan being entered, whereupon

the entering of the final waypoint is presented to the pilot;

receive fourth input data representative of a completion of the entering of the final waypoint, whereupon

the final waypoint is entered into the flight plan and the voice mode of the second active enterable widget is deactivated, where

the fourth input data is received via the second active enterable widget in response to a predefined voice command separate from the first waypoint being entered, via an inactive enterable widget in response to a tactile selection and into which no waypoint has been entered, or via a selectable widget in response to a tactile selection only of an auto-completion entry in a pop-up widget; and

receive, via a selectable widget in response to a tactile selection only, fifth input data representative of a completion of the entering of the flight plan, whereby

a user system of the flight plan is notified of the completion.

7. The device of claim 6, wherein the tactile input device is a screen of the display unit.

8. The device of claim 6, wherein the voice input device employs a voice recognition system.

9. The device of claim 6, wherein

the bimodal interface processor is further configured to:

receive, via at least one third active enterable widget only and prior to the third input data being received, sixth input data representative of at least one waypoint in between the first and final waypoints being entered, whereupon

the entering of each waypoint of the at least one waypoint is presented to the pilot; and

receive seventh input data representative of a completion of each waypoint of the at least one waypoint being entered, whereupon

each waypoint of the at least one waypoint is entered into the flight plan and the voice mode of the its active enterable widget is deactivated, where

the seventh input data is received via the at least one third active enterable widget in response to a predefined voice command separate from its waypoint being entered, via an inactive enterable widget in response to a tactile selection and into which no waypoint has been entered, or via a selectable widget in response to a tactile selection only of an auto-completion entry in a pop-up widget.

10. The device of claim 6, wherein

the bimodal interface processor is further configured to:

generate image data representative of a second image comprised of at least one second selectable widget, a third selectable widget for each second selectable widget, and at least one third active enterable widget for the third selectable widget presented by the display unit and configured for revising the flight plan, where

each second selectable widget is responsive to pilot input received via the tactile input device only, and

each third selectable widget is responsive to pilot input received via the tactile input device or the voice input device;

receive, via a second selectable widget only, sixth input data representative of a symbol being selected, whereupon

at least one predefined waypoint command is presented to the pilot in a third selectable widget;

receive, via the third selectable widget only, seventh input data representative of one predefined waypoint command for the selected symbol, whereupon

flight plan revision information is presented to the pilot in a third active enterable widget;

receive, via the third active enterable widget only, eighth input data representative of flight plan revision information being entered, whereupon

the entering of the flight plan revision information is presented to the pilot; and

receive ninth input data representative of a completion of the entering of the flight plan revision information, whereby

the user system of the flight plan is notified of the completion of the entering of the flight plan revision information.

11. A bimodal user interface method employed to streamline a pilot'"'"'s interface with a display unit by selectively restricting the availability and use of tactile and voice modes, comprising:

generating, by a bimodal interface processor including at least one processor coupled to a non-transitory processor-readable medium storing processor-executable code and via a first active enterable widget only, image data representative of an image comprised of at least one enterable widget and at least one selectable widget presented by a display unit and configured for bimodal entering of a flight plan by a pilot, where

each enterable widget and each selectable widget are graphical user interfaces for facilitating a pilot'"'"'s interaction,

each enterable widget and each selectable widget include a tactile mode and a voice mode,

each enterable widget is either an inactive enterable widget or an active enterable widget, where

an inactive enterable widget is a widget with its tactile mode activated and voice mode deactivated, such that the inactive enterable widget is responsive to pilot input received via a tactile input device only, and

an active enterable widget is a widget with its tactile mode deactivated and voice mode activated, such that the active enterable widget is responsive to pilot input received via a voice input device only;

receiving, via an inactive enterable widget included in the image, selection data representative of its selection by the pilot to begin the entering of at least first and final waypoints of the flight plan, whereupon

the selected inactive enterable widget changes to a first active enterable widget;

receiving, via the first active enterable widget only, first input data representative of the first waypoint of the flight plan being entered, whereupon

the entering of the first waypoint is presented to the pilot;

receiving second input data representative of a completion of the first waypoint being entered, whereupon

the first waypoint is entered into the flight plan and the voice mode of the first active enterable widget is deactivated, where

the second input data is received via the first active enterable widget in response to a predefined voice command separate from the first waypoint being entered, via an inactive enterable widget in response to a tactile selection and into which no waypoint has been entered, or via a selectable widget in response to a tactile selection only of an auto-completion entry in a pop-up widget;

receiving, via a second active enterable widget only, third input data representative of the final waypoint of the flight plan being entered, whereupon

the entering of the final waypoint is presented to the pilot;

receiving fourth input data representative of a completion of the entering of the final waypoint, whereupon

the final waypoint is entered into the flight plan and the voice mode of the second active enterable widget is deactivated, where

the fourth input data is received via the second active enterable widget in response to a predefined voice command separate from the first waypoint being entered, via one inactive enterable widget in response to a tactile selection and into which no waypoint has been entered, or via a selectable widget in response to a tactile selection only of an auto-completion entry in a pop-up widget; and

receiving, via a selectable widget in response to a tactile selection only, fifth input data representative of a completion of the entering of the flight plan, whereby

a user system of the flight plan is notified of the completion.

12. The method of claim 11, wherein the tactile input device is a screen of the display unit.

13. The method of claim 11, wherein the voice input device employs a voice recognition system.

14. The method of claim 11, further comprising:

receiving, via at least one third active enterable widget only and prior to the third input data being received, sixth input data representative of at least one waypoint in between the first and final waypoints being entered, whereupon

the entering of each waypoint of the at least one waypoint is presented to the viewer; and

receiving seventh input data representative of a completion of each waypoint of the at least one waypoint being entered, whereupon

each waypoint of the at least one waypoint is entered into the flight plan and the voice mode of the its active enterable widget is deactivated, where

the seventh input data is received via the at least one third active enterable widget in response to a predefined voice command separate from its waypoint being entered, via an inactive enterable widget in response to a tactile selection and into which no waypoint has been entered, or via a selectable widget in response to a tactile selection only of an auto-completion entry in a pop-up widget.

15. The method of claim 11, further comprising:

generating image data representative of a second image comprised of at least one second selectable widget, a third selectable widget for each second selectable widget, and at least one third active enterable widget for the third selectable widget presented by the display unit and configured for revising the flight plan, where

each second selectable widget is responsive to pilot input received via the tactile input device only, and

each third selectable widget is responsive to pilot input received via the tactile input device or the voice input device; and

receiving, via a second selectable widget only, sixth input data representative of a symbol being selected, whereupon

at least one predefined waypoint command is presented to the pilot in a third selectable widget;

receiving, via the third selectable widget only, seventh input data representative of one predefined waypoint command for the selected symbol, whereupon

flight plan revision information is presented to the pilot in a third active enterable widget;

receiving, via the third active enterable widget only, eighth input data representative of flight plan revision information being entered, whereupon

the entering of the flight plan revision information is presented to the pilot; and

receiving ninth input data representative of a completion of the entering of the flight plan revision information, whereby

the user system of the flight plan is notified of the completion of the entering of the flight plan revision information.