Multi-touch cockpit interface for controlling aircraft systems

• US 9,703,476 B1
• Filed: 05/30/2012
• Issued: 07/11/2017
• Est. Priority Date: 12/23/2010
• Status: Active Grant

• Current Assignee: The Boeing Co.
• Original Assignee: The Boeing Co.
• Inventors: Pappas, Nicholas William, Nikolic, Mark Ivan
• Primary Examiner(s): Tryder, Gregory J
• Assistant Examiner(s): Sonners, Scott E
• Application Number:
  US13/484,029
• Time in Patent Office: 1,868 Days
• Field of Search: 345661
• CPC Class Codes: B64D 43/00 Arrangements or adaptations of instruments arrangements of cameras B64D47/08; aeronautical measuring instruments per se G01C
• CPC Class Codes: G06F 2203/04803 Split screen, i.e. subdividing the display area or the window area into separate subareas
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus

Abstract

A system and method for controlling an aircraft system. An operator interaction with a control is received. The control comprises an image representing a hardware control for controlling the aircraft system. The image representing the hardware control is changed to represent a change in a state of the control in response to receiving the operator interaction. The aircraft system is operated in the manner indicated by the state of the control as represented in the image.

Claims

1. A method for controlling an aircraft system, comprising:

displaying an image, representing a control for the aircraft system, on a touch screen;

continuously enhancing a hit accuracy characteristic of the image, via adjusting a touch recognition threshold of the image, via using instructions in a controller and based upon the controller receiving an altitude and a speed, of a system on an aircraft;

receiving, on the touch screen, a first input, the touch screen configured for;

automatically selecting, via determining a position and an orientation in the aircraft of the touch screen, an aircraft system displayed on and in communication with the touch screen;

virtually partitioning the touch screen;

simultaneously recognizing a first input, on a first partition of the touch screen as being from a first source on a first side of the touch screen, and a second input, on a second partition of the touch screen as being from a second source on a second side of the touch screen, via recognizing an orientation, relative to the touch screen, of the first source and of the second source, respectively, and simultaneously processing the first input and the second input;

determining, via the second input being along an edge of the second partition and coming from the first source on the first side of the touch screen, the second input as an accidental input; and

responsive to recognizing the accidental input, at least one of;

correcting, and ignoring, the accidental input;

transmitting the first input from the touch screen to an interface controller within the controller for an aircraft comprising the aircraft system;

changing, based upon the first input, a state of the control to an active state and indicating activating the control via increasing a size of the image via program code in the interface controller communicating with the touch screen;

subsequent to activating the control, the interface controller receiving a subsequent input to change the state of the control; and

an aircraft system controller within the controller operating the aircraft system in a manner indicated by the state of the control.

2. The method of claim 1, further comprising:

the image representing a dial; and

the first input comprising selecting the dial for a selected duration.

3. The method of claim 1, further comprising:

the image representing a dial;

changing a rotational position of the image representing the dial to a new rotational position in response to the first input touching an edge of the image representing the dial at a position; and

indicating the new rotational position and indicating the state of the control via the rotational position of the image representing the dial.

4. The method of claim 1, further comprising:

the image representing a dial;

changing a rotational position of the image representing the dial to a new rotational position in response via a sliding input along an edge of the image representing the dial to indicate an amount of rotation of the image representing the dial; and

indicating, via the rotational position of the image representing the dial, the state of the control represented by the image representing the dial.

5. The method of claim 1, further comprising:

the image representing a switch;

the controller interface changing the image representing the switch from an image of the switch in a first position to an image of the switch in a second position in response to receiving an input touching the image representing the switch at the second position; and

the controller interface changing the image representing the switch from the image of the switch in the second position to the image of the switch in the first position in response to stopping the input touching the image representing the switch at the second position, wherein the state of the control represented by the image representing the switch is indicated by the position of the switch in the image representing the switch.

6. The method of claim 1, wherein the aircraft system is a radio on the aircraft.

7. An apparatus, that comprises:

a touch screen display located in a cockpit of an aircraft and configured to;

transition, responsive to a reception on the touch screen display of a 4-finger swipe right, from a map to a plan image on the touch screen display;

transition, responsive to a reception on the touch screen display of a 4-finger swipe left, from a plan to a map image on the touch screen display; and

present, responsive to a six-plus fingers down swipe on the touch screen display, a keyboard on the touch screen display; and

a controller comprising;

processor unit, an interface controller, and an aircraft system controller, the interface controller comprising a data processing device comprising instructions programmed to;

display on the touch screen display an image that represents a control for an aircraft system;

automatically select, based upon a position and an orientation in the aircraft of the touch screen display, an aircraft system displayed on and in communication with the touch screen display;

virtually partition the touch screen display into multiple partitions;

simultaneously recognize;

a first input, on a first partition of the touch screen display as being from a first source on a first side of the touch screen display; and

a second input, on a second partition of the touch screen display as being from a second source on a second side of the touch screen display, based upon a recognition of an orientation, relative to the touch screen display, of the first source and of the second source, respectively, and simultaneously process the first input and the second input;

continuously enhance a hit accuracy characteristic of the image, via instructions programmed to;

adjust a touch recognition threshold of the image based upon the controller receiving an altitude and a speed of a system on the aircraft;

determine, based upon the second input being along an edge of the second partition and coming from the first source on the first side of the touch screen display, the second input as an accidental input; and

responsive to a recognition of the accidental input, at least one of;

correct, and ignore, the accidental input;

receive from the touch screen display, an indication of the first input to the control, and, responsive to the first input, change a state of the control; and

change the image to represent the change in the state of the control in response to receiving the indication; and

the aircraft system controller comprising instructions programmed to operate the aircraft system in a manner indicated by the state of the control.

8. The apparatus of claim 7, further comprising the image representing a dial, and wherein the controller comprises instructions programmed to change a size of the image representing the dial in response to receiving the indication for a selected duration.

9. The apparatus of claim 8, further comprising the controller comprising instructions programmed to increase the size of the image representing the dial to a larger size and to change the state of the control corresponding to the image representing the dial to an active state when the image representing the dial is changed to the larger size, such that with the control in the active state, the controller interface comprises instructions programmed to change another state of the control based upon input from the touch screen display.

10. The apparatus of claim 7, further comprising the image representing a dial and the interface controller comprising instructions programmed to change a rotational position of the image representing the dial to a new rotational position in response to inputs that touch an edge of the image representing the dial on the touch screen display at a position indicating the new rotational position for a selected duration, wherein the state of the control represented by the image representing the dial is indicated by the rotational position of the image representing the dial.

11. The apparatus of claim 7, further comprising the image representing a dial and the interface controller comprising instructions programmed to change a rotational position of the image representing the dial to a new rotational position in response to an input that slides a touch along an edge of the image representing the dial on the touch screen display to indicate an amount of rotation of the image representing the dial, wherein the state of the control represented by the image representing the dial is indicated by the rotational position of the image representing the dial.

12. The apparatus of claim 7, wherein the control comprises an image representing a switch and the controller comprises instructions programmed to:

change the image representing the switch from an image of the switch in a first position to an image of the switch in a second position in response to an indication of a touch on the image representing the switch on the touch screen display at the second position; and

change the image representing the switch from the image of the switch in the second position to the image of the switch in the first position in response to stopping the indication, wherein the state of the control represented by the image representing the switch is indicated by the position of the switch in the image representing the switch.

13. The apparatus of claim 7, wherein the aircraft system is a radio on the aircraft.

14. An apparatus that comprises:

a touch screen display;

a controller that comprises;

a processor unit, an interface controller, and a system controller, such that the interface controller comprises a data processing device that comprises instructions programmed to;

display on the touch screen display a number of controls for controlling a radio, wherein the number of controls comprises images selected from images of dials, images of switches, images of buttons, and images of sliders;

automatically select, based upon a position and an orientation in an aircraft of the touch screen display, an aircraft system displayed on and in communication with the touch screen display;

virtually partition the touch screen display into multiple partitions;

simultaneously recognize;

a first input, on a first partition of the touch screen display as being from a first source on a first side of the touch screen display; and

a second input, on a second partition of the touch screen display as being from a second source on a second side of the touch screen display, based upon a recognition of an orientation, relative to the touch screen display, of the first source and of the second source, respectively, and simultaneously process the first input and the second input;

continuously enhance a hit accuracy characteristic of each of the images, via instructions programmed to;

adjust a touch recognition threshold of each of the images based upon the controller receiving, from a system, an altitude and a speed, of an aircraft;

determine, based upon the second input being along an edge of the second partition and coming from the first source on the first side of the touch screen display, the second input as an accidental input;

responsive to a recognition of the accidental input, at least one of;

correct, and ignore, the accidental input;

remove, responsive to receipt of a mode-less gesture, a menu image on the touch screen display; and

provide instant haptic feedback based upon a validity of the first input; and

receive, from the touch screen display, indications of the first input with the number of controls, and, responsive to the first input, change a state of the number of controls;

change the images to represent a change in the state of the number of controls in response to receiving the first input; and

the system controller comprising instructions programmed to operate the radio in a manner indicated by the state of the number of controls as represented in the images.

15. The apparatus of claim 14, further comprising the controller comprising instructions programmed to change the images to indicate the change in the state of the number of controls to an active state in response to receiving the indications, wherein other states of the number of controls are changeable by the first input with the number of controls when the number of controls is in the active state.

16. The apparatus of claim 14, further comprising the controller comprising instructions programmed to change the images to represent changes in the state of the number of controls in response to receiving indications corresponding to a drag and drop operation input on the touch screen display.

17. The apparatus of claim 14, wherein the touch screen display is located in a cockpit of an aircraft and the radio is located on the aircraft.

18. A method for enhancing a hit accuracy characteristic of a control for an aircraft system, the method comprising:

displaying the control for the aircraft system via displaying an image of the control;

automatically selecting, via determining a position and an orientation of a touch screen in an aircraft, the aircraft system displayed on and in communication with the touch screen;

virtually partitioning the touch screen;

continuously enhancing the hit accuracy characteristic of the image, via instructions programmed for;

adjusting a touch recognition threshold of the image based upon a controller receiving an altitude and a speed of a system on the aircraft;

simultaneously recognizing a first input, on a first partition of the touch screen as being from a first source on a first side of the touch screen, and a second input, on a second partition of the touch screen as being from a second source on a second side of the touch screen, via recognizing an orientation, relative to the touch screen, of the first source and of the second source, respectively, and simultaneously processing the first input and the second input;

determining, via the second input being along an edge of the second partition and coming from the first source on the first side of the touch screen, the second input as an accidental input;

responsive to recognizing the accidental input, at least one of;

correcting, and ignoring, the accidental input;

removing, responsive to receiving a mode-less gesture, a menu on the touch screen; and

providing instant haptic feedback based upon determining a validity of the first input; and

receiving an interaction with the image, wherein the image represents a dial configured to move between a discrete number of rotational positions;

changing the image representing the dial to indicate a change in a rotational position of the dial in response to receiving the interaction; and

operating the aircraft system in a manner indicated by the rotational position of the dial as represented in the image.

19. The method of claim 18 further comprising:

increasing a size of the image, representing the dial, to a larger size in response to receiving the interaction, with the image representing the dial, lasting for a selected duration; and

changing a state of the control to an active state when the image changes to the larger size, wherein the rotational position of the image representing the dial is changeable by an interface controller receiving the interaction with the image representing the dial when the control corresponding to the image representing the dial is in the active state.

20. The method of claim 18, wherein changing the image representing the dial to indicate the change in the rotational position of the dial comprises changing the rotational position of the image representing the dial by one discrete rotational position in a direction in response to receiving the interaction in which an operator touches the image representing the dial on a side of the image representing the dial corresponding to the direction.

21. The method of claim 18, wherein the aircraft system is a radio on the aircraft.

22. An apparatus that comprises:

a touch screen display located in a cockpit of an aircraft;

a controller comprising;

a processor unit, an interface controller, and an aircraft system controller, the interface controller comprising a data processing device comprising instructions programmed to;

display on the touch screen display an image representing a dial, wherein the dial is configured to be moved between a discrete number of rotational positions;

automatically select, based upon a position and an orientation in an aircraft of the touch screen display, an aircraft system displayed on and in communication with the touch screen display;

virtually partition the touch screen display into multiple partitions;

continuously enhance a hit accuracy characteristic of the image, via instructions programmed to at least one of;

adjust a touch recognition threshold of the image based upon the controller receiving an altitude and a speed, of a system on the aircraft;

simultaneously recognize;

a first input, on a first partition of the touch screen display as being from a first source on a first side of the touch screen display; and

a second input, on a second partition of the touch screen display as being from a second source on a second side of the touch screen display, based upon a recognition of an orientation, relative to the touch screen, of the first source and of the second source, respectively, and simultaneously process the first input and the second input;

determine, based upon the second input being along an edge of the second partition and coming from the first source on the first side of the touch screen display, the second input as an accidental input; and

responsive to a recognition of the accidental input, at least one of;

correct, and ignore, the accidental input;

remove, responsive to receipt of a mode-less gesture, a menu image on the touch screen display; and

provide instant haptic feedback based upon a validity of the input; and

receive from the touch screen display an indication of an interaction with the image representing the dial, and, responsive to the interaction, change a state of the control; and

change the image representing the dial to indicate a change in a rotational position of the dial in response to receiving the indication; and

the aircraft system controller comprising instructions programmed to operate the system in a manner indicated by the rotational position of the dial as represented in the image.

23. The apparatus of claim 22, wherein the controller further comprises instructions programmed to:

increase a size of the image representing the dial to a larger size in response to receiving the interaction with the image representing the dial such that the interaction comprised the image representing the dial receiving an input for a selected duration; and

change a state of the control corresponding to the image representing the dial to an active state when the image representing the dial is changed to the larger size, wherein the rotational position of the image representing the dial changes via-inputs received at-the image representing the dial when the control corresponding to the image representing the dial is in the active state.

24. The apparatus of claim 22, wherein the controller further comprises instructions programmed to change the rotational position of the image representing the dial by one discrete rotational position in a direction in response to receiving an input at the image representing the dial on a side of the image representing the dial corresponding to the direction.

25. The apparatus of claim 22, wherein the system is a radio on the aircraft.

26. A method of configuring a haptically-enabled, moveable, and networked flexible display/control interface for a task, the method comprising:

executing, in a data processing device in an interface controller, instructions programmed for;

displaying on a touch screen display in a cockpit of an aircraft, an image representing a dial configured to be moved between a discrete number of rotational positions;

automatically selecting, via determining a position and an orientation of the touch screen display in the aircraft, an aircraft system displayed on and in communication with the touch screen display;

virtually partitioning the touch screen display;

continuously enhancing a hit accuracy characteristic of the image, via instructions programmed for;

adjusting a touch recognition threshold of the image based upon a controller, comprising the interface controller, receiving information about a speed and an altitude, of the aircraft;

recognizing simultaneous inputs from distinct areas on the touch screen display; and

simultaneously recognizing a first input, on a first partition of the touch screen display as being from a first source on a first side of the touch screen display, and a second input, on a second partition of the touch screen display as being from a second source on a second side of the touch screen, via recognizing an orientation, relative to the touch screen display, of the first source and of the second source, respectively, and simultaneously processing the first input and the second input;

determining, via the second input being along an edge of the second partition and coming from the first source on the first side of the touch screen display, the second input as an accidental input;

responsive to recognizing the accidental input, at least one of;

correcting, and ignoring, the accidental input;

removing, responsive to receipt of a mode-less gesture, a menu image on the touch screen display;

providing instant haptic feedback based upon a validity of the first input;

receiving from the touch screen display an indication of an interaction with the image representing the dial, and, responsive to the interaction, changing a state of the control; and

changing the image representing the dial and indicating a change in a rotational position of the dial in response to receiving the indication; and

operating an aircraft system, via an aircraft system controller comprising programmed instructions, in a manner indicated by the rotational position of the dial as represented in the image.