System and method for providing an ergonomic three-dimensional, gesture based, multimodal interface for use in flight deck applications

• US 9,785,243 B2
• Filed: 01/30/2014
• Issued: 10/10/2017
• Est. Priority Date: 01/30/2014
• Status: Expired due to Fees

• Current Assignee: Honeywell International Inc.
• Original Assignee: Honeywell International Inc.
• Inventors: Kawalkar, Amit Nishikant
• Primary Examiner(s): Kim, Sang H
• Application Number:
  US14/168,426
• Publication Number:
  US 20150212581A1
• Time in Patent Office: 1,349 Days
• CPC Class Codes: B64C 13/042 operated by hand
• CPC Class Codes: B64C 19/00 Aircraft control not otherwise provided for
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

A system and method for operating an aircraft in response to input gestures is provided. The method is comprised of generating a valid interaction volume substantially surrounding a user'"'"'s hand, based on the location of the user'"'"'s arm and hand relative to a pilot support apparatus, recognizing when a gesture performed within the valid interaction volume indicates a valid input, and generating an associated system command.

Claims

1. A method for operating an aircraft, the method comprising:

detecting, with respect to a sensor volume generated by a sensor, (i) a position and location of a chair from a flight deck dashboard, and (ii) a position and location of an arm-rest from the flight deck dashboard;

retrieving a chair model from a memory device, wherein the chair model comprises a chair coupled to an arm-rest, a height of the arm-rest, and a length of the arm-rest extending past an elbow contact point;

retrieving normalized arm anthropology models comprising (i) a normalized angle between a hand at a neutral position and the arm-rest, (ii) a normalized arm and arm-rest intersection point, and (iii) a normalized arm length from the arm and arm-rest intersection point, from the memory device;

detecting, within the sensor volume, gestures performed by a user'"'"'s palm and fingers;

generating, based on the chair model, the normalized arm anthropology models, and the position and location of (i) the chair and (ii) the arm-rest, a palm ergonomic movement volume;

generating, based on the palm ergonomic movement volume, a valid interaction volume substantially surrounding the user'"'"'s palm and fingers, the valid interaction volume being smaller than, and located within, the sensor volume;

recognizing when a gesture performed is within the valid interaction volume and indicates a valid gesture input; and

generating an associated system command based on the valid gesture input.

2. The method of claim 1, wherein generating a valid interaction volume further comprises:

sensing the position of the user'"'"'s palm and fingers within the valid interaction volume; and

analyzing at least one of a length of the arm-rest, a height of the arm-rest, an angle between the arm-rest and an arm, a palm end point, a palm bottom joint, an elbow-arm-rest contact point, and a chair location.

3. The method of claim 1, wherein recognizing when a gesture performed within the valid interaction volume indicates a valid gesture input comprises:

generating a three dimensional gesture interaction model comprising a plurality of rules, wherein the rules comprise one or more valid gesture components;

distinguishing, during a gesture made within the valid interaction volume, static and dynamic features of a palm and fingers;

parsing the static and dynamic features of the palm and fingers; and

comparing the static and dynamic features of the palm and fingers to rules defined in the three dimensional gesture interaction model.

4. The method of claim 3, wherein the step of comparing comprises:

grouping one or more parsed features into component groups based upon the three dimensional gesture interaction model.

5. The method of claim 4, further comprising:

obtaining user feedback descriptors from a source of user feedback data;

comparing component groups to user feedback descriptors; and

generating visual and aural user feedback.

6. The method of claim 5, wherein the step of generating visual feedback comprises:

generating a gesture pointer; and

displaying the gesture pointer on a display unit.

7. The method of claim 5, wherein the step of generating visual feedback comprises:

generating a three dimensional image representative of the user'"'"'s palm and fingers based upon the task; and

displaying the three dimensional image on a display unit.

8. The method of claim 5, wherein the step of generating visual feedback comprises:

generating an image representative of a user interface element with at least one associated affordance; and

displaying the image on a display unit.

9. The method of claim 4, further comprising:

obtaining system command descriptors from a source of system command data; and

comparing component groups to system command descriptors.

10. A volumetric computing system for an aircraft, comprising:

a chair coupled to an arm-rest;

a sensor generating a predetermined sensor volume;

a display device; and

a processor coupled to the chair, the sensor, and the display device, the processor configured to

(a) detect, within the predetermined sensor volume, (i) a position and location of the chair from a flight deck dashboard, and (ii) a position and location of the arm-rest from the flight deck dashboard;

(b) retrieve a chair model from a memory device, wherein the chair model comprises a chair coupled to an arm rest, a height of the arm-rest, and a length of the arm-rest extending past an elbow contact point;

(c) detect, within the predetermined sensor volume, gestures performed by a user'"'"'s palm and fingers;

(d) generate, based on the chair model, the position and location of the chair and the arm-rest, and normalized arm anthropology models comprising (i) a normalized angle between a hand at a neutral position and an arm-rest, and (ii) a normalized arm length from an intersection point of an arm and the arm-rest, a valid interaction volume substantially surrounding the user'"'"'s palm and fingers, the valid interaction volume being smaller than, and located within, the sensor volume;

(e) recognize when a detected gesture performed is within the valid interaction volume and indicates a valid gesture input; and

(f) generate an associated system command based on the valid gesture input.

11. The system of claim 10, wherein the processor is further configured to:

sense the position of the user'"'"'s palm and fingers within the predetermined sensor volume; and

analyze at least one of a length of the arm-rest, a height of the arm-rest, an angle between the arm-rest and an arm, a palm end point, a palm bottom joint, an elbow-arm-rest contact point, and a chair location.

12. The system of claim 10, further comprising:

a source of three dimensional gesture interaction model data, wherein the source of model data is coupled to the processor, and wherein the model data comprises a plurality of rules associated with one or more valid gesture components; and

wherein the processor is further configured to (a) distinguish, during a gesture made within the valid interaction volume, static and dynamic features of a palm and fingers;

(b) parse the static and dynamic features of the palm and fingers; and

(c) compare the parsed features to rules defined in the three dimensional gesture interaction model.

13. The system of claim 12, further comprising:

an audible device coupled to the processor; and

a source of user feedback descriptor data coupled to the processor;

wherein the processor is further configured to generate visual feedback on the display device and to generate aural feedback on the audible device; and

wherein the generation of the visual and aural feedback are based on the user feedback descriptor data.

14. The system of claim 12, further comprising:

a source of system command descriptor data coupled to the processor

wherein the processor is further configured to generate system commands based on the user system command descriptor data.

15. A method for operating an aircraft, the method comprising:

positioning, with respect to at least one sensor, an arm-rest that is coupled to a chair;

constructing a predetermined sensor volume based on the position and location of the chair from a flight deck dashboard, the position and location of the arm-rest from the flight deck dashboard, and the at least one sensor;

generating, based on

(a) normalized arm anthropology models comprising (i) a normalized angle between a hand at a neutral position and an arm-rest, and (ii) a normalized arm length from an intersection point of an arm and the arm-rest,

(b) chair models comprising a height of the arm-rest and a length of the arm-rest extending past an elbow contact point, and

(c) the position and location of the chair and arm rest measured from a flight deck dashboard,

relative to the predetermined sensor volume, a valid interaction volume substantially surrounding a user'"'"'s hand, the valid interaction volume being smaller than, and located within, the predetermined sensor volume;

analyzing static and dynamic movements of a user'"'"'s palm and fingers within the valid interaction volume;

recognizing when the static and dynamic movements of the user'"'"'s palm and fingers within the valid interaction volume indicate a valid gesture input; and

generating an associated system command based on the valid gesture input.

16. The method of claim 15, further comprising:

sorting valid gesture input into at least one of (a) gesture entry point, (b) gesture start, (c) gesture body, and (e) gesture end.

17. The method of claim 16, wherein the step of sorting further comprises:

evaluating gesture recognition success based upon the gesture body.