Human machine interface system

• US 9,938,010 B2
• Filed: 12/02/2015
• Issued: 04/10/2018
• Est. Priority Date: 12/02/2014
• Status: Active Grant

• Current Assignee: Sikorsky Aircraft Corporation (Martin Marietta Corporation)
• Original Assignee: Sikorsky Aircraft Corporation (Martin Marietta Corporation)
• Inventors: Lampazzi, Margaret MacIsaac, Guido, Thomas, Howland, Gary, Bertuccelli, Luca F., Pupalaikis, Robert
• Primary Examiner(s): Kleinman, Lail A
• Application Number:
  US14/957,067
• Publication Number:
  US 20160152336A1
• Time in Patent Office: 860 Days
• Field of Search: None
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: G01C 23/00 Combined instruments indicating more than one navigational value, e.g. for aircraft; Combined measuring devices for measuring two or more variables of movement, e.g. distance, speed or acceleration
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G08G 5/0008 with other aircraft
• CPC Class Codes: G08G 5/0021 located in the aircraft
• CPC Class Codes: G08G 5/0069 specially adapted for an unmanned aircraft

Abstract

An electronic human machine interface (HMI) module is configured to receive at least one input from an operator of a manually-operated vehicle. The electronic HMI module includes an electronic graphical display unit and an electronic control module. The electronic graphical display unit is configured to display information corresponding to the manually-operated vehicle and information corresponding to an autonomously operated unmanned vehicle (UV) located remotely from the manually-operated vehicle. The electronic control module is in signal communication with the UV and is configured to receive an electronic image signal from the UV. The electronic control module is further configured to display a real-time image captured by the UV on the electronic graphical display unit based on the image signal.

Claims

1. An electronic human machine interface (HMI) module configured to receive at least one input from an operator of a manually-operated vehicle, the electronic HMI module comprising:

an electronic graphical display unit configured to display information corresponding to the manually-operated vehicle and information corresponding to an autonomously operated unmanned vehicle (UV) located remotely from the manually-operated vehicle, wherein the graphical display unit further displays at least one of a handover location associated with establishing a handover communication session and an estimated time of arriving at the handover location; and

an electronic control module in signal communication with the UV, the electronic control module configured to receive an electronic image signal from the UV and to display a real-time image captured by the UV on the electronic graphical display unit based on the image signal during the handover communication session;

wherein the electronic control module is operable to establish the handover communication session between the manually-operated vehicle and the UV when the manually-operated vehicle and the UV are within a distance threshold, wherein the manually-operated vehicle continuously obtains an up-to-date analysis of a point of interest from the UV while travelling to the point of interest, the continuous up-to-date analysis being initiated in response to establishing the handover communication.

2. The electronic HMI module of claim 1, wherein the electronic control module selectively operates the electronic graphical display unit in a first mode and a second mode during the handover communication session.

3. The electronic HMI module of claim 2, wherein the graphical display unit displays an interface of selectable actions to control operation of the manually-operated vehicle and the UV when operating in the first mode, and wherein the graphical display unit displays real-time image captured by the UV when operating in the second mode.

4. The electronic HMI module of claim 3, wherein the interface of selectable actions include a first set of selectable actions to control at least one of the handover communication session and navigation of the UV, and a second set of selectable action to control operation of at least one UV image sensor installed on the UV.

5. The electronic HMI module of any one of claims 1-4, wherein the electronic control module outputs an autonomous operation control signal that commands the UV to traverse to a geographical point of interest while simultaneously transmitting the image signal to the electronic control module.

6. The electronic HMI module of claim 5, wherein the UV automatically initiates the at least one image sensor to capture the real-time image in response to establishing the handover communication session.

7. The electronic HMI module of claim 6, wherein the electronic control module receives an input commanding the at least one UV sensor to selectively output the image signal as one of an infrared image signal or a video signal.

8. A method of visually enhancing a geographical point of interest using a human machine interface (HMI) system, the method comprising:

displaying at least one of a handover location associated with establishing a handover communication session and an estimated time of arriving at the handover location on a graphical display of the HMI module installed in the manually-operated vehicle;

establishing the handover communication session to electrically communicate at least one electrical signal between a manually-operated vehicle and an autonomously operated unmanned vehicle (UV) located remotely from the manually-operated vehicle when the manually-operated vehicle and the UV are within a distance threshold;

automatically navigating the UV to a point of interest designated using an electronic HMI module installed on the manually-operated vehicle;

capturing a real-time image of the point of interest using at least one image sensor installed on the UV, and electrically transmitting the real-time image to the manually-operated vehicle wherein the manually-operated vehicle continuously obtains an up-to-date analysis of the point of interest from the UV while travelling to the point of interest, the continuous up-to-date analysis being initiated in response to establishing the handover communication; and

displaying the real-time image captured by the UV on the graphical display.

9. The method of claim 8, further comprising automatically establishing the handover communication session in response to satisfying the distance threshold between the manually-operated vehicle and the UV.

10. The method of claim 9, wherein the UV automatically begins capturing the real-time image and transmitting the real-time image to the manually-operated vehicle in response to establishing the handover communication session.

11. The method of claim 8, wherein the manually-operated vehicle is a manually-operated aerial vehicle and the UV is an unmanned aerial vehicle.

12. The method of claim 11, wherein the point of interest is a landing area or evacuation area of the unmanned aerial vehicle.

13. The method of claim 12, wherein the unmanned aerial vehicle automatically navigates to the point of interest in response to establishing the handover communication session.

14. The method of claim 13, wherein the unmanned aerial vehicle continuously traverses about a flight path above the point of interest area while transmitting the real-time image to the manually-operated aerial vehicle in response to reaching the point of interest.

15. The method of claim 14, further comprising displaying the real-time image of the point of interest on the graphical display unit before the manually-operated vehicle arrives at the point of interest.