Multi-sensor control system and method for remote signaling control of unmanned vehicles

• US 10,459,524 B2
• Filed: 04/14/2015
• Issued: 10/29/2019
• Est. Priority Date: 04/14/2015
• Status: Active Grant

• Current Assignee: Northrop Grumman Systems Corporation (Northrop Grumman Corporation)
• Original Assignee: Northrop Grumman Systems Corporation (Northrop Grumman Corporation)
• Inventors: Clark, Ryan T.
• Primary Examiner(s): Yodichkas, Aneeta
• Application Number:
  US14/686,370
• Publication Number:
  US 20170031446A1
• Time in Patent Office: 1,659 Days
• CPC Class Codes: A63H 27/12 Helicopters A63H27/04 takes precedence; Flying tops
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64U 10/17 Helicopters flying platforms B64U10/13
• CPC Class Codes: B64U 2201/00 UAVs characterised by their flight controls
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: B64U 50/13 using external fans or propellers
• CPC Class Codes: G05D 1/0016 characterised by the operator's input device input arrangements for computing systems in general G06F3/00
• CPC Class Codes: G05D 1/0022 characterised by the communication link data switching networks in general H04L12/00
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: H04W 4/80 Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication

Abstract

An apparatus includes a wearable device having a multi-sensor detector to sense operator gestures directed at an unmanned vehicle (UV). The multi-sensor detector includes at least two sensors to detect motion and direction of the operator gestures with respect to operator hand movement, operator hand movement with respect to the earth, rotational movement of the operator hand, and finger movement on the operator hand. A controller monitors the multi-sensor detector to determine the operator gesture based on input data received from the sensors. The controller generates a command to the UV based on the determined operator gesture.

Claims

1. An apparatus, comprising:

a set of wearable devices each having a multi-sensor detector to sense an operator gesture directed at an unmanned vehicle (UV), wherein each multi-sensor detector comprises;

a plurality of sensors including one of a sensor mounted on a respective wearable device proximal to an operator'"'"'s finger to detect finger movement; and

another sensor to detect one of motion and direction of the operator gesture with respect to one or more operator hand movement, operator hand movement with respect to a planet'"'"'s magnetic field, and rotational movement of an operator hand; and

a set of controllers to monitor respective multi-sensor detectors to determine a corresponding operator gesture based on input data provided by the plurality of sensors of a respective wearable device, wherein each controller generates a respective command for the UV based on the corresponding operator gesture;

a communications module configured to provide the respective command to the UV to control a flight path of the UV; and

a wearable display device to be worn on or about a head of an operator, the wearable display device being configured to receive flight path information for the UV from the communication module for display thereon.

2. The apparatus of claim 1, wherein the other sensor is one of an accelerometer to detect the motion and the direction of the operator gesture with respect to the operator hand movement, a magnetometer to detect the operator hand movement with respect to the planet'"'"'s magnetic field, and a gyroscope to detect the rotational movement of the operator hand.

3. The apparatus of claim 1, wherein the controller sends the respective command to the communications module, the communications module to relay the respective command to the UV to control the UV.

4. The apparatus of claim 3, wherein the controller sends the respective command via a Bluetooth or a universal serial bus connection (USB) to the communications module.

5. The apparatus of claim 4, further comprising a mission planner application, wherein the mission planner application is to communicate with the communications module to receive the respective command and generate a corresponding control command to control the UV based on the respective command.

6. The apparatus of claim 5, wherein the mission planner application executes on a computer and communicates via a wireless link with the communications module to receive the respective command.

7. The apparatus of claim 6, wherein:

the wearable display device is wearable glasses;

the wearable glasses are to receive mission status information provided by the mission planner application; and

the wearable glasses are to display the mission status information to the operator.

8. The apparatus of claim 6, wherein the mission planner application communicates a software packet via a scripting interface to a generic protocol interface that translates commands in the generic protocol to an application that communicates with the communications module.

9. The apparatus of claim 8, wherein the generic protocol is a transport control protocol (TCP) and the software packet is communicated via a Python script.

10. The apparatus of claim 1, wherein the other sensor is a magnetometer to detect the operator hand movement with respect to the planet'"'"'s magnetic field.

11. A system, comprising:

a plurality of wearable devices each having a multi-sensor detector to sense an operator gesture directed at an unmanned vehicle (UV), wherein each multi-sensor detector comprises a plurality of sensors including;

a sensor mounted on a respective wearable device of the plurality of wearable devices proximal to an operator'"'"'s finger to detect finger movement; and

another sensor that includes one of an accelerometer to detect motion and direction of the operator gesture with respect to operator hand movement, a magnetometer to detect operator hand movement with respect to a planet'"'"'s magnetic field, and a gyroscope to detect rotational movement of the operator'"'"'s hand; and

a plurality of controllers to monitor respective multi-sensor detectors to determine a corresponding operator gesture based on input data provided by the plurality of sensors of a respective wearable device, wherein each controller generates a respective command for the UV based on the corresponding operator gesture;

a communications module to communicate the respective command to the UV to control a flight path of the UV; and

a wearable display device to be worn on or about a head of a user, the wearable display device being configured to receive flight path information for the UV from the communications module for display thereon.

12. The system of claim 11, wherein the communications module is a mobile device.

13. The system of claim 12, wherein the controller sends the respective command via a Bluetooth or a universal serial bus connection (USB) to the communications module.

14. The system of claim 13, further comprising a mission planner application wherein the mission planner application is to communicate with the communications module to receive the respective command and generate a control command based on the respective command to control the UV.

15. The system of claim 14, wherein the mission planner application executes on a computer and communicates via a wireless link with the communications module to receive the respective command.

16. The system of claim 15, wherein:

the wearable display device is wearable glasses;

the wearable glasses are to receive mission status information provided by the mission planner application; and

the wearable glasses are to display the mission status information to the operator.

17. The system of claim 16, wherein the mission planner application communicates a software packet via a scripting interface to a generic protocol interface that translates commands in the generic protocol to an application that communicates with the communications module.

18. The system of claim 17, wherein the generic protocol is a transport control protocol (TCP) and the software packet is communicated via a Python script.

19. A method, comprising:

sensing, via respective controllers a corresponding operator gesture directed at an unmanned vehicle (UV) from an wearable device having multi-sensor detectors, wherein each multi-sensor detector comprises at least two sensors, the at least two sensors including a first sensor mounted on a respective wearable device proximal to an operator'"'"'s finger to detect finger movement and a second sensor to detect one of motion and direction of the operator gesture with respect to one of operator hand movement, operator hand movement with respect to a planet'"'"'s magnetic field, and rotational movement of the operator'"'"'s hand;

analyzing, via the respective controllers, sensor data provided by the at least two sensors of a respective wearable device to detect motion and direction of the operator gestures with respect to operator hand movement, operator hand movement with respect to the earth, rotational movement of the operator hand, and the finger movement on the operator'"'"'s hand;

determining, via the respective controllers, the corresponding operator gesture based on the analyzing of the sensor data provided by the at least two sensors of the respective wearable device;

communicating, via a communications module controller, a corresponding command to the UV to control an operation of the UV based on the corresponding determined operator gesture; and

communicating, via the communications module, flight path information for the UV to a wearable display device to be worn on or about a head of an operator for display thereon.

20. The method of claim 19, wherein the communicating comprises communicating the corresponding command to a mission planner application, wherein the mission planner application generates a control command to control the operation of the UV based on the corresponding command.