System and method for controlling swarm of remote unmanned vehicles through human gestures

• US 8,214,098 B2
• Filed: 02/27/2009
• Issued: 07/03/2012
• Est. Priority Date: 02/28/2008
• Status: Active Grant

• Current Assignee: The Boeing Co.
• Original Assignee: The Boeing Co.
• Inventors: Murray, Paul, Troy, James J., Erignac, Charles A., Wojcik, Richard H., Finton, David J., Margineantu, Dragos D.
• Primary Examiner(s): Camby, Richard M.
• Application Number:
  US12/395,019
• Publication Number:
  US 20090222149A1
• Time in Patent Office: 1,222 Days
• Field of Search: 701/23, 701/27, 701/28, 701/70, 701/220, 701/223, 701 337- 342, 701/514, 382/100, 382/103, 382/107, 382/117, 382/118, 382/153, 382/170, 348/95, 348/222.1
• US Class Current: 701/23
• CPC Class Codes: A63H 30/04 using wireless transmission
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64U 10/10 Rotorcrafts
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: G05D 1/0016 characterised by the operator's input device input arrangements for computing systems in general G06F3/00
• CPC Class Codes: G05D 1/0027 involving a plurality of vehicles, e.g. fleet or convoy travelling fleet control of land vehicles from a control room G05D1/0297; traffic control systems for road vehicles G08G1/00; for marine craft G08G3/00; for aircraft G08G5/00
• CPC Class Codes: G05D 1/0033 by having the operator tracking the vehicle either by direct line of sight or via one or more cameras located remotely from the vehicle
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

A method is disclosed for controlling at least one remotely operated unmanned object. The method may involve defining a plurality of body movements of an operator that correspond to a plurality of operating commands for the unmanned object. Body movements of the operator may be sensed to generate the operating commands. Wireless signals may be transmitted to the unmanned object that correspond to the operating commands that control operation of the unmanned object.

Claims

1. A method for controlling at least one remotely operated unmanned object, comprising:

defining a plurality of body movements of an operator that correspond to a plurality of operating commands for said unmanned object;

optically illuminating the unmanned object and an article worn by the operator;

optically illuminating a marker on said unmanned object;

sensing said body movements of said operator by first optical reflections from said article, and sensing movements of said unmanned object by second optical reflections from said unmanned object;

based on said reflected first and second optical signals, generating said operating commands; and

transmitting signals that correspond to said operating commands to said unmanned object to control operation of said unmanned object.

2. The method of claim 1, wherein defining a plurality of body movements of an operator that correspond to a plurality of operating commands comprises defining a hand position of said operator as a stop command.

3. The method of claim 1, wherein defining a plurality of body movements of an operator that correspond to a plurality of operating commands comprises defining a hand motion of said operator as a command for said unmanned object to move in a predetermined direction.

4. The method of claim 1, wherein said defining a plurality of body movements of an operator that correspond to a plurality of operating commands for an unmanned object comprises defining a plurality of body movements that correspond to a plurality of operating commands for an unmanned vehicle.

5. The method of claim 1, further comprising using a plurality of unmanned objects;

and

wherein said defining a plurality of body movements of an operator that correspond to a plurality of operating commands comprises defining a motion of the head of the operator as a command to select a specific one of said unmanned objects to be controlled.

6. The method of claim 5, wherein defining a head motion of the operator comprises defining a side to side rotation of said head as a command to reject a selection of a specific one of said unmanned objects, and wherein an nodding motion of said head defines a command to confirm a selection of a specific one of said unmanned objects as being a recipient of further commands of from said operator.

7. The method of claim 1, wherein defining a plurality of body movements of an operator that correspond to a plurality of operating commands comprises defining a circular hand motion in a plane, said circular hand motion defining a command to cause said unmanned object to begin moving from a stationary condition.

8. The method of claim 1, wherein defining a plurality of body movements of an operator that correspond to a plurality of operating commands comprises using both hands of said operator to form a T shape, to designate a command that causes said unmanned object to ignore subsequent commands.

9. The method of claim 1, wherein defining a plurality of body movements of an operator that correspond to a plurality of operating commands comprises defining a position of a hand of said operator being positioned over the head of said operator as a command to begin a dialog with said unmanned object.

10. The method of claim 1, wherein controlling said unmanned object comprises controlling a plurality of unmanned objects;

and

wherein defining a plurality of body movements of an operator that correspond to a plurality of operating commands comprise defining a pointing motion of one hand of said operator as defining a three dimensional conical area, and wherein further commands by said operator are limited to ones of said unmanned objects present within said conical area.

11. The method of claim 1, wherein said controlling an unmanned object comprises controlling a plurality of unmanned objects;

and

wherein defining a plurality of body movements of an operator that correspond to a plurality of operating commands comprises defining the placement of the palms of the hands of the operator together against one another as a command to group a plurality of ones of said unmanned objects together as a group, to enable said group of unmanned objects to be controlled by subsequently received commands from said operator.

12. The method of claim 1, wherein said controlling an unmanned object comprises controlling a plurality of unmanned objects;

and

wherein defining a plurality of body movements of an operator that correspond to a plurality of operating commands comprises defining a circular motion of a hand of said operator about an axis of a coordinate system associated with the hand as a command that defines a three dimensional conical area and selects only specific ones of said unmanned objects that are present within said conical area, and that selects said ones of said unmanned objects in a predetermined spatial order.

13. The method of claim 12, wherein said predetermined spatial order comprises one of:

top to bottom, front to back, and left to right; and

back to front, bottom to top and left to right.

14. The method of claim 1, wherein said controlling said unmanned object comprises controlling an unmanned airborne vehicle (UAV).

15. The method of claim 14, wherein controlling an unmanned airborne vehicle comprises controlling an unmanned rotorcraft.

16. A method for controlling operation of an unmanned vehicle, comprising:

providing an unmanned vehicle having a plurality of optically reflective markers thereon;

defining at least one gesture of a body part of an operator that corresponds to an operating command for controlling motion of the unmanned vehicle;

using an article worn by an operator, the article having optically reflective markers, to enable the operator responsible for remotely controlling operation of the unmanned vehicle to signal the gesture through movement of the article;

optically illuminating the unmanned vehicle and the article worn by the operator;

sensing optical reflections from the markers of the unmanned vehicle and the markers on the article;

using a motion capture system to interpret the sensed optical reflections and to generate data representing a position of the unmanned vehicle and a motion of the article; and

processing the data to interpret the gesture being given by the operator via motion of the article and to determine a position of the unmanned vehicle, and to generate a command for controlling the unmanned vehicle in accordance with the gesture; and

wirelessly transmitting the command to the unmanned vehicle.

17. A system for remotely controlling at least one unmanned vehicle through body movements of an operator, the system comprising:

an article worn on a body part of said operator, the article having a first plurality of optically reflective markers secured thereto;

a second plurality of optically reflective markers secured to said unmanned vehicle;

a plurality of lights for illuminating the first and second pluralities of optically reflective markers;

a motion capture system for optically tracking a position and an orientation of said article via first reflected optical signals from said first plurality of optically reflective markers, and a position and orientation of said unmanned vehicle via second reflected optical signals from said second plurality of optically reflective markers, and generating data representing positions of each of said first and second pluralities of optically reflective markers;

a processing system adapted to process the data from said motion capture system and to generate commands for controlling motion of the unmanned vehicle, the processing system including;

a gesture recognition module for interpreting portions of said data received from said motion capture system that define positions and orientations of said first plurality of markers on said articles being worn by said operator, as predefined operating commands to be provided to said unmanned vehicle; and

a feedback control subsystem that receives the navigation signals from the gesture recognition module and generates low-level vehicle control commands for controlling the unmanned vehicle.

18. The system of claim 17, further comprising a transmitter for wirelessly transmitting the low level vehicle control commands to the unmanned vehicle.

19. The system of claim 17, wherein the unmanned vehicle includes a control system for receiving the wirelessly transmitted low level vehicle control commands.

20. The system of claim 17, wherein said article comprises at least one of an article worn on a hand of said operator and an article worn on a head of said operator.