METHOD AND DEVICE FOR CONTROLLING UNMANNED AERIAL VEHICLE WITH GESTURE

• US 20180196419A1
• Filed: 01/11/2016
• Published: 07/12/2018
• Est. Priority Date: 09/24/2015
• Status: Active Grant

• Current Assignee: Beijing Zero Zero Infinity Technology Co., Ltd. (Shenzhen Zero Zero Infinite Technology Co. Ltd.)
• Original Assignee: Beijing Zero Zero Infinity Technology Co., Ltd. (Shenzhen Zero Zero Infinite Technology Co. Ltd.)
• Inventors: WANG, Mengqiu, ZHANG, Tong, LI, Qicheng, LU, Jia, LIU, Lixin
• Granted Patent: US 10,261,507 B2
• Time in Patent Office: Days
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64D 47/08 Arrangements of cameras
• CPC Class Codes: B64U 20/87 Mounting of imaging devices, e.g. mounting of gimbals
• CPC Class Codes: B64U 2201/00 UAVs characterised by their flight controls
• CPC Class Codes: G05D 1/0016 characterised by the operator's input device input arrangements for computing systems in general G06F3/00
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G05D 1/08 Control of attitude, i.e. control of roll, pitch, or yaw
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A method and device for controlling an unmanned aerial vehicle with a gesture are provided. A camera is arranged in the unmanned aerial vehicle, and the method includes: detecting a gesture in an image by using a gesture detection framework; judging whether the gesture is a predetermined gesture for controlling the unmanned aerial vehicle; acquiring a motion trajectory of the gesture in a case that it is determined that the gesture is the predetermined gesture for controlling the unmanned aerial vehicle; and controlling, based on the motion trajectory of the gesture, the unmanned aerial vehicle to perform a control operation corresponding to the motion trajectory of the gesture, where a correspondence between a motion trajectory of a gesture and a control operation is predetermined.

Claims

1. A method for controlling an unmanned aerial vehicle with a gesture, wherein the unmanned aerial vehicle is provided with a camera and the method comprises:

detecting a gesture in an image by using a gesture detection framework;

judging whether the gesture is a predetermined gesture for controlling the unmanned aerial vehicle;

acquiring a motion trajectory of the gesture in a case that it is judged that the gesture is the predetermined gesture for controlling the unmanned aerial vehicle; and

controlling, based on the motion trajectory of the gesture, the unmanned aerial vehicle to perform a control operation corresponding to the motion trajectory of the gesture, wherein the motion trajectory of the gesture and the control operation corresponding thereto are predetermined.

2. The method for controlling an unmanned aerial vehicle with a gesture according to claim 1, wherein the motion trajectory of the gesture is a motion trajectory formed by a motion of a single gesture or a motion trajectory formed by alternate operations of at least two gestures.

3. The method for controlling an unmanned aerial vehicle with a gesture according to claim 2, wherein in a case that the motion trajectory of the gesture is the motion trajectory formed by the motion of the single gesture, the acquiring a motion trajectory of the gesture comprises:

obtaining a motion parameter of a gesture in a current frame relative to a gesture in a last frame based on the gesture in the current frame and the gesture in the last frame, wherein the motion parameter comprises a rotation matrix R and a translation vector T;

accumulating motion parameters between every two adjacent frames from the first frame to the current frame; and

obtaining the motion trajectory of the gesture based on the accumulated motion parameters.

4. The method for controlling an unmanned aerial vehicle with a gesture according to claim 2, wherein in a case that the motion trajectory of the gesture is a motion trajectory formed by alternate operations of two gestures H₁ and H₂, the acquiring a motion trajectory of the gesture comprises:

acquiring M H₁s from the image, and judging whether a frame span K of the M H₁s is smaller than a preset number N of frames, wherein M and N each are integers;

dividing K into (M−

1) intervals equally and judging whether each of the (M−

1) intervals comprises H₂, in a case that it is judged that the frame span K of the M H₁s is smaller than the preset number N of frames; and

judging that the alternate operations of the two gestures H₁ and H₂ occurs and determining the motion trajectory of the gesture based on the alternate operations of the two gestures H₁ and H₂, in a case that it is judged that each of the (M−

1) intervals comprises H₂.

5. The method for controlling an unmanned aerial vehicle with a gesture according to claim 3, wherein the obtaining a motion parameter of a gesture in a current frame relative to a gesture in a last frame based on the gesture in the current frame and the gesture in the last frame comprises:

extracting a FAST, features from accelerated segment test, feature point from regions where the gestures are located with respect to images in the current frame and in the last frame

calculating a BRIEF, binary robust independent elementary features, feature vector based on the FAST feature points;

performing a brute force match on the BRIEF feature vector, and filtering out a wrong matching point in a brute force matching result using RANSAC, random sample consensus, to obtain a filtered matching point pair; and

calculating an eigenmatrix E using the matching point and obtaining R and T by decomposing E.

6. A device for controlling an unmanned aerial vehicle with a gesture, wherein the unmanned aerial vehicle is provided with a camera and the device comprises:

a gesture detecting unit configured to detect a gesture in an image by using a gesture detection framework;

a gesture judging unit configured to judge whether the gesture is a predetermined gesture for controlling the unmanned aerial vehicle;

a gesture motion trajectory acquiring unit configured to acquire a motion trajectory of the gesture in a case that it is judged that the gesture is the predetermined gesture for controlling the unmanned aerial vehicle; and

a control unit configured to control the unmanned aerial vehicle to perform a control operation corresponding to the motion trajectory of the gesture based on the motion trajectory of the gesture, wherein the motion trajectory of the gesture and the control operation corresponding thereto are predetermined.

7. The device for controlling an unmanned aerial vehicle with a gesture according to claim 6, wherein the motion trajectory of the gesture is a motion trajectory formed by a motion of a single gesture or a motion trajectory formed by alternate operations of at least two gestures.

8. The device for controlling an unmanned aerial vehicle with a gesture according to claim 7, wherein in a case that the motion trajectory of the gesture is a motion trajectory formed by the motion of the single gesture, the gesture motion trajectory acquiring unit comprises:

a motion parameter acquiring subunit configured to obtain a motion parameter of a gesture in a current frame relative to a gesture in a last frame based on the gesture in the current frame and the gesture in the last frame, wherein the motion parameter comprises a rotation matrix R and a translation vector T;

an accumulating subunit configured to accumulate motion parameters between every two adjacent frames from the first frame to the last frame; and

a trajectory acquiring subunit configured to obtain the motion trajectory of the gesture based on the accumulated motion parameters.

9. The device for controlling an unmanned aerial vehicle with a gesture according to claim 7, wherein in a case that the motion trajectory of the gesture is a motion trajectory formed by alternate operations of two gestures H₁ and H₂, the gesture motion trajectory acquiring unit comprises:

a frame acquiring subunit configured to acquire M H₁s from the image;

a first judging subunit configured to judge whether a frame span K of the M H₁s is smaller than a preset number N of frames, wherein M and N each are integers;

a second judging subunit configured to divide K into (M−

1) intervals equally and judge whether each of the (M−

1) intervals comprises H₂, in a case that the first judging subunit determines that the frame span K of the M H₁s is smaller than the preset number N of frames; and

a determining subunit configured to determine that alternate operations of two gestures H₁ and H₂ occurs, and determine the motion trajectory of the gesture based on the alternate operations of H₁ and H₂, in a case that the second judging subunit judges that each of the (M−

1) intervals includes H₂.

10. The device for controlling an unmanned aerial vehicle with a gesture according to claim 8, wherein the motion parameter acquiring subunit comprises:

a FAST feature point extracting subunit configured to extract a FAST feature point from regions where the gestures are located with respect to images in the current frame and in the last frame;

a BRIEF feature vector obtaining subunit configured to calculate a BRIEF feature vector based on the FAST feature points;

a matching subunit configured to perform a brute force match on the BRIEF feature vector and filter out a wrong matching point in a brute force matching result using RANSAC to obtain a filtered matching point pair; and

a decomposing subunit configured to calculate an eigenmatrix E using the matching point, and obtaining R and T by decomposing E.