Method and device for controlling unmanned aerial vehicle with gesture

• US 10,261,507 B2
• Filed: 01/11/2016
• Issued: 04/16/2019
• Est. Priority Date: 09/24/2015
• Status: Active Grant

• Current Assignee: Beijing Zero Zero Infinity Technology Co., Ltd. (Shenzhen Zero Zero Infinite Technology Co. Ltd.)
• Original Assignee: Beijing Zero Zero Infinity Technology Co., Ltd. (Shenzhen Zero Zero Infinite Technology Co. Ltd.)
• Inventors: Wang, Mengqiu, Zhang, Tong, Li, Qicheng, Lu, Jia, Liu, Lixin
• Primary Examiner(s): Black, Thomas G
• Assistant Examiner(s): Huynh, Luat T
• Application Number:
  US15/108,390
• Publication Number:
  US 20180196419A1
• Time in Patent Office: 1,191 Days
• Field of Search: 701 2, 701 3, 701 23
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64D 47/08 Arrangements of cameras
• CPC Class Codes: B64U 20/87 Mounting of imaging devices, e.g. mounting of gimbals
• CPC Class Codes: B64U 2201/00 UAVs characterised by their flight controls
• CPC Class Codes: G05D 1/0016 characterised by the operator's input device input arrangements for computing systems in general G06F3/00
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G05D 1/08 Control of attitude, i.e. control of roll, pitch, or yaw
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A method and device for controlling an unmanned aerial vehicle with a gesture are provided. A camera is arranged in the unmanned aerial vehicle, and the method includes: detecting a gesture in an image by using a gesture detection framework; judging whether the gesture is a predetermined gesture for controlling the unmanned aerial vehicle; acquiring a motion trajectory of the gesture in a case that it is determined that the gesture is the predetermined gesture for controlling the unmanned aerial vehicle; and controlling, based on the motion trajectory of the gesture, the unmanned aerial vehicle to perform a control operation corresponding to the motion trajectory of the gesture, where a correspondence between a motion trajectory of a gesture and a control operation is predetermined.

Claims

1. A method for controlling an unmanned aerial vehicle with a gesture, wherein the unmanned aerial vehicle is provided with a camera and the method comprises:

detecting a gesture in an image by using a gesture detection framework;

judging whether the gesture is a predetermined gesture for controlling the unmanned aerial vehicle;

acquiring a motion trajectory of the gesture in a case that it is judged that the gesture is the predetermined gesture for controlling the unmanned aerial vehicle; and

controlling, based on the motion trajectory of the gesture, the unmanned aerial vehicle to perform a control operation corresponding to the motion trajectory of the gesture, wherein the motion trajectory of the gesture and the control operation corresponding thereto are predetermined;

wherein the motion trajectory of the gesture is a motion trajectory formed by alternate operations of two gestures H₁ and H₂; and

wherein the acquiring a motion trajectory of the gesture comprises;

acquiring M H₁s from the image, and judging whether a frame span K of the M H₁s is smaller than a preset number N of frames, wherein M and N each are integers;

dividing K into (M−

1) intervals equally and judging whether each of the (M−

1) intervals comprises H₂, in a case that it is judged that the frame span K of the M H₁s is smaller than the preset number N of frames; and

judging that the alternate operations of the two gestures H₁ and H₂ occurs and determining the motion trajectory of the gesture based on the alternate operations of the two gestures H₁ and H₂, in a case that it is judged that each of the (M−

1) intervals comprises H₂.

2. A device for controlling an unmanned aerial vehicle with a gesture, wherein the unmanned aerial vehicle is provided with a camera and the device comprises:

a gesture detecting unit configured to detect a gesture in an image by using a gesture detection framework;

a gesture judging unit configured to judge whether the gesture is a predetermined gesture for controlling the unmanned aerial vehicle;

a gesture motion trajectory acquiring unit configured to acquire a motion trajectory of the gesture in a case that it is judged that the gesture is the predetermined gesture for controlling the unmanned aerial vehicle; and

a control unit configured to control the unmanned aerial vehicle to perform a control operation corresponding to the motion trajectory of the gesture based on the motion trajectory of the gesture, wherein the motion trajectory of the gesture and the control operation corresponding thereto are predetermined;

wherein the motion trajectory of the gesture is a motion trajectory formed by alternate operations of two gestures H₁ and H₂; and

wherein the gesture motion trajectory acquiring unit comprises;

a frame acquiring subunit configured to acquire M H₁s from the image;

a first judging subunit configured to judge whether a frame span K of the M H₁s is smaller than a preset number N of frames, wherein M and N each are integers;

a second judging subunit configured to divide K into (M−

1) intervals equally and judge whether each of the (M−

1) intervals comprises H₂, in a case that the first judging subunit determines that the frame span K of the M H₁s is smaller than the preset number N of frames; and

a determining subunit configured to determine that alternate operations of two gestures H₁ and H₂ occurs, and determine the motion trajectory of the gesture based on the alternate operations of H₁ and H₂, in a case that the second judging subunit judges that each of the (M−

1) intervals includes H₂.

3. A method for controlling an unmanned aerial vehicle with a gesture, wherein the unmanned aerial vehicle is provided with a camera and the method comprises:

detecting a gesture in an image by using a gesture detection framework;

judging whether the gesture is a predetermined gesture for controlling the unmanned aerial vehicle;

acquiring a motion trajectory of the gesture in a case that it is judged that the gesture is the predetermined gesture for controlling the unmanned aerial vehicle; and

controlling, based on the motion trajectory of the gesture, the unmanned aerial vehicle to perform a control operation corresponding to the motion trajectory of the gesture, wherein the motion trajectory of the gesture and the control operation corresponding thereto are predetermined;

wherein the motion trajectory of the gesture is the motion trajectory formed by the motion of the single gesture;

wherein the acquiring a motion trajectory of the gesture comprises;

obtaining a motion parameter of a gesture in a current frame relative to a gesture in a last frame based on the gesture in the current frame and the gesture in the last frame, wherein the motion parameter comprises a rotation matrix R and a translation vector T;

accumulating motion parameters between every two adjacent frames from the first frame to the current frame; and

obtaining the motion trajectory of the gesture based on the accumulated motion parameters; and

wherein the obtaining a motion parameter of a gesture in a current frame relative to a gesture in a last frame based on the gesture in the current frame and the gesture in the last frame comprises;

extracting, by a feature extractor, feature point from regions where the gestures are located with respect to images in the current frame and in the last frame

calculating, by a feature descriptor, a feature vector based on the feature points;

performing a brute force match on the feature vector, and filtering out a wrong matching point in a brute force matching result using a outlier detection method, to obtain a filtered matching point pair; and

calculating an eigenmatrix E using the matching point and obtaining R and T by decomposing E.

4. The method for controlling an unmanned aerial vehicle with a gesture according to claim 3, wherein:

the feature extractor is FAST, features from accelerated segment test;

the feature descriptor is BRIEF, binary robust independent elementary features; and

the outlier detection method is RANSAC, random sample consensus.