System and method for providing autonomous photography and videography

• US 10,375,289 B2
• Filed: 06/29/2017
• Issued: 08/06/2019
• Est. Priority Date: 03/31/2017
• Status: Active Grant

• Current Assignee: Hangzhou Zero Zero Technology Co.Ltd (Shenzhen Zero Zero Infinite Technology Co. Ltd.)
• Original Assignee: Hangzhou Zero Zero Technology Co.Ltd (Shenzhen Zero Zero Infinite Technology Co. Ltd.)
• Inventors: Wang, Mengqiu, Liu, Lixin, Zhang, Tong, Lu, Jia
• Primary Examiner(s): Luo, Kate H
• Application Number:
  US15/637,954
• Publication Number:
  US 20180288303A1
• Time in Patent Office: 768 Days
• Field of Search: None
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64D 47/08 Arrangements of cameras
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: G01S 17/02 Systems using the reflection of electromagnetic waves other than radio waves G01S17/66 takes precedence
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 3/7864 T.V. type tracking systems
• CPC Class Codes: G05D 1/0011 associated with a remote control arrangement
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G06V 20/13 Satellite images
• CPC Class Codes: G06V 20/17 taken from planes or by drones
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 2201/07 Target detection
• CPC Class Codes: G06V 40/166 using acquisition arrangements
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/66 Remote control of cameras or camera parts, e.g. by remote control devices
• CPC Class Codes: H04N 7/185 : from a mobile camera, e.g. for remote control

Abstract

An aerial system, including a processing system, an optical system, an actuation system and a lift mechanism, includes an autonomous photography and/or videography system 70, implemented, at least in part, by the processing system 22, the optical system 26, the actuation system 28 and the lift mechanism 32. The autonomous photograph and/or videography system performs the steps of establishing a desired flight trajectory, detecting a target, controlling the flight of the aerial system as a function of the desired flight trajectory relative to the target using the lift mechanism and controlling the camera to capture pictures and/or video.

Claims

1. An aerial system, comprising:

a body;

a lift mechanism coupled to the body;

an optical system including an optical sensor mounted to the body, the optical sensor movable with respect to the body by an actuation system;

an inertial measurement unit (IMU) coupled to the body including a vibration sensor configured to sense a vibration of the body; and

a processing system coupled to the lift mechanism, the optical system, the IMU, and the actuation system, the processing system includes a processor configured to;

establish a desired flight trajectory;

detect a target;

establish a position and a velocity of the target relative to the aerial system;

control the flight of the aerial system to maintain the aerial system at a distance from the target as a function of the desired flight trajectory and the established position and velocity of the target using the lift mechanism;

control the actuation system to automatically adjust an angle of the optical sensor relative to the body to maintain the target within a field of view of the optical sensor during the flight of the aerial system;

control the optical system to capture pictures and/or video;

retrieve a vibration measurement for each time period associated with each captured picture and/or video from the IMU;

select a moving window time interval and calculate a variance/standard deviation of vibration within the moving window time interval; and

automatically select pictures and/or videos by comparing the calculated the variance/standard deviation of vibration to a pre-defined threshold.

2. An aerial system, as set forth in claim 1, the system further including one or more position sensors, wherein the processing system is configured to integrate positioning data from the position sensors to establish a position of the aerial system, the processing system further including an auto-detection and tracking module, the auto-detection and tracking module being configured to:

detect and locate the target from a video feed received from the optical system; and

,

responsively establish the position and the velocity of the target relative to the aerial system.

3. An aerial system, as set forth in claim 2, wherein the target is a person and the auto-detection and tracking module utilizes facial recognition to detect and track the target.

4. An aerial system, as set forth in claim 2, further including additional sensor(s) associated with the target, the additional sensor(s) being configured to establish position data associated with the target to and to communicate the position data to the processing system.

5. An aerial system, as set forth in claim 1, wherein the desired flight trajectory is from a list of possible flight trajectories.

6. An aerial system, as set forth in claim 5, wherein the desired flight trajectory is selected from the list of possible flight trajectories by a user.

7. An aerial system, as set forth in claim 5, wherein the processing system selects the desired flight trajectory from the list of possible flight trajectories.

8. An aerial system, as set forth in claim 5, wherein the list of possible flight trajectories includes one or more of the following:

(a) a trajectory in which the aerial system follows the target from behind at a fixed distance, indefinitely or for a finite amount of time;

(b) a trajectory in which the aerial system may lead the target at the front while aiming at the target, keeping a fixed distance, indefinitely or for a finite amount of time;

(c) a trajectory in which the aerial system orbits around the target at a fixed distance with a constant or varying speed, indefinitely or for a finite amount of time;

(d) a trajectory in which the aerial system moves closer to or further away from the target at a specified camera aiming angle, with a constant/varying speed, for a finite amount of time;

(e) a trajectory in which the aerial system moves in a certain direction, in world coordinates or in target coordinates, while the optical system is aimed the target, with a constant/varying speed, for a finite amount of time;

(f) a trajectory in which one or more degrees of freedom associated with the aerial system are held constant and other degrees of freedom are used to track the target;

(g) a trajectory which has been pre-recorded by a user; and

,

(h) a trajectory comprising of any of the above trajectories, in a pre-defined, random, or pseudo-random sequence.

9. An aerial system, as set forth in claim 1, wherein the processing system includes an auto-selection module, the auto-selection module being configured to automatically select picture(s) and/or video and/or truncated video clip(s) using a lower frequency vibration filter.

10. An aerial system, as set forth in claim 9, wherein the auto-selection module automatically selects picture(s) and/or video and/or truncated video clip(s) based on over/under exposure and/or composition.

11. An aerial system, as set forth in claim 10, wherein the processing system includes an auto-editing module, the auto-editing module being configured to edit the selected picture(s), video, and/or video clip(s) based on predetermined editing parameters.

12. A method for operating an aerial system, the aerial system including a body, a lift mechanism, an optical system, and a body, the optical system including an optical sensor mounted to the body and movable with respect to the body by an actuation system, an inertial measurement unit (IMU) coupled to the body including a vibration sensor configured to sense a vibration of the body, and a processing system, method including the steps of:

establishing, by the processing system, a desired flight trajectory;

detecting a target;

establishing, by the processing system, a position and a velocity of the target relative to the aerial system;

controlling, by the processing system, the flight of the aerial system to maintain the aerial system at a distance from the target as a function of the desired flight trajectory and the established position and velocity of the target using the lift mechanism;

controlling, by the processing system, the actuation system to automatically adjust an angle of the optical sensor relative to the body to maintain the target within a field of view of the optical sensor during the flight of the aerial system;

controlling, by the processing system, the optical system to capture pictures and/or video

retrieving a vibration measurement for each time period associated with each captured picture and/or video from the IMU;

selecting a moving window time interval and calculating a variance/standard deviation of vibration within the moving window time interval; and

automatically selecting pictures and/or videos by comparing the calculated the variance/standard deviation of vibration to a pre-defined threshold.

13. A method system, as set forth in claim 12, including the steps of:

integrating positioning data from position sensors to establish a position of the aerial system;

detecting and locating the target from a video feed received from the optical system; and

,

responsively establishing the position and the velocity of the target relative to the aerial system.

14. A method, as set forth in claim 13, wherein the target is a person and the step of detecting and tracking the target utilizes facial recognition.

15. A method, as set forth in claim 13, further including the steps of establishing position data associated with the target using additional sensor(s) associated with the target to and communicating the position data to the aerial system.

16. A method aerial system, as set forth in claim 12, wherein the desired flight trajectory is from a list of possible flight trajectories.

17. A method, as set forth in claim 16, wherein the desired flight trajectory is selected from the list of possible flight trajectories by a user.

18. A method, as set forth in claim 16, wherein the aerial system automatically selects the desired flight trajectory from the list of possible flight trajectories.

19. A method, as set forth in claim 16, wherein the list of possible flight trajectories includes one or more of the following:

(a) a trajectory in which the aerial system follows the target from behind at a fixed distance, indefinitely or for a finite amount of time;

(b) a trajectory in which the aerial system may lead the target at the front while aiming at the target, keeping a fixed distance, indefinitely or for a finite amount of time;

(c) a trajectory in which the aerial system orbits around the target at a fixed distance with a constant or varying speed, indefinitely or for a finite amount of time;

(d) a trajectory in which the aerial system moves closer to or further away from the target at a specified camera aiming angle, with a constant/varying speed, for a finite amount of time;

(e) a trajectory in which the aerial system moves in a certain direction, in world coordinates or in target coordinates, while the optical system is aimed the target, with a constant/varying speed, for a finite amount of time;

(f) a trajectory in which one or more degrees of freedom associated with the aerial system are held constant and other degrees of freedom are used to track the target;

(g) a trajectory which has been pre-recorded by a user; and

,

(h) a trajectory comprising of any of the above trajectories, in a pre-defined, random, or pseudo-random sequence.

20. A method, as set forth in claim 12, including the step of automatically selecting picture(s) and/or video and/or truncated video clip(s) using a lower frequency vibration filter.

21. A method, as set forth in claim 20, wherein the step of automatically selecting picture(s) and/or video and/or truncated video clip(s) includes selecting picture(s) and/or video and/or truncated video clip(s) based on over/under exposure and/or composition.

22. A method, as set forth in claim 21, including the step of automatically editing the selected picture(s), video, and/or video clip(s) based on predetermined editing parameters.