System and method for providing autonomous photography and videography

• US 10,979,615 B2
• Filed: 06/21/2019
• Issued: 04/13/2021
• Est. Priority Date: 03/31/2017
• Status: Active Grant

• Current Assignee: Hangzhou Zero Zero Technology Co.Ltd (Shenzhen Zero Zero Infinite Technology Co. Ltd.)
• Original Assignee: Hangzhou Zero Zero Technology Co.Ltd (Shenzhen Zero Zero Infinite Technology Co. Ltd.)
• Inventors: Wang, Mengqiu, Liu, Lixin, Zhang, Tong, Lu, Jia
• Primary Examiner(s): Luo, Kate H
• Application Number:
  US16/448,535
• Publication Number:
  US 20190335084A1
• Time in Patent Office: 662 Days
• Field of Search: None
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64D 47/08 Arrangements of cameras
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: G01S 17/02 Systems using the reflection of electromagnetic waves other than radio waves G01S17/66 takes precedence
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 3/7864 T.V. type tracking systems
• CPC Class Codes: G05D 1/0011 associated with a remote control arrangement
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G06V 20/13 Satellite images
• CPC Class Codes: G06V 20/17 taken from planes or by drones
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 2201/07 Target detection
• CPC Class Codes: G06V 40/166 using acquisition arrangements
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/66 Remote control of cameras or camera parts, e.g. by remote control devices
• CPC Class Codes: H04N 7/185 : from a mobile camera, e.g. for remote control

Abstract

An aerial system, including a processing system, an optical system, an actuation system and a lift mechanism, includes an autonomous photography and/or videography system 70, implemented, at least in part, by the processing system 22, the optical system 26, the actuation system 28 and the lift mechanism 32. The autonomous photograph and/or videography system performs the steps of establishing a desired flight trajectory, detecting a target, controlling the flight of the aerial system as a function of the desired flight trajectory relative to the target using the lift mechanism and controlling the camera to capture pictures and/or video.

Claims

1. An aerial system, comprising:

a body;

a lift mechanism coupled to the body;

an optical system controllably mounted to the body by an actuation system; and

a processing system coupled to the lift mechanism, the optical system, and the actuation system, the processing system includes a processor configured to;

establish a first control strategy selected from a set of predefined control strategies;

establish a desired flight trajectory as a function of the first control strategy, the desired flight trajectory being established relative a current position of a target;

establish the current position and a current velocity of the target relative to the aerial system;

control the flight of the aerial system to maintain the aerial system at a distance from the target as a function of the desired flight trajectory and the established current position and current velocity of the target using the lift mechanism; and

control the optical system to capture pictures and/or video, wherein a quality of the pictures and/or video is determined by whether the target is within a preferred position within a frame of the pictures and/or video, an exposure value of the target in the pictures and/or video, and blurriness and image-stability of the target in the pictures and/or video.

2. An aerial system, as set forth in claim 1, wherein the set of predefined control strategies includes:

(a) following the target from behind at a fixed distance, indefinitely or for a finite amount of time;

(b) leading the target at the front while aiming at the target, keeping a fixed distance, indefinitely or for a finite amount of time;

(c) orbiting around the target at a fixed distance with a constant or varying speed, indefinitely or for a finite amount of time;

(d) moving closer to or further away from the target at a specified camera aiming angle, with a constant/varying speed, for a finite amount of time;

(e) moving in a certain direction, in world coordinates or in target coordinates, while the optical system is aimed the target, with a constant/varying speed, for a finite amount of time;

(f) holding one or more degrees of freedom associated with the aerial system constant and controlling other degrees of freedom to track the target; and

,

(g) following a pre-recorded trajectory relative to the target.

3. An aerial system, as set forth in claim 1, wherein the set of predefined control strategies are presented to a user, wherein the first control strategy is selected by the user.

4. An aerial system, as set forth in claim 1, wherein the first control strategy is selected from the set of predefined control strategies by the processor.

5. An aerial system, as set forth in claim 1, wherein a second control strategy is established and the desired flight trajectory is established as a function of the first control strategy and the second control strategy.

6. An aerial system, as set forth in claim 5, wherein the set of predefined control strategies are presented to a user, wherein the first control strategy and the second control strategy are selected by the user.

7. An aerial system, as set forth in claim 5, wherein the first control strategy and the second control strategy are selected from the set of predefined control strategies by the processor.

8. An aerial system, as set forth in claim 1, wherein the first control strategy and the second control strategy are selected from the set of predefined control strategies by the processor.

9. An aerial system, as set forth in claim 1, including an auto-shooting module configured to establish a current shooting mode from a set of predefined shooting modes, wherein the processing system is configured to control the optical system as a function of the established current shooting mode.

10. An aerial system, as set forth in claim 9, wherein each control strategy in the set of predefined control strategies has at least one shooting modes from the set of predefined shooting modes.

11. An aerial system, as set forth in claim 9, wherein set of predefined shooting modes are presented to a user and the established shooting mode is selected by the user.

12. An aerial system, as set forth in claim 1, the system further including one or more position sensors, wherein the processing system is configured to integrate positioning data from the position sensors to establish a position of the aerial system, the processing system further including an auto-detection and tracking module, the auto-detection and tracking module being configured to:

detect and locate the target from a video feed received from the optical system; and

,

responsively establish the position and the velocity of the target relative to the aerial system.

13. An aerial system, as set forth in claim 1, wherein the processing system includes an auto-selection module, the auto-selection module being configured to automatically select picture(s) and/or video and/or truncated video clip(s) based on predetermined criteria.

14. An aerial system, as set forth in claim 13, wherein the predetermined criteria includes over/under exposure and/or composition.

15. An aerial system, as set forth in claim 14, wherein the processing system includes an auto-editing module, the auto-editing module being configured to edit the selected picture(s), video, and/or video clip(s) based on predetermined editing parameters.

16. A method for operating an aerial system, the aerial system including a body, a lift mechanism, an optical system, and a body by an actuation system, method including the steps of:

establishing a first control strategy selected from a set of predefined control strategies;

establishing a desired flight trajectory as a function of the first control strategy, the desired flight trajectory being established relative a current position of a target;

establishing the current position and a current velocity of the target relative to the aerial system;

controlling the flight of the aerial system to maintain the aerial system at a distance from the target as a function of the desired flight trajectory and the established current position and current velocity of the target using the lift mechanism; and

controlling the optical system through the actuation system to capture pictures and/or video, wherein a quality of the pictures and/or video is determined by whether the target is within a preferred position within a frame of the pictures and/or video, an exposure value of the target in the pictures and/or video, and blurriness and image-stability of the target in the pictures and/or video.

17. A method, as set forth in claim 16, wherein the set of predefined control strategies includes:

following the target from behind at a fixed distance, indefinitely or for a finite amount of time;

leading the target at the front while aiming at the target, keeping a fixed distance, indefinitely or for a finite amount of time;

orbiting around the target at a fixed distance with a constant or varying speed, indefinitely or for a finite amount of time;

moving closer to or further away from the target at a specified camera aiming angle, with a constant/varying speed, for a finite amount of time;

moving in a certain direction, in world coordinates or in target coordinates, while the optical system is aimed the target, with a constant/varying speed, for a finite amount of time;

holding one or more degrees of freedom associated with the aerial system constant and

controlling other degrees of freedom to track the target; and

,

following a pre-recorded trajectory relative to the target.

18. A method, as set forth in claim 16, wherein the set of predefined control strategies are presented to a user, wherein the first control strategy is selected by the user.

19. A method, as set forth in claim 16, wherein the first control strategy is selected from the set of predefined control strategies by the processor.

20. A method, as set forth in claim 16, wherein a second control strategy is established and the desired flight trajectory is established as a function of the first control strategy and the second control strategy.

21. A method, as set forth in claim 20, wherein the set of predefined control strategies are presented to a user, wherein the first control strategy and the second control strategy are selected by the user.

22. A method, as set forth in claim 20, wherein the first control strategy and the second control strategy are selected from the set of predefined control strategies by the processor.

23. A method, as set forth in claim 16, wherein the first control strategy and the second control strategy are selected from the set of predefined control strategies by the processor.

24. A method, as set forth in claim 16, including the step of establishing a current shooting mode from a set of predefined shooting modes, wherein the processing system is configured to control the optical system as a function of the established current shooting mode.

25. A method, as set forth in claim 24, wherein each control strategy in the set of predefined control strategies has at least one shooting modes from the set of predefined shooting modes.

26. A method, as set forth in claim 24, wherein set of predefined shooting modes are presented to a user and the established shooting mode is selected by the user.

27. A method, as set forth in claim 16, including the steps of:

integrating positioning data from position sensors to establish a position of the aerial system;

detecting and locating the target from a video feed received from the optical system; and

, responsively establishing the position and the velocity of the target relative to the aerial system.

28. A method, as set forth in claim 16, including the step of automatically select picture(s) and/or video and/or truncated video clip(s) based on predetermined criteria.

29. A method, as set forth in claim 28, wherein the predetermined criteria includes over/under exposure and/or composition.

30. A method, as set forth in claim 29, including the step of automatically editing the selected picture(s), video, and/or video clip(s) based on predetermined editing parameters.