Multi-camera system and method of use

• US 10,303,185 B2
• Filed: 08/01/2018
• Issued: 05/28/2019
• Est. Priority Date: 01/23/2017
• Status: Active Grant

• Current Assignee: Hangzhou Zero Zero Technology Co.Ltd (Shenzhen Zero Zero Infinite Technology Co. Ltd.)
• Original Assignee: Hangzhou Zero Zero Technology Co.Ltd (Shenzhen Zero Zero Infinite Technology Co. Ltd.)
• Inventors: Zhang, Tong, Li, Wei, Li, Xiang, Lu, Jia
• Primary Examiner(s): Saini, Amandeep
• Application Number:
  US16/052,494
• Publication Number:
  US 20180341278A1
• Time in Patent Office: 300 Days
• Field of Search: 382104
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64D 47/08 Arrangements of cameras
• CPC Class Codes: B64U 10/10 Rotorcrafts
• CPC Class Codes: B64U 10/14 with four distinct rotor axes, e.g. quadcopters
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: G05D 1/102 specially adapted for vertical take-off of aircraft
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/51 Housings
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 23/66 : Remote control of cameras or camera parts, e.g. by remote control devices
• CPC Class Codes: H04N 23/74 : by influencing the scene brightness using illuminating means
• CPC Class Codes: H04N 23/80 : Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/185 : from a mobile camera, e.g. for remote control

Abstract

A method for operating a system including a plurality of cameras, the method including: selecting a subset of the cameras, determining a subset of pixels captured by the camera subset, determining a pixel depth associated with each pixel of the pixel subset, and controlling system operation based on the pixel depth.

Claims

1. A method for aircraft operation comprising, at an aircraft comprising a set of cameras:

selecting a first camera group from the set of cameras, the first camera group comprising a lateral camera and a first downward-facing camera, wherein a lateral camera view region and a first downward-facing camera view region each comprise a first overlapped subset;

selecting a second camera group from the set of cameras, the second camera group comprising the first downward-facing camera and a second downward-facing camera, wherein the first downward-facing camera view region and a second downward-facing camera view region each comprise a second overlapped subset;

sampling a set of images, comprising;

at the first downward-facing camera, sampling a first image during a first time period;

at the second downward-facing camera, sampling a second image during the first time period; and

at the lateral camera, sampling a third image during the first time period;

based on the first camera group, determining a first dataset associated with a first region of the first image and a region of the third image, wherein the first region of the first image and the region of the third image are each representative of a first obstacle within the first overlapped subset;

based on the first dataset, determining a lateral clearance between the aircraft and the first obstacle;

based on the second camera group, determining a second dataset associated with a second region of the first image and a region of the second image, wherein the second region of the first image and the region of the second image are each representative of a second obstacle within the second overlapped subset;

based on the second dataset, determining an downward clearance between the aircraft and the second obstacle; and

based on the lateral clearance and the downward clearance, controlling flight of the aircraft.

2. The method of claim 1, wherein the aircraft defines a lateral plane, the aircraft further comprising:

a set of rotors comprising a first rotor, a second rotor, a third rotor, and a fourth rotor, each rotor of the set defining a respective rotor rotation axis and a respective rotor plane normal the respective rotor rotation axis, wherein each respective rotor plane is coplanar with the lateral plane; and

a housing defining a perimeter, the perimeter surrounding the set of rotors within the lateral plane.

3. The method of claim 2, wherein:

the housing comprises a central portion arranged between the first and second rotors; and

the first and second downward-facing cameras are mounted to the central portion.

4. The method of claim 3, wherein the lateral camera is arranged on the perimeter.

5. The method of claim 1, further comprising, based on the lateral clearance and an aircraft velocity, determining a potential collision with the first obstacle, wherein controlling flight of the aircraft comprises changing the aircraft velocity to avoid the potential collision.

6. The method of claim 1, wherein selecting the first camera group comprises selecting the lateral camera based on an aircraft velocity vector, wherein a lateral camera view vector within the lateral camera view region is parallel the aircraft velocity vector.

7. The method of claim 6, further comprising, after the first time period:

controlling the aircraft to change velocity to a second aircraft velocity vector non-parallel the aircraft velocity vector;

selecting a third camera group from the set of cameras based on the second aircraft velocity vector, the third camera group comprising a second lateral camera and the second downward-facing camera, wherein a second lateral camera view region and the second downward-facing camera view region each comprise a third overlapped subset;

sampling a second set of images, comprising;

at the second downward-facing camera, sampling a fourth image during a second time period; and

at the second lateral camera, sampling a fifth image during the second time period;

based on the third camera group, selecting a third dataset comprising a region of the fourth image and a region of the fifth image, wherein the region of the fourth image and the region of the fifth image are each representative of a third obstacle within the third overlapped subset;

based on the third dataset, determining a second lateral clearance between the aircraft and the third obstacle; and

based on the second lateral clearance, controlling flight of the aircraft.

8. The aircraft of claim 1, wherein the first image, the second image, and the third image are sampled at a first time.

9. A method for aircraft operation comprising, at an aircraft comprising a first camera, a second camera, and a third camera:

sampling a set of images, comprising;

at the first camera, sampling a first image;

concurrent with sampling the first image, at the second camera, sampling a second image, wherein a first camera view region and a second camera view region both comprise a first overlapped subset; and

concurrent with sampling the first image, at the third camera, sampling a third image, wherein a third camera view region and the second camera view region both comprise a second overlapped subset;

determining a first dataset associated with a region of the first image and a first region of the second image, wherein the region of the first image and the first region of the second image are both representative of a first obstacle within the first overlapped subset;

based on the first dataset, determining a first obstacle clearance between the aircraft and the first obstacle;

determining a second dataset associated with a second region of the second image and a region of the third image, wherein the region of the third image and the second region of the second image are both representative of a second obstacle within the second overlapped subset and outside the first overlapped subset;

based on the second dataset, determining a second obstacle clearance between the aircraft and the second obstacle; and

based on the first obstacle clearance and the second obstacle clearance, controlling flight of the aircraft.

10. The method of claim 9, wherein the aircraft defines a lateral plane, the aircraft further comprising:

a set of rotors comprising a first rotor, a second rotor, a third rotor, and a fourth rotor, each rotor of the set defining a respective rotor rotation axis and a respective rotor plane normal the respective rotor rotation axis, wherein each respective rotor plane is coplanar with the lateral plane; and

a housing defining a perimeter, the perimeter surrounding the set of rotors within the lateral plane.

11. The method of claim 10, wherein the first and second cameras are arranged on the perimeter.

12. The method of claim 11, wherein:

the aircraft defines a central axis normal to the lateral plane, wherein the central axis is arranged within a convex hull of the rotor rotation axes of the set of rotors;

the perimeter comprises;

a first side arranged proximal the first and second rotors with respect to the central axis; and

a second side arranged proximal the third rotor and distal the first rotor with respect to the central axis;

the first and second cameras are arranged along the first side; and

the third camera is arranged along the second side.

13. The method of claim 12, wherein the second side is arranged proximal the fourth rotor with respect to the central axis.

14. The method of claim 11, wherein:

the housing comprises a central portion arranged between the first and second rotors; and

the third camera is mounted to the central portion.

15. The method of claim 9, wherein:

a first angle between a first camera central view axis and a second camera central view axis is greater than a threshold angle; and

a second angle between a second camera central view axis and the third camera central view axis is greater than the threshold angle.

16. The method of claim 15, wherein the threshold angle is 60 degrees.

17. The method of claim 9, wherein:

the third camera view region further comprises the first overlapped subset; and

the first dataset further comprises a second region of the third image, the second region of the third image representative of the first obstacle.

18. The method of claim 9, wherein a first camera view vector within the first camera view region is parallel an aircraft velocity vector and a second camera view vector within the second camera view region is parallel the aircraft velocity vector.

19. An aircraft comprising:

a housing;

a set of cameras mechanically coupled to the housing, the set of cameras comprising a first camera, a second camera, and a third camera;

a motive mechanism mechanically coupled to the housing; and

a processor mechanically coupled to the housing;

wherein the processor is configured to;

receive a set of images from the set of cameras, the set of images comprising;

a first image sampled by the first camera, the first image representative of a first obstacle;

a second image sampled by the second camera, the second image representative of the first obstacle and a second obstacle; and

a third image sampled by the third camera, the third image representative of the second obstacle;

based on the first and second images, determine a first obstacle clearance between the aircraft and the first obstacle;

based on the third image, determine a second obstacle clearance between the aircraft and the second obstacle; and

based on the first obstacle clearance and the second obstacle clearance, control operation of the motive mechanism to control flight of the aircraft.

20. The aircraft of claim 19, wherein:

the aircraft defines a lateral plane;

the aircraft further comprises a set of rotors comprising a first rotor, a second rotor, a third rotor, and a fourth rotor, each rotor of the set defining a respective rotor rotation axis and a respective rotor plane normal the respective rotor rotation axis, wherein each respective rotor plane is coplanar with the lateral plane;

the housing defines a perimeter, the perimeter surrounding the set of rotors within the lateral plane;

the perimeter defines a first side arranged proximal the first and second rotors; and

the first and second cameras are arranged along the first side.

21. The aircraft of claim 19, wherein:

the housing comprises a central portion arranged between the first and second rotors; and

the second and third cameras are mounted to the central portion.