Multi-camera system and method of use

• US 10,067,513 B2
• Filed: 01/19/2018
• Issued: 09/04/2018
• Est. Priority Date: 01/23/2017
• Status: Active Grant

• Current Assignee: Hangzhou Zero Zero Technology Co.Ltd (Shenzhen Zero Zero Infinite Technology Co. Ltd.)
• Original Assignee: Hangzhou Zero Zero Technology Co.Ltd (Shenzhen Zero Zero Infinite Technology Co. Ltd.)
• Inventors: Zhang, Tong, Li, Wei, Li, Xiang, Lu, Jia
• Primary Examiner(s): Saini, Amandeep
• Application Number:
  US15/875,967
• Publication Number:
  US 20180210465A1
• Time in Patent Office: 228 Days
• Field of Search: 382104
• CPC Class Codes: B64U 10/10 Rotorcrafts
• CPC Class Codes: B64U 10/14 with four distinct rotor axes, e.g. quadcopters
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: G05D 1/102 specially adapted for vertical take-off of aircraft
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/51 Housings
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 23/66 Remote control of cameras or camera parts, e.g. by remote control devices
• CPC Class Codes: H04N 23/74 by influencing the scene brightness using illuminating means
• CPC Class Codes: H04N 23/80 : Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/185 : from a mobile camera, e.g. for remote control

Abstract

A method for operating a system including a plurality of cameras, the method including: selecting a subset of the cameras, determining a subset of pixels captured by the camera subset, determining a pixel depth associated with each pixel of the pixel subset, and controlling system operation based on the pixel depth.

Claims

1. A method for aircraft operation comprising, at an aircraft comprising a set of cameras:

selecting a first camera group from the set of cameras, the first camera group comprising a lateral camera and a first upward-facing camera, wherein a lateral camera view region and a first upward-facing camera view region each comprise a first overlapped subset;

selecting a second camera group from the set of cameras, the second camera group comprising the first upward-facing camera and a second upward-facing camera, wherein the first upward-facing camera view region and a second upward-facing camera view region each comprise a second overlapped subset;

sampling a set of images, comprising;

at the first upward-facing camera, sampling a first image at a first time;

at the second upward-facing camera, sampling a second image substantially at the first time; and

at the lateral camera, sampling a third image substantially at the first time;

based on the first camera group, selecting a first dataset comprising a first region of the first image and a region of the third image, wherein the first region of the first image and the region of the third image are each representative of a first obstacle within the first overlapped subset;

based on the first dataset, determining a lateral clearance between the aircraft and the first obstacle;

based on the second camera group, selecting a second dataset comprising a second region of the first image and a region of the second image, wherein the second region of the first image and the region of the second image are each representative of a second obstacle within the second overlapped subset;

based on the second dataset, determining an upward clearance between the aircraft and the second obstacle; and

based on the lateral clearance and the upward clearance, controlling flight of the aircraft.

2. The method of claim 1, wherein:

a first angle between a lateral camera central view axis and a first upward-facing camera central view axis is greater than a threshold angle; and

a second angle between the lateral camera central view axis and a second upward-facing camera central view axis is greater than the threshold angle.

3. The method of claim 2, wherein the threshold angle is 60 degrees.

4. The method of claim 1, wherein selecting the second dataset comprises selecting the second region of the first image and the region of the second image based on an orientation of the aircraft.

5. The method of claim 4, wherein, at the first time, the aircraft and the second obstacle both intersect a vertical axis, the vertical axis parallel a gravity vector.

6. The method of claim 1, further comprising, based on the lateral clearance and an aircraft velocity, determining a potential collision with the first obstacle, wherein controlling flight of the aircraft comprises changing the aircraft velocity to avoid the potential collision.

7. The method of claim 1, wherein:

the second obstacle is a ceiling of a room containing the aircraft; and

controlling flight of the aircraft comprises substantially maintaining the upward clearance at a target ceiling clearance value.

8. The method of claim 1, wherein the first upward-facing camera comprises a fisheye lens.

9. The method of claim 8, wherein the second upward-facing camera comprises a second fisheye lens and the lateral camera comprises a substantially rectilinear lens.

10. The method of claim 1, wherein selecting the first camera group comprises selecting the lateral camera based on an aircraft velocity vector, wherein a lateral camera view vector within the lateral camera view region is parallel the aircraft velocity vector.

11. The method of claim 10, further comprising, after the first time:

controlling the aircraft to change velocity to a second aircraft velocity vector substantially non-parallel the aircraft velocity vector;

selecting a third camera group from the set of cameras based on the second aircraft velocity vector, the third camera group comprising a second lateral camera and the second upward-facing camera, wherein a second lateral camera view region and the second upward-facing camera view region each comprise a third overlapped subset;

sampling a second set of images, comprising;

at the second upward-facing camera, sampling a fourth image at a second time; and

at the second lateral camera, sampling a fifth image substantially at the second time;

based on the third camera group, selecting a third dataset comprising a region of the fourth image and a region of the fifth image, wherein the region of the fourth image and the region of the fifth image are each representative of a third obstacle within the third overlapped subset;

based on the third dataset, determining a second lateral clearance between the aircraft and the third obstacle; and

based on the second lateral clearance, controlling flight of the aircraft.

12. The method of claim 1, wherein:

the first camera group further comprises a second lateral camera, wherein a second lateral camera view region comprises the first overlapped subset;

sampling the set of images further comprises, at the second lateral camera, sampling a fourth image substantially at the first time; and

the first dataset further comprises a region of the fourth image representative of the first obstacle.

13. A method for aircraft operation comprising, at an aircraft comprising a set of cameras:

based on an aircraft traversal vector, selecting a camera group from the set of cameras, the camera group comprising;

a first camera, a second camera, and a third camera, wherein;

a first camera view region, a second camera view region, and a third camera view region each comprise an overlapped subset;

a first angle between a first camera central view axis and a third camera central view axis is greater than a threshold angle; and

a second angle between a second camera central view axis and the third camera central view axis is greater than the threshold angle;

sampling a set of images, comprising;

at the first camera, sampling a first image;

substantially concurrent with sampling the first image, at the second camera, sampling a second image; and

substantially concurrent with sampling the first image, at the third camera, sampling a third image;

based on the camera group, selecting a first dataset comprising a region of the first image, a region of the second image, and a first region of the third image, wherein the region of the first image, the region of the second image, and the first region of the third image are each representative of a first obstacle within the overlapped subset;

based on the first dataset, determining a first obstacle clearance between the aircraft and the first obstacle;

selecting a second dataset comprising a second region of the third image, wherein the second region of the third image is representative of a second obstacle outside the overlapped subset;

based on the second dataset, determining a second obstacle clearance between the aircraft and the second obstacle; and

based on the first obstacle clearance and the second obstacle clearance, controlling flight of the aircraft.

14. The method of claim 13, wherein the threshold angle is 75 degrees.

15. The method of claim 13, wherein a first camera view vector within the first camera view region is parallel the aircraft traversal vector and a second camera view vector within the second camera view region is parallel the aircraft traversal vector.

16. The method of claim 13, further comprising:

selecting a second camera group from the set of cameras, the second camera group comprising the third camera and a fourth camera, wherein;

the third camera view region and a fourth camera view region each comprise a second overlapped subset;

a third angle between the first camera central view axis and a fourth camera central view axis is greater than the threshold angle; and

a fourth angle between the second camera central view axis and the fourth camera central view axis is greater than the threshold angle; and

substantially concurrent with sampling the first image, at the fourth camera, sampling a fourth image, wherein the second dataset further comprises a region of the fourth image representative of the second obstacle.

17. The method of claim 13, wherein selecting the second dataset comprises selecting the second region of the third image based on an orientation of the aircraft.

18. The method of claim 17, wherein, at the first time, the aircraft and the second obstacle both intersect a vertical axis, the vertical axis parallel a gravity vector.

19. The method of claim 13, wherein the first camera comprises a first substantially rectilinear lens, the second camera comprises a second substantially rectilinear lens, and the third camera comprises a fisheye lens.

20. The method of claim 13, wherein controlling flight of the aircraft comprises:

determining that the first obstacle clearance is below a target clearance value; and

in response to determining that the first obstacle clearance is below the target clearance value, controlling the aircraft to enter a stationary hovering mode.