Detecting optical discrepancies in captured images

• US 10,379,545 B2
• Filed: 07/03/2017
• Issued: 08/13/2019
• Est. Priority Date: 07/03/2017
• Status: Active Grant

• Current Assignee: Skydio, Inc.
• Original Assignee: Skydio, Inc.
• Inventors: Kennedy, Ryan, Henry, Peter, Bachrach, Abraham
• Primary Examiner(s): Shin, Soo
• Application Number:
  US15/641,021
• Publication Number:
  US 20190004543A1
• Time in Patent Office: 771 Days
• Field of Search: None
• CPC Class Codes: B64U 10/14 with four distinct rotor axes, e.g. quadcopters
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: B64U 30/20 Rotors; Rotor supports
• CPC Class Codes: G05D 1/102 specially adapted for vertical take-off of aircraft
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/30168 Image quality inspection
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: H04N 13/00 Stereoscopic video systems; Multi-view video systems; Details thereof
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/69 Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04N 23/71 Circuitry for evaluating the brightness variation
• CPC Class Codes: H04N 23/811 by dust removal, e.g. from surfaces of the image sensor or processing of the image signal output by the electronic image sensor
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

Embodiments are described for detecting optical discrepancies associated with image capture analyzing pixels in multiple images corresponding to common points of reference in a physical environment. In an embodiment, photometric error values are averaged over time to compute the mean error at each pixel. Once the estimate of the mean error has a sufficient number of updates above a specified value, the estimate is thresholded to provide a mask of any optical discrepancies occurring in the stereo pair of images. Applications include detecting optical discrepancies in images captured for use by a visual navigation system in guiding an autonomous vehicle (e.g., an unmanned aerial vehicle).

Claims

1. An unmanned aerial vehicle UAV) configured for autonomous flight through a physical environment, the UAV comprising:

a first image capture device;

a second image capture device, the second image capture device different than the first image capture device;

a propulsion system;

a visual navigation system configured to;

process a first plurality of images received from the first image capture device; and

generate control commands based at least in part on the first plurality of images received from the first image capture device, the control commands configured to maneuver the UAV using the propulsion system;

a subject tracking system configured to automatically adjust an orientation of the second image capture device relative to the UAV such that a subject in the physical environment remains in a field of view of the second image capture device while the UAV is in flight; and

an optical discrepancy detection system configured to;

receive, from the first image capture device, a first image of the physical environment from a first position;

receive, from the second image capture device, a second image of the physical environment from a second position;

process the first image and the second image to compare photometric characteristics of pixels in the first image and the second image corresponding to a common point of reference in the physical environment; and

detect an optical discrepancy associated with the capture of the first image and/or the second image based on the processing.

2. The UAV of claim 1, wherein the first image capture device includes an array of stereoscopic cameras arranged along a perimeter of the UAV so as to provide a stereoscopic view at a plurality of angles around the UAV.

3. The UAV of claim 1, wherein the subject tracking system includes a hybrid mechanical-digital gimbal system, the hybrid mechanical digital gimbal system including:

a mechanical gimbal configured to rotate the second image capture device about a first axis relative to the UAV; and

an image processing system configured to rotate one or more of a second plurality of images captured by the second image capture device about a second axis relative to the UAV.

4. The UAV of claim 1, further comprising:

an image processing system configured to automatically adjust the first image and/or second image to correct for the detected optical discrepancy.

5. The UAV of claim 1, further comprising:

a notification system configured to;

generate a notification in response to detecting the optical discrepancy; and

transmit the notification, via a wireless communication link, to a remote computing device.

6. The UAV of claim 1, wherein the optical discrepancy detection system is further configured to:

identify a region of the first image and/or second image corresponding to the detected optical discrepancy; and

apply a mask to the region of the first image and/or second image corresponding to the detected optical discrepancy.

7. The UAV of claim 6, wherein applying the mask to the region of the first image and/or second image causes the visual navigation system to ignore the region of the first image and/or second image when generating control commands to maneuver the UAV.