DETECTING OPTICAL DISCREPANCIES IN CAPTURED IMAGES

• US 20190004543A1
• Filed: 07/03/2017
• Published: 01/03/2019
• Est. Priority Date: 07/03/2017
• Status: Active Grant

• Current Assignee: Skydio, Inc.
• Original Assignee: Skydio, Inc.
• Inventors: Kennedy, Ryan, Henry, Peter, Bachrach, Abraham
• Granted Patent: US 10,379,545 B2
• Time in Patent Office: Days
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64D 47/08 Arrangements of cameras
• CPC Class Codes: B64U 10/14 with four distinct rotor axes, e.g. quadcopters
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: B64U 30/20 Rotors; Rotor supports
• CPC Class Codes: G05D 1/102 specially adapted for vertical take-off of aircraft
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/30168 Image quality inspection
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: H04N 13/00 Stereoscopic video systems; Multi-view video systems; Details thereof
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/69 Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04N 23/71 Circuitry for evaluating the brightness variation
• CPC Class Codes: H04N 23/811 by dust removal, e.g. from surfaces of the image sensor or processing of the image signal output by the electronic image sensor
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

Embodiments are described for detecting optical discrepancies associated with image capture analyzing pixels in multiple images corresponding to common points of reference in a physical environment. In an embodiment, photometric error values are averaged over time to compute the mean error at each pixel. Once the estimate of the mean error has a sufficient number of updates above a specified value, the estimate is thresholded to provide a mask of any optical discrepancies occurring in the stereo pair of images. Applications include detecting optical discrepancies in images captured for use by a visual navigation system in guiding an autonomous vehicle (e.g., an unmanned aerial vehicle).

Claims

1. A method for alleviating the effects of optical discrepancies in images used to guide the autonomous flight of an unmanned aerial vehicle (UAV) through a physical environment, the images captured by an image capture device coupled to the UAV, the method comprising:

receiving, from the image capture device, a first image of the physical environment from a first position and second image of the physical environment from a second position

processing the first image and the second image to compare photometric characteristics of pixels in the first image and the second image corresponding to a common point of reference in the physical environment;

detecting an optical discrepancy associated with the capture of the first image and/or the second image based on the processing;

determining that the detected optical discrepancy is indicative of an unreliable portion of the first image and/or second image;

applying a mask to the first image and/or second image based on the unreliable portion; and

generating control commands configured to guide autonomous flight of the UAV based on the received first image and second image with the applied mask so as to ignore the unreliable portion.

2. A method comprising:

receiving, by a computer system, a first image of a physical environment from a first position and second image of the physical environment from a second position;

processing, by the computer system, the first image and the second image to compare photometric characteristics of pixels in the first image and the second image corresponding to a common point of reference in the physical environment; and

detecting, by the computer system, an optical discrepancy associated with the capture of the first image and/or the second image based on the processing.

3. The method of claim 2, wherein the first image and second image are utilized by a visual navigation system to guide autonomous flight by an unmanned aerial vehicle (UAV), the method further comprising:

determining that the detected optical discrepancy is indicative of an unreliable portion of the first image and/or second image; and

causing the visual navigation system to ignore the unreliable portion of the first image and/or second image.

4. The method of claim 2, wherein the first image and second image are captured by an image capture device onboard an unmanned aerial vehicle (UAV), the method further comprising:

causing the UAV to maneuver to adjust a position of the image capture device; and

continuing to process the first image and the second image to confirm detection of the optical discrepancy.

5. The method of claim 2, wherein detecting the optical discrepancy includes:

tracking, over a period of time, a difference between the photometric characteristics of pixels in the first image and in the second image corresponding to the common point of reference in the physical environment.

6. The method of claim 2, further comprising:

determining a cause of the optical discrepancy based on a characteristic of the optical discrepancy.

7. The method of claim 6, wherein the determined cause of the optical discrepancy includes any of:

a foreign material on a surface of a lens of an image capture device that captured the first image and/or the second image;

an imperfection in an optical component of the image capture device that captured the first image and/or the second image;

a failure in a processing system associated with the image capture device that captured the first image and/or the second image;

or

an error in calibration of the image capture device that captured the first image and/or the second image.

8. The method of claim 2, wherein processing the first image and the second image includes calculating a plurality of photometric error values for pixels in the first image and the second image, wherein a particular photometric error value of the plurality of photometric error values is based on a comparison between a first pixel in the first image and a second pixel in the second image, the first pixel and second pixel corresponding to the common point of reference in the physical environment.

9. The method of claim 8, wherein calculating the particular photometric error value includes:

determining a photometric value of the first pixel in the first image, the first pixel corresponding to the common point of reference;

determining a plurality of photometric values of a plurality of pixels in the second image along an epipolar line corresponding to common point of reference

calculating a plurality of photometric error values based on a difference between the photometric value of the first pixel in the first image and each of the plurality of photometric values of the plurality of pixels in the second image;

wherein the particular photometric error value is one of the plurality of calculated photometric error values.

10. The method of claim 8, wherein the particular photometric error value is the minimum photometric error value among the plurality of calculated photometric error values.

11. The method of claim 8, wherein the second pixel and the plurality of other pixels in the second image are along a common row in the second image if the first image and the second image are rectified.

12. The method of claim 8, wherein detecting the optical discrepancy includes:

generating a threshold map of average photometric error across a field of view corresponding to the first image and the second image; and

identifying a region of the threshold map that includes average photometric error values above a particular threshold;

wherein the identified region is indicative of the optical discrepancy.

13. The method of claim 12, further comprising:

identifying a cause of the optical discrepancy by analyzing a characteristic of the identified region of the threshold map.

14. The method of claim 13, wherein analyzing the characteristic of the identified region includes applying a machine-learning based appearance model to classify the characteristic of the identified region as indicative of one or more of a plurality of possible causes of the optical discrepancy.

15. The method of claim 13, wherein the characteristic of the identified region includes any of:

a shape of the identified region;

a size of the identified region;

a location of the identified region in the threshold map;

or

a duration of the identified region.

16. The method of claim 12, wherein the first image and second image are utilized by a visual navigation system to guide autonomous flight by an unmanned aerial vehicle, the method further comprising:

determining that the region of the threshold map that includes average photometric error values above the particular threshold corresponds with an unreliable portion of the first image and/or second image;

generating an image mask based on the threshold map; and

causing the visual navigation system to ignore the unreliable portion of the first image and/or second image by applying the generated image mask to the first image and/or the second image.

17. The method of claim 12, further comprising:

causing display of the generated threshold map.

18. The method of claim 2, wherein the first image is captured by a first camera and the second image is captured by a second camera.

19. The method of claim 18, wherein the first camera and the second camera are part of a stereo image capture device.

20. The method of claim 2, wherein the first image is captured by an image capture device at a first point in time and the second image is captured by the image capture device at a second point in time.

21. The method of claim 2, wherein:

the first image is captured by a first image capture device, the first image capture device configured to capture images of the physical environment for use by a visual navigation system to guide autonomous flight by the UAV; and

the second image is captured by a second image capture device, the second image capture device configured to capture images of physical environment for display via a display device.

22. The method of claim 2, wherein processing the first image and the second image includes rectifying the first image and the second image.

23. The method of claim 22, wherein rectifying the first image and the second image includes transforming the first image and/or the second image.

24. The method of claim 2, wherein processing the first image and the second image includes down-sampling the first image and/or the second image.

25. The method of claim 2, further comprising:

generating an output indicative of the optical discrepancy.

26. The method of claim 2, further comprising:

automatically adjusting the first image and/or second image to correct for the detected optical discrepancy.

27. The method of claim 2, further comprising:

automatically adjusting a calibration setting of an image capture device associated with the first image or the second image to correct for the detected optical discrepancy.

28. An unmanned aerial vehicle (UAV) autonomous navigation system comprising:

a processing unit

a memory unit coupled to the processing unit, the memory unit having instructions stored thereon, which when executed by the processing unit cause the system to;

receive, from an image capture device, a first image of a physical environment from a first position and second image of the physical environment from a second position

process the first image and the second image to compare photometric characteristics of pixels in the first image and the second image corresponding to a common point of reference in the physical environment;

detect an optical discrepancy associated with the capture of the first image and/or the second image based on the processing;

determine that the detected optical discrepancy is indicative of an unreliable portion of the first image and/or second image;

apply a mask to first image and/or second image based on the unreliable portion; and

generate control commands configured to guide autonomous flight of the UAV based on the received first image and second image with the applied mask so as to ignore the unreliable portion.

29. An unmanned aerial vehicle UAV) configured for autonomous flight through a physical environment, the UAV comprising:

a first image capture device configured to capture images of the physical environment;

a second image captured device configured to capture images of the physical environment;

a propulsion system;

a visual navigation system configured to;

process images received from the first image capture device; and

generate control commands based at least in part on the images received from the first image capture device, the control commands configured to maneuver the UAV using the propulsion system;

a subject tracking system configured to automatically adjust an orientation of the second image capture device relative to the UAV so as to track image capture of a subject in the physical environment while the UAV is in flight; and

an optical discrepancy detection system configured to;

receive, from the first image capture device, a first image of the physical environment from a first position;

receive, from the second image capture device, a second image of the physical environment from a second position;

process the first image and the second image to compare photometric characteristics of pixels in the first image and the second image corresponding to a common point of reference in the physical environment; and

detect an optical discrepancy associated with the capture of the first image and/or the second image based on the processing.

30. The UAV of claim 29, wherein the first image capture device includes an array of stereoscopic cameras arranged along a perimeter of the UAV so as to provide a stereoscopic view at a plurality of angles around the UAV.

31. The UAV of claim 29, wherein the subject tracking system includes a hybrid mechanical-digital gimbal system, the hybrid mechanical digital gimbal system including:

a mechanical gimbal configured to rotate the second image capture device about a first axis relative to the UAV; and

an image processing system configured to rotate images captured by the second image capture device about a second axis relative to the UAV.

32. The UAV of claim 29, further comprising:

an image processing system configured to automatically adjust the first image and/or second image to correct for the detected optical discrepancy.

33. The UAV of claim 29, further comprising:

a notification system configured to;

generate a notification in response to detecting the optical discrepancy; and

transmit the notification, via a wireless communication link, to a remote computing device.

34. The UAV of claim 29, wherein the optical discrepancy detection system is further configured to:

determine that the detected optical discrepancy is indicative of an unreliable portion of the first image and/or second image; and

applying a mask to the first image and/or second image based on the unreliable portion.

35. The UAV of claim 34, wherein the visual navigation system is configured to ignore masked regions of the first image when generating control commands to maneuver the UAV.