Video capture, processing, calibration, computational fiber artifact removal, and light-field pipeline

• US 10,565,734 B2
• Filed: 03/07/2017
• Issued: 02/18/2020
• Est. Priority Date: 04/15/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Bevensee, Brendan, Du, Tingfang, Karafin, Jon, Merritt, Joel, Petrovich, Duane, Spor, Gareth
• Primary Examiner(s): Berardesca, Paul M
• Application Number:
  US15/451,831
• Publication Number:
  US 20170243373A1
• Time in Patent Office: 1,078 Days
• CPC Class Codes: G01S 17/08 for measuring distance only indirect measurement G01S17/46; active triangulation systems G01S17/48
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 7/4817 relating to scanning
• CPC Class Codes: G02B 3/0056 arranged along two different directions in a plane, e.g. honeycomb arrangement of lenses G02B3/0043 takes precedence; miniaturised objectives for electronic devices employing wafer level optics G02B13/0085
• CPC Class Codes: G02B 6/08 with fibre bundle in form of plate
• CPC Class Codes: G03B 17/17 with reflectors arranged in beam forming the photographic image, e.g. for reducing dimensions of camera
• CPC Class Codes: G03B 35/10 having single camera with stereoscopic-base-defining system
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/557 from light fields, e.g. from plenoptic cameras
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/57 Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 23/957 : Light-field or plenoptic cameras or camera modules
• CPC Class Codes: H04N 25/61 : the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4"
• CPC Class Codes: H04N 25/705 : Pixels for depth measurement, e.g. RGBZ
• CPC Class Codes: H04N 7/22 : Adaptations for optical transmission

Abstract

An image capture system includes a plurality of image sensors arranged in a pattern such that gaps exist between adjacent image sensors of the plurality of image sensors. Each of the image sensors may be configured to capture sensor image data. The image capture system may also have a main lens configured to direct incoming light along an optical path, a microlens array positioned within the optical path, and a plurality of tapered fiber optic bundles. Each tapered fiber optic bundle may have a leading end positioned within the optical path, and a trailing end positioned proximate one of the image sensors. The leading end may have a larger cross-sectional area than the trailing end. Sensor data from the image sensors may be combined to generate a single light-field image that is substantially unaffected by the gaps.

Claims

1. A method for capturing and processing image data, the method comprising:

receiving incoming light along an optical path in an image capture device, the image capture device comprising;

a plurality of image sensors arranged in a pattern such that gaps exist between adjacent image sensors of the plurality of image sensors, wherein the plurality of image sensors includes a first image sensor and a second image sensor;

a main lens;

a microlens array;

a plurality of tapered fiber optic bundles, each of which comprises a leading end positioned within the optical path, and a trailing end positioned proximate one of the plurality of image sensors, the leading end having a larger cross-sectional area than the trailing end, wherein the plurality of tapered fiber optic bundles includes;

a first tapered fiber optic bundle of a first length positioned proximate the first image sensor;

a second tapered fiber optic bundle of a second length longer than the first length positioned proximate the second image sensor; and

a beam splitter to direct light from the main lens to the plurality of tapered fiber optic bundles, and wherein the tapered fiber optic bundles are arranged along a first plane and a second plane;

with the main lens, directing the incoming light along the optical path;

directing the incoming light through the microlens array;

directing at least some of the incoming light through the tapered fiber optic bundles including the first and second tapered fiber optic bundles to the image sensors;

with the image sensors including the first and second image sensors, receiving the incoming light to generate image sensor data; and

combining the image sensor data generated by the image sensors including the first and second image sensors to define a single light-field image such that the single light-field image is substantially unaffected by the gaps.

2. The method of claim 1, further comprising:

directing some of the incoming light to a preview camera; and

;

with the preview camera, generating a preview image representative of the single light-field image.

3. The method of claim 2, further comprising, prior to directing the incoming light along the optical path, calibrating at least one of the image capture device and the preview camera to align an attribute of the preview camera with a corresponding attribute of the image capture device, wherein the attribute is selected from:

an ISO setting;

a focus setting;

a zoom setting; and

an image crop setting.

4. The method of claim 1, further comprising, prior to directing the incoming light along the optical path, calibrating the image capture device to mitigate differences in sensitivity between the image sensors.

5. The method of claim 1, further comprising, prior to directing the incoming light along the optical path, calibrating the image capture device to mitigate at least one of:

hex noise patterns existing within the tapered fiber optic bundles;

fiber noise patterns existing within the tapered fiber optic bundles; and

fault line defects existing within the tapered fiber optic bundles.

6. The method of claim 1, further comprising, prior to combining the image sensor data to define the single light-field image, transmitting the image sensor data to a plurality of servers;

wherein combining the image sensor data to define the single light-field image comprises combining the image sensor data at the plurality of servers.

7. The method of claim 1, further comprising, after combining the image sensor data to define the single light-field image, performing one or more light-field image processing steps on the single light-field image.

8. The method of claim 1, further comprising, prior to directing the incoming light along the optical path:

calibrating the image capture device to perform at least one of;

ISO calibration;

sensor dark current calibration;

dark fiber artifact removal;

sensor cropping;

grid warp calibration;

fault line artifact removal; and

after calibration of the image capture device, securing the microlens array relative to the image sensors.

9. The method of claim 1, further comprising, prior to directing the incoming light along the optical path, calibrating the image capture device to perform at least one of:

calibrating an aperture versus a focal length of the main lens; and

capturing a white image for demodulation.

10. The method of claim 1, further comprising, prior to directing the incoming light along the optical path, calibrating the image capture device to perform 2D geometry distortion correction by:

with the image capture device, capturing a calibration image of a checkerboard;

running a corner detection algorithm to detect corners of the checkerboard in the calibration image; and

running a grid fitting algorithm to define a reference grid fitted to the corners to generate vectors that can be used to generate an undistorted image from a distorted image.

11. The method of claim 1, further comprising, prior to directing the incoming light along the optical path, calibrating the image capture device to perform 2D geometry distortion correction by:

with the image capture device, capturing a plurality of calibration images of a checkerboard, with the checkerboard incrementally offset in each successive image;

generating a plurality of product images by performing a pixel-by-pixel multiplication of one vertical line image and one horizontal line on each of the plurality of calibration images;

for each of the product images, calculating a center-of-mass of each of a plurality of box-shaped areas to generate a grid; and

for each of the grids, applying a global optimization routine to identify a reference grid that fits the grid to generate vectors that can be used to generate an undistorted image from a distorted image.

12. The method of claim 1, further comprising, prior to directing the incoming light along the optical path, calibrating the image capture device by carrying out fault line artifact removal.

13. The method of claim 1, further comprising, prior to directing the incoming light along the optical path, calibrating the image capture device by performing tapered fiber optic bundle image alignment to generate geometry transform data;

wherein combining the image sensor data generated by the image sensors to define the single light-field image comprises using the geometry transform data to stitch images of the image sensor data together.

14. The method of claim 1, further comprising, prior to directing the incoming light along the optical path, calibrating the image capture device by generating a plurality of masks;

wherein combining the image sensor data generated by the image sensors to define the single light-field image comprises;

detecting seams between images of the image sensor data by finding center lines of each overlapping subregion between masks of adjacent image sensors; and

blending pixels proximate the seams.

15. The method of claim 1, further comprising, prior to directing the incoming light along the optical path, calibrating the image capture device by performing dark frame calibration for each of the image sensors.

16. The method of claim 1, further comprising, prior to directing the incoming light along the optical path, calibrating the image capture device by performing flat-field calibration for each of the image sensors.

17. The method of claim 1, further comprising, prior to combining the image sensor data generated by the image sensors to define the single light-field image, processing the image sensor data by applying at least one of white balancing, demosaicing, and chroma suppression.

18. The method of claim 1, wherein combining the image sensor data generated by the image sensors to define the single light-field image comprises carrying out fiber artifact removal on the image sensor data.

19. The method of claim 1, wherein combining the image sensor data generated by the image sensors to define the single light-field image comprises blending images of the sensor image data together.

20. The method of claim 1, wherein combining the image sensor data generated by the image sensors to define the single light-field image comprises conducting seams artifact removal on the image sensor data.

21. The method of claim 1, wherein combining the image sensor data generated by the image sensors to define the single light-field image comprises:

using the image sensor data to generate a single light-field; and

projecting the single light-field to generate the single light-field image.

22. The method of claim 1, wherein combining the image sensor data generated by the image sensors to define the single light-field image comprises processing depth and projection.

37. The method of claim 1, wherein the image capture device further comprises a face plate having a first face plate length and a second face plate length different from the first face plate length, and wherein a leading end of the first tapered fiber optic bundle is positioned proximate a portion of the face plate having the first face plate length and a leading end of the second tapered fiber optic bundle is positioned proximate a portion of the face plate having the second face plate length.

38. The method of claim 1, wherein the microlens array is divided into at least two strips and is secured relative to the plurality of tapered fiber optic bundles.

39. The method of claim 1, wherein the tapered fiber optic bundles are bonded together and polished to provide a polished fiber face plate surface.

23. A non-transitory computer-readable medium with instructions for capturing and processing image data through use of an image capture device configured to receive incoming light, the image capture device comprising:

a plurality of image sensors arranged in a pattern such that gaps exist between adjacent image sensors of the plurality of image sensors, wherein the plurality of image sensors includes a first image sensor and a second image sensor;

a main lens configured to direct incoming light along an optical path;

a microlens array positioned in the optical path to receive the incoming light;

a plurality of tapered fiber optic bundles configured to direct at least some of the incoming light to the plurality of image sensors, each tapered fiber optic bundle comprising a leading end positioned within the optical path, and a trailing end positioned proximate one of the plurality of image sensors, the leading end having a larger cross-sectional area than the trailing end, wherein the plurality of tapered fiber optic bundles includes;

a first tapered fiber optic bundle of a first length positioned proximate the first image sensor; and

a second tapered fiber optic bundle of a second length longer than the first length positioned proximate the second image sensor; and

a beam splitter to direct light from the main lens to the plurality of tapered fiber optic bundles, and wherein the tapered fiber optic bundles are arranged along a first plane and a second plane;

wherein the image sensors are configured to receive, through at least the first and second tapered fiber optic bundles, the incoming light to generate image sensor data; and

wherein the non-transitory computer-readable medium comprises instructions stored thereon, that when executed by a processor, combine the image sensor data generated by the image sensors including the first and second image sensors to define a single light-field image such that the single light-field image is substantially unaffected by the gaps.

24. The non-transitory computer-readable medium of claim 23, wherein the image capture device further comprises a preview camera into which some of the incoming light is directed;

wherein the preview camera generates a preview image representative of the single light-field image;

and wherein the non-transitory computer-readable medium further comprises instructions stored thereon, that when executed by a processor, prior to directing the incoming light along the optical path, calibrate at least one of the image capture device and the preview camera to align an attribute of the preview camera with a corresponding attribute of the image capture device, wherein the attribute is selected from;

an ISO setting;

a focus setting;

a zoom setting; and

an image crop setting.

25. The non-transitory computer-readable medium of claim 23, further comprising instructions stored thereon, that when executed by a processor, prior to directing the incoming light along the optical path, perform at least one of:

calibrating the image capture device to mitigate differences in sensitivity between the image sensors; and

calibrating the image capture device to mitigate at least one of;

hex noise patterns existing within the tapered fiber optic bundles;

fiber noise patterns existing within the tapered fiber optic bundles; and

fault line defects existing within the tapered fiber optic bundles.

26. The non-transitory computer-readable medium of claim 23, further comprising instructions stored thereon, that when executed by a processor, prior to directing the incoming light along the optical path and prior to securing the microlens array relative to the image sensors, calibrate the image capture device to perform at least one of:

ISO calibration;

sensor dark current calibration;

dark fiber artifact removal;

sensor cropping;

grid warp calibration; and

fault line artifact removal.

27. The non-transitory computer-readable medium of claim 23, further comprising instructions stored thereon, that when executed by a processor, prior to directing the incoming light along the optical path, calibrate the image capture device to perform at least one of:

calibrating an aperture versus a focal length of the main lens; and

capturing a white image for demodulation.

28. The non-transitory computer-readable medium of claim 23, further comprising instructions stored thereon, that when executed by a processor, prior to directing the incoming light along the optical path, calibrate the image capture device to perform 2D geometry distortion correction by:

with the image capture device, capturing a calibration image of a checkerboard;

running a corner detection algorithm to detect corners of the checkerboard in the calibration image; and

running a grid fitting algorithm to define a reference grid fitted to the corners to generate vectors that can be used to generate an undistorted image from a distorted image.

29. The non-transitory computer-readable medium of claim 23, further comprising instructions stored thereon, that when executed by a processor, prior to directing the incoming light along the optical path, calibrate the image capture device to perform 2D geometry distortion correction by:

with the image capture device, capturing a plurality of calibration images of a checkerboard, with the checkerboard incrementally offset in each successive image;

generating a plurality of product images by performing a pixel-by-pixel multiplication of one vertical line image and one horizontal line on each of the plurality of calibration images;

for each of the product images, calculating a center-of-mass of each of a plurality of box-shaped areas to generate a grid; and

for each of the grids, applying a global optimization routine to identify a reference grid that fits the grid to generate vectors that can be used to generate an undistorted image from a distorted image.

30. The non-transitory computer-readable medium of claim 23, further comprising instructions stored thereon, that when executed by a processor, prior to directing the incoming light along the optical path, calibrate the image capture device by carrying out fault line artifact removal.

31. The non-transitory computer-readable medium of claim 23, further comprising instructions stored thereon, that when executed by a processor, prior to directing the incoming light along the optical path, calibrate the image capture device by performing tapered fiber optic bundle image alignment to generate geometry transform data;

wherein combining the image sensor data generated by the image sensors to define the single light-field image comprises using the geometry transform data to stitch images of the image sensor data together.

32. The non-transitory computer-readable medium of claim 23, further comprising instructions stored thereon, that when executed by a processor, prior to directing the incoming light along the optical path, calibrate the image capture device by generating a plurality of masks;

wherein combining the image sensor data generated by the image sensors to define the single light-field image comprises;

detecting seams between images of the image sensor data by finding center lines of each overlapping subregion between masks of adjacent image sensors; and

blending pixels proximate the seams.

33. The non-transitory computer-readable medium of claim 23, further comprising instructions stored thereon, that when executed by a processor, prior to directing the incoming light along the optical path, calibrate the image capture device by performing at least one of:

dark frame calibration for each of the image sensors; and

flat-field calibration for each of the image sensors.

34. The non-transitory computer-readable medium of claim 23, further comprising instructions stored thereon, that when executed by a processor, prior to combining the image sensor data generated by the image sensors to define the single light-field image, process the image sensor data by applying at least one of white balancing, demosaicing, and chroma suppression.

35. The non-transitory computer-readable medium of claim 23, wherein combining the image sensor data generated by the image sensors to define the single light-field image comprises at least one of:

carrying out fiber artifact removal on the image sensor data;

blending images of the sensor image data together; and

conducting seams artifact removal on the image sensor data.

36. The non-transitory computer-readable medium of claim 23, wherein combining the image sensor data generated by the image sensors to define the single light-field image comprises:

using the image sensor data to generate a single light-field; and

projecting the single light-field to generate the single light-field image.

40. A device for capturing and processing image data, the device comprising:

a plurality of image sensors arranged in a pattern such that gaps exist between adjacent image sensors of the plurality of image sensors, wherein the plurality of image sensors includes a first image sensor and a second image sensor;

a main lens;

a microlens array;

a plurality of tapered fiber optic bundles, each of which includes a leading end positioned within an optical path, and a trailing end positioned proximate one of the plurality of image sensors, the leading end having a larger cross-sectional area than the trailing end, wherein the plurality of tapered fiber optic bundles includes;

a first tapered fiber optic bundle of a first length positioned proximate the first image sensor;

a second tapered fiber optic bundle of a second length longer than the first length positioned proximate the second image sensor; and

a beam splitter to direct light from the main lens to the plurality of tapered fiber optic bundles, and wherein the tapered fiber optic bundles are arranged along a first plane and a second plane; and

wherein;

the main lens directs incoming light along the optical path through the microlens array and through the tapered fiber optic bundles including the first and second tapered fiber optic bundles to the image sensors;

the first image sensor and the second image sensor generate image sensor data; and

the device combines the image sensor data generated by the first and second image sensors to define a single light-field image such that the single light-field image is substantially unaffected by the gaps between adjacent image sensors.

41. The device of claim 40, wherein combining the image sensor data generated by the first and second image sensors includes carrying out fiber artifact removal on the image sensor data.

42. The device of claim 40, wherein combining the image sensor data generated by the first and second image sensors includes blending images of the sensor image data together.

43. The device of claim 40, wherein combining the image sensor data generated by the first and second image sensors includes conducting seams artifact removal on the image sensor data.