Systems and methods for performing high speed video capture and depth estimation using array cameras

• US 9,942,474 B2
• Filed: 04/17/2015
• Issued: 04/10/2018
• Est. Priority Date: 04/17/2015
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: Fotonation Cayman Limited
• Inventors: Venkataraman, Kartik, Huang, Yusong, Jain, Ankit K., Chatterjee, Priyam
• Primary Examiner(s): FOSSELMAN, JOEL W
• Application Number:
  US14/690,346
• Publication Number:
  US 20160309084A1
• Time in Patent Office: 1,089 Days
• Field of Search: 3482181, 345427, 382154
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: H04N 13/00 Stereoscopic video systems; Multi-view video systems; Details thereof
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 25/531 by controlling rolling shutters in CMOS SSIS
• CPC Class Codes: H04N 5/2226 Determination of depth image, e.g. for foreground/background separation determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10

Abstract

High speed video capture and depth estimation using array cameras is disclosed. Real world scenes typically include objects located at different distances from a camera. Therefore, estimating depth during video capture by an array camera can result in smoother rendering of video from image data captured of real world scenes. One embodiment of the invention includes cameras that capture images from different viewpoints, and an image processing pipeline application that obtains images from groups of cameras, where each group of cameras starts capturing image data at a staggered start time relative to the other groups of cameras. The application then selects a reference viewpoint and determines scene-dependent geometric corrections that shift pixels captured from an alternate viewpoint to the reference viewpoint by performing disparity searches to identify the disparity at which pixels from the different viewpoints are most similar. The corrections can then be used to render frames of video.

Claims

1. An array camera, comprising:

a plurality of cameras that capture images of a scene from different viewpoints;

memory containing an image processing pipeline application;

wherein the image processing pipeline application directs the processor to;

obtain image data from a plurality of groups of cameras from within the plurality of cameras, where each group of cameras starts capturing image data at a staggered start time relative to the other groups of cameras;

select a reference viewpoint and determine scene-dependent geometric corrections that shift pixels captured from an alternate viewpoint to the reference viewpoint by performing disparity searches to identify the disparity at which pixels from the different viewpoints are most similar; and

render frames of video, where a given frame of video is rendered using pixels comprising pixels from at least one group of cameras captured during a given frame capture time interval and by shifting pixels captured from alternate viewpoints to the reference viewpoint using scene-dependent geometric corrections determined for the pixels captured from the alternate viewpoints,

wherein the image processing pipeline application further directs the processor to render frames of video using;

pixels captured by at least one group of cameras during the given frame capture time interval and determined to be moving during the given frame capture time interval; and

pixels from a previously rendered frame that are determined to be non-moving during at least the given frame capture time interval.

2. The array camera of claim 1, wherein each group of cameras is a single camera selected from the group consisting of:

a monochrome camera, a Bayer camera, and an infrared camera.

3. The array camera of claim 1, wherein each group of cameras includes multiple cameras.

4. The array camera of claim 1, wherein the image processing pipeline application further directs the processor to determine scene-dependent geometric corrections that shift pixels captured from an alternate viewpoint to the reference viewpoint by:

selecting at least a portion of a frame captured from a first viewpoint during a specific frame capture time interval;

interpolating at least a portion of a frame from a second viewpoint during the specific frame capture time interval based upon image data captured from the second viewpoint at other times, where the second viewpoint differs from the first viewpoint; and

identifying the disparity at which pixels from the at least a portion of a frame selected from the first viewpoint and the at least a portion of a frame interpolated from the second viewpoint are most similar.

5. The array camera of claim 4, wherein the first viewpoint is the alternate viewpoint and the second viewpoint is the reference viewpoint.

6. The array camera of claim 4, wherein the groups of cameras each comprise multiple cameras.

7. The array camera of claim 6, wherein:

the plurality of cameras comprises a camera that captures image data from the reference viewpoint;

the first viewpoint is the alternate viewpoint; and

the second viewpoint is the reference viewpoint.

8. The array camera of claim 6, wherein:

the plurality of cameras comprises a camera that captures image data from the reference viewpoint in a first color channel;

the alternate viewpoint is the viewpoint of a camera in a specific group of cameras that captures image data in a second color channel;

the first viewpoint is the viewpoint of a camera in the specific group of cameras that captures image data in the first color channel;

the second viewpoint is the reference viewpoint; and

wherein the image processing pipeline application further directs the processor to determine scene-dependent geometric corrections that shift pixels captured from the alternate viewpoint to the reference viewpoint by;

estimating depth of at least one pixel visible in the alternate viewpoint based upon the disparity at which pixels from the at least a portion of a frame selected from the first viewpoint and the at least a portion of a frame interpolated from the second viewpoint are most similar; and

determining scene-dependent geometric corrections to apply to shift the at least one pixel captured from the alternate viewpoint to the reference viewpoint based upon the estimated depth of the at least one pixel.

9. The array camera of claim 6, wherein:

the plurality of cameras comprises a reference camera that captures image data from the reference viewpoint in a first color channel, where the reference camera is part of a first group of cameras;

the alternate viewpoint is the viewpoint of a camera from the first group of cameras that captures image data in a second color channel;

the first viewpoint is the reference viewpoint;

the second viewpoint is the viewpoint of a camera in a second group of cameras that captures image data in the first color channel; and

wherein the image processing pipeline application further directs the processor to determine scene-dependent geometric corrections that shift pixels captured from the alternate viewpoint to the reference viewpoint by;

estimating depth of at least one pixel visible in the alternate viewpoint based upon the disparity at which pixels from the at least a portion of a frame selected from the first viewpoint and the at least a portion of a frame interpolated from the second viewpoint are most similar; and

determining scene-dependent geometric corrections to apply to shift the at least one pixel captured from the alternate viewpoint to the reference viewpoint based upon the estimated depth of the at least one pixel.

10. The array camera of claim 1, wherein the image processing pipeline application further directs the processor to determine scene-dependent geometric corrections that shift pixels captured from an alternate viewpoint to the reference viewpoint by:

selecting moving pixels from image data captured from a first viewpoint during a specific frame capture time interval, where the moving pixels are pixels determined to be moving during at least the specific frame capture time interval;

interpolating moving pixels from a second viewpoint during the specific frame capture time interval based upon image data captured from the second viewpoint at other times, where the second viewpoint differs from the first viewpoint and the moving pixels are pixels determined to be moving during the specific frame capture time interval; and

identifying the disparity at which the moving pixels from image data captured from the first viewpoint and the moving pixels interpolated from the second viewpoint are most similar.

11. The array camera of claim 10, wherein the image processing pipeline application further directs the processor to perform a disparity search to identify the disparity at which the moving pixels from image data captured from the first viewpoint and the moving pixels interpolated from the second viewpoint are most similar by comparing the similarity of individual moving pixels at different disparities.

12. The array camera of claim 10, wherein the image processing pipeline application further directs the processor to perform a disparity search to identify the disparity at which the moving pixels from image data captured from the first viewpoint and the moving pixels interpolated from the second viewpoint are most similar by comparing the similarity of individual clusters of moving pixels at different disparities.

13. The array camera of claim 10, wherein the image processing pipeline application further directs the processor to perform a disparity search to identify the disparity at which the moving pixels from image data captured from the first viewpoint and the moving pixels interpolated from the second viewpoint are most similar by comparing the similarity of all moving pixels at different disparities.

14. The array camera of claim 1, wherein the image processing pipeline application further directs the processor to render an initial frame from an image captured using one of the plurality of cameras.

15. The array camera of claim 1, wherein the image processing pipeline application further directs the processor to render an initial frame by:

capturing a set of images using an initial group of cameras;

determining depth estimates for pixel locations in an image that is from the reference viewpoint using at least a subset of the set of images, wherein generating a depth estimate for a given pixel location in the image from the reference viewpoint comprises;

identifying pixels in the at least a subset of the set of images that correspond to the given pixel location in the image from the reference viewpoint based upon expected disparity at a plurality of depths;

comparing the similarity of the corresponding pixels identified at each of the plurality of depths; and

selecting the depth from the plurality of depths at which the identified corresponding pixels have the highest degree of similarity as a depth estimate for the given pixel location in the image from the reference viewpoint; and

rendering a frame from the reference viewpoint using the set of images and the depth estimates for pixel locations in a subset of the set of images to shift pixels captured from alternate viewpoints to the reference viewpoint.

16. The array camera of claim 15, wherein the initial group of cameras comprises a 3×

3 π

filter group incorporating Red, Green and Blue cameras.

17. The array camera of claim 15, wherein the initial groups of cameras comprises all of the cameras in the plurality of groups of cameras.

18. The array camera of claim 17, wherein:

the plurality of cameras have electronic rolling shutters;

the set of images are each captured over a rolling shutter time interval;

the given frame capture time interval is shorter than a rolling shutter time interval, where the rolling shutter time interval is the time taken to complete read out of image data from a camera in the plurality of cameras; and

different portions of the images in the set of images captured by the plurality of groups of cameras that are captured during successive frame capture time intervals are used to render a sequence of frames.

19. The array camera of claim 1, wherein:

the plurality of cameras have electronic rolling shutters; and

the given frame capture time interval is shorter than a rolling shutter time interval, where the rolling shutter time interval is the time taken to complete read out of image data from a camera in the plurality of cameras.

20. The array camera of claim 19, wherein the staggered start times of the cameras are coordinated so that each of N groups of cameras captures at least a 1/N portion of an image during a given frame capture time interval.

21. The array camera of claim 20, wherein a given frame of video is rendered using pixels comprising pixels from the N groups of cameras captured during the given frame capture time interval.

22. The array camera of claim 21, wherein the image processing pipeline application further directs the processor to determine scene-dependent geometric corrections that shift pixels captured from an alternate viewpoint to the reference viewpoint by:

selecting at least a 1/N portion of an image captured from a first viewpoint during a specific frame capture time interval;

interpolating at least a portion of an image from a second viewpoint during the specific frame capture time interval based upon image data captured from the second viewpoint at other times, where the second viewpoint differs from the first viewpoint;

identifying the disparity at which pixels from the selected at least a 1/N portion of an image captured from the first viewpoint and the at least a portion of an image interpolated from the second viewpoint are most similar.

23. The array camera of claim 1, wherein the plurality of cameras have electronic snap-shot shutters.

24. The array camera of claim 23, wherein the image processing pipeline application further directs the processor to determine scene-dependent geometric corrections that shift pixels captured from an alternate viewpoint to the reference viewpoint by:

selecting an image captured from a first viewpoint during a specific frame capture time interval;

interpolating at least a portion of an image from a second viewpoint during the specific frame capture time interval based upon image data captured from the second viewpoint at other times, where the second viewpoint differs from the first viewpoint; and

identifying the disparity at which pixels from the image captured from the first viewpoint and the at least a portion of an image interpolated from the second viewpoint are most similar.

25. The array camera of claim 1, wherein the image processing pipeline application further directs the processor to determine scene-dependent geometric corrections that shift pixels captured from an alternate viewpoint to the reference viewpoint by:

capturing a set of images using a group of cameras comprising at least two cameras that capture image data in the same at least one color channel;

determining depth estimates for pixel locations in an image from the reference viewpoint using at least a subset of the set of images, wherein generating a depth estimate for a given pixel location in the image from the reference viewpoint comprises;

identifying pixels in the at least a subset of the set of images that correspond to the given pixel location in the image from the reference viewpoint based upon expected disparity at a plurality of depths;

comparing the similarity of the corresponding pixels identified at each of the plurality of depths; and

selecting the depth from the plurality of depths at which the identified corresponding pixels have the highest degree of similarity as a

depth estimate for the given pixel location in the image from the reference viewpoint.

26. The array camera of claim 1, wherein the image processing pipeline application further directs the processor to determine scene-dependent geometric corrections that shift pixels captured from an alternate viewpoint to the reference viewpoint by estimating depths for individual pixels determined to be moving during the given frame capture time interval.

27. The array camera of claim 1, wherein the image processing pipeline application further directs the processor to determine scene-dependent geometric corrections that shift pixels captured from an alternate viewpoint to the reference viewpoint by estimating depths for clusters of pixels determined to be moving during the given frame capture time interval.

28. The array camera of claim 1, wherein the image processing pipeline application further directs the processor to determine scene-dependent geometric corrections that shift pixels captured from an alternate viewpoint to the reference viewpoint by estimating depths for all pixels determined to be moving during the given frame capture time interval.

29. An array camera, comprising:

a plurality of cameras that capture images of a scene from different viewpoints, where the plurality of cameras have electronic rolling shutters and capture an image during a rolling shutter time interval;

memory containing an image processing pipeline application;

wherein the image processing pipeline application directs the processor to;

select a reference viewpoint;

render an initial frame by;

capturing a set of images using an initial group of cameras;

determining depth estimates for pixel locations in an image from the set of images that is from the reference viewpoint using at least a subset of the set of images, wherein generating a depth estimate for a given pixel location in the image from the reference viewpoint comprises;

identifying pixels in the at least a subset of the set of images that correspond to the given pixel location in the image from the reference viewpoint based upon expected disparity at a plurality of depths;

comparing the similarity of the corresponding pixels identified at each of the plurality of depths; and

selecting the depth from the plurality of depths at which the identified corresponding pixels have the highest degree of similarity as a depth estimate for the given pixel location in the image from the reference viewpoint;

rendering the initial frame from the reference viewpoint using the set of images and the depth estimates for pixel locations in a subset of the set of images to shift pixels captured from alternate viewpoints to the reference viewpoint;

render subsequent frames by;

obtaining image data from a plurality of groups of cameras from within the plurality of cameras, where each group of cameras starts capturing image data at a staggered start time relative to the other groups of cameras and the staggered start times of the cameras are coordinated so that each of N groups of cameras captures at least a 1/N portion of a frame during a given frame capture time interval that is shorter than the rolling shutter time intervals of each of the plurality of cameras;

determining pixels captured by the N groups of cameras during a given frame capture time interval that are moving during the given frame capture time interval; and

determining scene-dependent geometric corrections that shift moving pixels captured from an alternate viewpoint to the reference viewpoint by performing disparity searches to identify the disparity at which moving pixels from the different viewpoints are most similar, where the disparity searches comprise;

selecting moving pixels from image data captured from a first viewpoint during the given frame capture time interval;

interpolating moving pixels from a second viewpoint during the given frame capture time interval based upon image data captured from the second viewpoint at other times, where the second viewpoint differs from the first viewpoint; and

identifying the disparity at which the moving pixels from image data captured from the first viewpoint and the moving pixels interpolated from the second viewpoint are most similar;

rendering frames of video, where a given frame of video is rendered using pixels comprising;

moving pixels from the N groups of cameras captured during the given frame capture time interval, where moving pixels captured from alternate viewpoints are shifted to reference viewpoint using scene-dependent geometric corrections determined for the pixels captured from the alternate viewpoints; and

non-moving pixels from a previously rendered frame from the reference viewpoint.

30. An array camera, comprising:

a plurality of cameras that capture images of a scene from different viewpoints;

memory containing an image processing pipeline application;

wherein the image processing pipeline application directs the processor to;

obtain image data from a plurality of groups of cameras from within the plurality of cameras, where each group of cameras starts capturing image data at a staggered start time relative to the other groups of cameras;

select a reference viewpoint and determine scene-dependent geometric corrections that shift pixels captured from an alternate viewpoint to the reference viewpoint by performing disparity searches to identify the disparity at which pixels from the different viewpoints are most similar; and

render frames of video, where a given frame of video is rendered using pixels comprising pixels from at least one group of cameras captured during a given frame capture time interval and by shifting pixels captured from alternate viewpoints to the reference viewpoint using scene-dependent geometric corrections determined for the pixels captured from the alternate viewpoints;

wherein the image processing pipeline application further directs the processor to render frames of video using;

pixels captured by at least one group of cameras during the given frame capture time interval and determined to be moving during the given frame capture time interval; and

pixels from a previously rendered frame that are determined to be non-moving during at least the given frame capture time interval;

wherein;

the plurality of cameras have electronic rolling shutters; and

the given frame capture time interval is shorter than a rolling shutter time interval, where the rolling shutter time interval is the time taken to complete read out of image data from a camera in the plurality of cameras.

31. The array camera of claim 30, wherein each group of cameras is a single camera selected from the group consisting of:

a monochrome camera, a Bayer camera, and an infrared camera.

32. The array camera of claim 30, wherein each group of cameras includes multiple cameras.

33. The array camera of claim 30, wherein the image processing pipeline application further directs the processor to determine scene-dependent geometric corrections that shift pixels captured from an alternate viewpoint to the reference viewpoint by:

selecting moving pixels from image data captured from a first viewpoint during a specific frame capture time interval, where the moving pixels are pixels determined to be moving during at least the specific frame capture time interval;

interpolating moving pixels from a second viewpoint during the specific frame capture time interval based upon image data captured from the second viewpoint at other times, where the second viewpoint differs from the first viewpoint and the moving pixels are pixels determined to be moving during the specific frame capture time interval; and

identifying the disparity at which the moving pixels from image data captured from the first viewpoint and the moving pixels interpolated from the second viewpoint are most similar.

34. The array camera of claim 33, wherein the image processing pipeline application further directs the processor to perform a disparity search to identify the disparity at which the moving pixels from image data captured from the first viewpoint and the moving pixels interpolated from the second viewpoint are most similar by comparing the similarity of individual moving pixels at different disparities.

35. The array camera of claim 33, wherein the image processing pipeline application further directs the processor to perform a disparity search to identify the disparity at which the moving pixels from image data captured from the first viewpoint and the moving pixels interpolated from the second viewpoint are most similar by comparing the similarity of individual clusters of moving pixels at different disparities.

36. The array camera of claim 33, wherein the image processing pipeline application further directs the processor to perform a disparity search to identify the disparity at which the moving pixels from image data captured from the first viewpoint and the moving pixels interpolated from the second viewpoint are most similar by comparing the similarity of all moving pixels at different disparities.

37. The array camera of claim 30, wherein the image processing pipeline application further directs the processor to render an initial frame from an image captured using one of the plurality of cameras.

38. An array camera, comprising:

a plurality of cameras that capture images of a scene from different viewpoints, wherein the plurality of cameras have electronic snap-shot shutters;

memory containing an image processing pipeline application;

wherein the image processing pipeline application directs the processor to;

obtain image data from a plurality of groups of cameras from within the plurality of cameras, where each group of cameras starts capturing image data at a staggered start time relative to the other groups of cameras;

select a reference viewpoint and determine scene-dependent geometric corrections that shift pixels captured from an alternate viewpoint to the reference viewpoint by performing disparity searches to identify the disparity at which pixels from the different viewpoints are most similar; and

render frames of video, where a given frame of video is rendered using pixels comprising pixels from at least one group of cameras captured during a given frame capture time interval and by shifting pixels captured from alternate viewpoints to the reference viewpoint using scene-dependent geometric corrections determined for the pixels captured from the alternate viewpoints;

wherein the image processing pipeline application further directs the processor to determine scene-dependent geometric corrections that shift pixels captured from an alternate viewpoint to the reference viewpoint by;

selecting an image captured from a first viewpoint during a specific frame capture time interval;

interpolating at least a portion of an image from a second viewpoint during the specific frame capture time interval based upon image data captured from the second viewpoint at other times, where the second viewpoint differs from the first viewpoint; and

identifying the disparity at which pixels from the image captured from the first viewpoint and the at least a portion of an image interpolated from the second viewpoint are most similar.

39. The array camera of claim 38, wherein each group of cameras is a single camera selected from the group consisting of:

a monochrome camera, a Bayer camera, and an infrared camera.

40. The array camera of claim 38, wherein each group of cameras includes multiple cameras.

41. The array camera of claim 38, wherein the image processing pipeline application further directs the processor to render frames of video using:

pixels captured by at least one group of cameras during the given frame capture time interval and determined to be moving during the given frame capture time interval; and

pixels from a previously rendered frame that are determined to be non-moving during at least the given frame capture time interval.

42. The array camera of claim 41, wherein the image processing pipeline application further directs the processor to determine scene-dependent geometric corrections that shift pixels captured from an alternate viewpoint to the reference viewpoint by:

selecting moving pixels from image data captured from a first viewpoint during a specific frame capture time interval, where the moving pixels are pixels determined to be moving during at least the specific frame capture time interval;

interpolating moving pixels from a second viewpoint during the specific frame capture time interval based upon image data captured from the second viewpoint at other times, where the second viewpoint differs from the first viewpoint and the moving pixels are pixels determined to be moving during the specific frame capture time interval; and

identifying the disparity at which the moving pixels from image data captured from the first viewpoint and the moving pixels interpolated from the second viewpoint are most similar.

43. The array camera of claim 42, wherein the image processing pipeline application further directs the processor to perform a disparity search to identify the disparity at which the moving pixels from image data captured from the first viewpoint and the moving pixels interpolated from the second viewpoint are most similar by comparing the similarity of individual moving pixels at different disparities.

44. The array camera of claim 42, wherein the image processing pipeline application further directs the processor to perform a disparity search to identify the disparity at which the moving pixels from image data captured from the first viewpoint and the moving pixels interpolated from the second viewpoint are most similar by comparing the similarity of individual clusters of moving pixels at different disparities.

45. The array camera of claim 42, wherein the image processing pipeline application further directs the processor to perform a disparity search to identify the disparity at which the moving pixels from image data captured from the first viewpoint and the moving pixels interpolated from the second viewpoint are most similar by comparing the similarity of all moving pixels at different disparities.

46. The array camera of claim 41, wherein the image processing pipeline application further directs the processor to render an initial frame from an image captured using one of the plurality of cameras.