Methods and systems for coded rolling shutter

• US 9,100,514 B2
• Filed: 10/28/2010
• Issued: 08/04/2015
• Est. Priority Date: 10/28/2009
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Inventors: Gu, Jinwei, Hitomi, Yasunobu, Mitsunaga, Tomoo, Nayar, Shree K.
• Primary Examiner(s): Aggarwal, Yogesh
• Application Number:
  US13/504,905
• Publication Number:
  US 20130070121A1
• Time in Patent Office: 1,741 Days
• Field of Search: 348/294, 348/298, 348/302, 348/308, 348/312, 348/362
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: H04N 23/689 Motion occurring during a rolling shutter mode
• CPC Class Codes: H04N 23/73 by influencing the exposure time
• CPC Class Codes: H04N 25/531 by controlling rolling shutters in CMOS SSIS
• CPC Class Codes: H04N 25/533 by using differing integration times for different sensor regions
• CPC Class Codes: H04N 25/583 with different integration times
• CPC Class Codes: H04N 25/75 Circuitry for providing, modifying or processing image signals from the pixel array
• CPC Class Codes: H04N 25/76 Addressed sensors, e.g. MOS or CMOS sensors
• CPC Class Codes: H04N 7/00 Television systems details H04N3/00, H04N5/00; methods or arrangements, for coding, decoding, compressing or decompressing digital video signals H04N19/00; selective content distribution H04N21/00
• CPC Class Codes: H04N 7/002 Special television systems not provided for by H04N7/007 - H04N7/18 still pictures via a television channel H04N1/00098

Abstract

Methods and systems for coded rolling shutter are provided. In accordance with some embodiments, methods and system are provided that control the readout timing and exposure length for each row of a pixel array in an image sensor, thereby flexibly sampling the three-dimensional space-time value of a scene and capturing sub-images that effectively encode motion and dynamic range information within a single captured image.

Claims

1. A method for reading an image of a scene detected in an image sensor comprising a pixel array having a plurality of rows of pixels, the method comprising:

exposing each of the plurality of rows of the pixel array to the image of the scene;

reading-out a first subset of the rows of the pixel array to extract a first sub-image from the image;

reading-out a second subset of the rows of the pixel array to extract a second sub-image from the image, wherein the first subset of the rows of the pixel array is different from the second subset of the rows of the pixel array;

estimating optical flow between the first sub-image and the second sub-image;

determining an intermediate image that is interpolated between the first sub-image and the second sub-image based at least in part on the estimated optical flow; and

combining the first sub-image, the intermediate image, and the second sub-image to create a video.

2. The method of claim 1, wherein a plurality of sub-images that includes the first sub-image and the second sub-image extracted from the image are uniformly distributed between the plurality of rows of the pixel array.

3. The method of claim 1, further comprising estimating the optical flow between the first sub-image and the second sub-image for at least one of recovering slow motion, substantially removing skew from the image, and substantially removing motion blur from the image.

4. The method of claim 3, wherein skew is substantially removed in the intermediate image.

5. The method of claim 1, wherein the first sub-image, the intermediate image, and the second sub-image are combined to create a slow motion interpolated video.

6. The method of claim 3, wherein the estimated optical flow corresponds to motion information associated with the image sensor, the method further comprising:

estimating a point spread function for the image based at least in part on the motion information; and

applying the estimated point spread function to enhance an output image generated from at least one of the first sub-image and the second sub-image, thereby substantially removing motion blur the output image.

7. A method for reading an image of a scene detected in an image sensor comprising a pixel array having a plurality of rows of pixels, wherein the plurality of rows includes a given row, a higher row that is higher in the pixel array than the given row, and a lower row that is lower in the pixel array than the given row, the method comprising:

receiving a coded pattern for controlling readout times in the pixel array and for extracting a plurality of sub-images from the image;

exposing each of the plurality of rows of the pixel array to the image of the scene;

reading-out the given row of the plurality of rows, wherein the given row is selected for readout based on the number of sub-images;

reading-out a first set of higher rows subsequent to reading-out the given row;

reading-out the lower row subsequent to reading-out the first set of higher rows, wherein the lower row is selected for readout based on the number of sub-images; and

reading-out a second set of higher rows subsequent to reading-out the lower row.

8. The method of claim 7, wherein the coded pattern assigns each of the plurality of rows to one of the plurality of sub-images and wherein the plurality of sub-images are uniformly distributed between the plurality of rows of the pixel array.

9. The method of claim 7, further comprising interpolating pixel values in each of the plurality of sub-images to obtain an interpolated image having full resolution.

10. The method of claim 7, further comprising combining the plurality of extracted sub-images to create a slow motion interpolated video.

11. A method for reading an image of a scene detected in an image sensor comprising a pixel array having a plurality of rows of pixels, the method comprising:

obtaining a first image of the scene;

determining an optimal exposure time for each of the plurality of rows of the pixel array based at least in part on scene radiance in the first image of the scene, wherein a first exposure time for a first row of the plurality of rows and a second exposure time for a second row of the plurality of rows are controlled and wherein the first exposure time is controlled to be different from the second exposure time;

obtaining a second image of the scene, wherein the first exposure time for the first row of the plurality of rows of the pixel array and the second exposure time for the second row of the plurality of rows of the pixel array are adjusted based at least in part on the determined optimal exposure time for that row; and

reading-out the first row and the second row to generate an output image.

12. The method of claim 11, further comprising determining an optimal exposure time for at least one of the first exposure time for the first row and the second exposure time for the second row.

13. The method of claim 11, further comprising normalizing pixel values of the second image of the scene with respect to the determined optimal exposure time applied for each of the plurality of rows of the pixel array.

14. The method of claim 11, further comprising controlling readout times of the plurality of rows of the pixel array.

15. The method of claim 14, further comprising extracting a plurality of sub-images from the image using the controlled readout times and the controlled exposure times, wherein the plurality of sub-images are uniformly distributed between the plurality of rows in the pixel array.

16. The method of claim 15, further comprising combining the plurality of sub-images extracted from the image to compose a high dynamic range image.

17. The method of claim 15, further comprising using two or more of the plurality of sub-images to compensate for motion blur.

18. The method of claim 15, further comprising:

estimating optical flow between a first sub-image and a second sub-image of the plurality of sub-images;

determining motion information based at least in part on the estimated optical flow; and

applying the determined motion information to enhance a high dynamic range image that is composed by combining the plurality of sub-images extracted from the image, thereby substantially removing motion blur from the high dynamic range image.

19. The method of claim 18, further comprising:

incrementally applying the determined motion information to each of the plurality of sub-images to substantially remove motion blur from each of the plurality of sub-images; and

combining the plurality of sub-images to compose a high dynamic range image.

20. A method for reading an image of a scene detected in an image sensor comprising a pixel array having a plurality of rows of pixels, the method comprising:

receiving a coded pattern that controls a plurality of exposure times and a plurality of readout times and wherein each of the plurality of exposure times and each of the plurality of readout times are associated with one of the plurality of rows of the pixel array;

exposing each of the plurality of rows of the pixel array to the image of the scene in accordance with the received coded pattern;

reading-out the plurality of rows of the pixel array in accordance with the received coded pattern to obtain a pixel value for each pixel;

reconstructing estimated pixel values for each pixel over time based on pixel values from neighboring rows in the pixel array; and

constructing a video using the reconstructed estimated pixel values, wherein skew is substantially reduced in the constructed video.

21. The method of claim 20, further comprising using the coded pattern to randomly assign the plurality of exposure times and the plurality of readout times for the plurality of rows of the pixel array.

22. The method of claim 20, further comprising:

normalizing pixel values of the image of the scene with respect to the plurality of exposure times, wherein each of the normalized pixel values corresponds to a set of sampled voxels;

searching through a plurality of voxels to find a matching set of sampled voxels;

interpolating additional voxels between the set of sampled voxels and the matching set of sampled voxels;

determining an estimated volume using at least the set of sampled voxels, the matching set of sampled voxels, and the interpolated additional voxels; and

creating the video based at least in part on the estimated volume by reconstructing each pixel over time.

23. The method of claim 22, further comprising:

receiving a preset number of the neighboring rows; and

adjusting the preset number of the neighboring rows based at least in part on the estimated volume.

24. The method of claim 22, further comprising repeating the searching, interpolating, and determining to create another estimated volume.

25. A system for reading an image of a scene, the system comprising:

an image sensor comprising a pixel array having a plurality of rows; and

at least one controller that;

exposes each of the plurality of rows of the pixel array to the image of the scene;

reads-out a first subset of the rows of the pixel array to extract a first sub-image from the image;

reads-out a second subset of the rows of the pixel array to extract a second sub-image from the image, wherein the first subset of the rows of the pixel array is different from the second subset of the rows of the pixel array;

estimates optical flow between the first sub-image and the second sub-image;

determines an intermediate image that is interpolated between the first sub-image and the second sub-image based at least in part on the estimated optical flow; and

combines the first sub-image, the intermediate image, and the second sub-image to create a video.

26. The system of claim 25, wherein a plurality of sub-images that includes the first sub-image and the second sub-image extracted from the image are uniformly distributed between the plurality of rows of the pixel array.

27. The system of claim 25, wherein the at least one controller is further configured to estimate the optical flow between the first sub-image and the second sub-image for at least one of recovering slow motion, substantially removing skew from the image, and substantially removing motion blur from the image.

28. The system of claim 27, wherein skew is substantially removed in the intermediate image.

29. The system of claim 25, wherein the at least one controller is further configured to combine the first sub-image, the intermediate image, and the second sub-image to create a slow motion interpolated video.

30. The system of claim 27, wherein the estimated optical flow corresponds to motion information associated with the image sensor and wherein the at least one controller is further configured to:

estimate a point spread function for the image based at least in part on the motion information; and

apply the estimated point spread function to enhance an output image generated from at least one of the first sub-image and the second sub-image, thereby substantially removing motion blur the output image.

31. A system for reading an image of a scene, the system comprising:

an image sensor comprising a pixel array having a plurality of rows of pixels, wherein the plurality of rows includes a given row, a higher row that is higher in the pixel array than the given row, and a lower row that is lower in the pixel array than the given row; and

at least one controller that;

receives a coded pattern for controlling readout times in the pixel array and for extracting a plurality of sub-images from the image;

exposes each of the plurality of rows of the pixel array to the image of the scene;

reads-out the given row of the plurality of rows, wherein the given row is selected for readout based on the number of sub-images;

reads-out a first set of higher rows subsequent to reading-out the given row;

reads-out the lower row subsequent to reading-out the first set of higher rows, wherein the lower row is selected for readout based on the number of sub-images; and

reads-out a second set of higher rows subsequent to reading-out the lower row.

32. The system of claim 31, wherein the coded pattern assigns each of the plurality of row to one of the plurality of sub-images and wherein the plurality of sub-images are uniformly distributed between the plurality of rows of the pixel array.

33. The system of claim 31, wherein the at least one controller is further configured to interpolate pixel values in each of the plurality of sub-images to obtain an interpolated image having full resolution.

34. The system of claim 31, wherein the at least one controller is further configured to combine the plurality of extracted sub-images to create a slow motion interpolated video.

35. A system for reading an image of a scene, the system comprising:

an image sensor comprising a pixel array having a plurality of rows; and

at least one controller that;

obtains a first image of the scene;

determines an optimal exposure time for each of the plurality of rows of the pixel array based at least in part on scene radiance, wherein a first exposure time for a first row of the plurality of rows and a second exposure time for a second row of the plurality of rows are controlled and wherein the first exposure time is controlled to be different from the second exposure time;

obtaining a second image of the scene, wherein the first exposure time for the first row of the plurality of rows of the pixel array and the second exposure time for the second row of the plurality of rows of the pixel array are adjusted based at least in part on the determined optimal exposure time for that row; and

reads-out the first row and the second row to generate an output image.

36. The system of claim 35, wherein the at least one controller is further configured to determine an optimal exposure time for at least one of the first exposure time for the first row and the second exposure time for the second row.

37. The system of claim 35, wherein the at least one controller is further configured to normalize pixel values of the second image of the scene with respect to the determined optimal exposure time applied for each of the plurality of rows of the pixel array.

38. The system of claim 35, wherein the at least one controller is further configured to control readout times of the plurality of rows of the pixel array.

39. The system of claim 38, wherein the at least one controller is further configured to extract a plurality of sub-images from the image using the controlled readout times and the controlled exposure times, wherein the plurality of sub-images are uniformly distributed between the plurality of rows in the pixel array.

40. The system of claim 39, wherein the at least one controller is further configured to combine the plurality of sub-images extracted from the image to compose a high dynamic range image.

41. The system of claim 39, wherein the at least one controller is further configured to use two or more of the plurality of sub-images to compensate for motion blur.

42. The system of claim 39, wherein the at least one controller is further configured to:

estimate optical flow between a first sub-image and a second sub-image of the plurality of sub-images;

determine motion information based at least in part on the estimated optical flow; and

apply the determined motion information to enhance a high dynamic range image that is composed by combining the plurality of sub-images extracted from the image, thereby substantially removing motion blur from the high dynamic range image.

43. The system of claim 42, wherein the at least one controller is further configured to:

incrementally apply the determined motion information to each of the plurality of sub-images to substantially remove motion blur from each of the plurality of sub-images; and

combine the plurality of sub-images to compose a high dynamic range image.

44. A system for reading an image of a scene, the system comprising:

an image sensor comprising a pixel array having a plurality of rows; and

at least one controller that;

receives a coded pattern that controls a plurality of exposure times and a plurality of readout times and wherein each of the plurality of exposure times and each of the plurality of readout times are associated with one of the plurality of rows of the pixel array;

exposes each of the plurality of rows of the pixel array to the image of the scene in accordance with the received coded pattern;

reads-out the plurality of rows of the pixel array in accordance with the received coded pattern to obtain a pixel value for each pixel;

reconstructs estimated pixel values for each pixel over time based on pixel values from neighboring rows in the pixel array; and

constructs a video using the reconstructed estimated pixel values, wherein skew is substantially reduced in the constructed video.

45. The system of claim 44, wherein the at least one controller is further configured to use the coded pattern to randomly assign the plurality of exposure times and the plurality of readout times for the plurality of rows of the pixel array.

46. The system of claim 44, wherein the at least one controller is further configured to:

normalize pixel values of the image of the scene with respect to the plurality of exposure times, wherein each of the normalized pixel values corresponds to a set of sampled voxels;

search through a plurality of voxels to find a matching set of sampled voxels;

interpolate additional voxels between the set of sampled voxels and the matching set of sampled voxels;

determine an estimated volume using at least the set of sampled voxels, the matching set of sampled voxels, and the interpolated additional voxels; and

create the video based at least in part on the estimated volume by reconstructing each pixel over time.

47. The system of claim 46, wherein the at least one controller is further configured to:

receive a preset number of the neighboring rows; and

adjust the preset number of the neighboring rows based at least in part on the estimated volume.

48. The system of claim 46, wherein the at least one controller is further configured to repeat the searching, interpolating, and determining to create another estimated volume.