Systems and methods for rolling shutter compensation using iterative process

• US 10,412,328 B2
• Filed: 01/25/2019
• Issued: 09/10/2019
• Est. Priority Date: 02/22/2017
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Fontvielle, Adrien, Kok, Sophana
• Primary Examiner(s): Henn, Timothy J
• Application Number:
  US16/257,290
• Publication Number:
  US 20190158768A1
• Time in Patent Office: 228 Days
• CPC Class Codes: G06T 3/60 Rotation of whole images or parts thereof
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 25/531 by controlling rolling shutters in CMOS SSIS
• CPC Class Codes: H04N 25/533 by using differing integration times for different sensor regions
• CPC Class Codes: H04N 25/60 Noise processing, e.g. detecting, correcting, reducing or removing noise

Abstract

Image captured with an image capture device with a rolling shutter may be deformed due to changes in imaging sensor orientation during image capture. Image deformities may occur due to rolling shutter that exposes rows of pixels to light at slightly different times during image capture. Deformities such as wobble, for example, and/or other deformities may be corrected by constructing an output image. The output image may be constructed by determining corresponding pixels within the input image. The location of the input pixel may be determined by performing one or more fixed point iterations to identify one or more input pixels within the input image. A value of the output pixel within the output image may be determined based on a value of a corresponding pixel within the input image.

Claims

1. A system for correcting digital image deformities, the system comprising:

one or more physical processors configured by machine readable instructions to;

obtain an input image defined by an input pixel array, the input pixel array captured by an imaging sensor, the input pixel array including input pixels characterized by input pixel positions within the input pixel array and input pixel values;

obtain acquisition times specifying time of capture of sets of input pixels within the input pixel array;

obtain orientation information specifying imaging sensor orientations at the acquisition times of the sets of input pixels within the input pixel array;

determine an output image defined by an output pixel array, the output pixel array determined based on the input pixel array of the input image, the output pixel array including output pixels characterized by output pixel positions within the output pixel array and output pixel values, wherein determination of a given output pixel includes;

(a) determining an acquisition time and an orientation information of the given output pixel within the output pixel array;

(b) obtaining identification of an input pixel based on the acquisition time and the orientation information of the given output pixel, the input pixel having a pixel position within the input pixel array;

(c) obtaining an input acquisition time of the input pixel based on the pixel position within the input pixel array;

(d) determining input orientation information of the input pixel based on the input acquisition time;

(e) obtaining identification of a subsequent input pixel based on the input orientation information;

(f) iterating over operations (c)-(e) for one or more other pixels of the input pixel array; and

(g) obtaining identification of a pixel within the input pixel array as an estimated input pixel based on differences in the input orientation information; and

determining the output pixel values for the given output pixel based on the input pixel values for the estimated input pixel.

2. The system of claim 1, wherein the output pixel array includes multiple cells that correspond to the input pixels in the input pixel array, the individual cells in the output pixel array being characterized by two-dimensional coordinates and identification of corresponding ones of the input pixel array.

3. The system of claim 1, wherein identification of pixels of the input pixel array corresponding to individual cells in the output pixel array includes specification of individual input pixel positions in the input pixel array.

4. The system of claim 3, wherein the individual input pixel positions are identified based on the individual orientation information.

5. The system of claim 4, wherein individual output pixels within the output pixel array are determined by obtaining identification of the estimate input pixel within the input pixel array based on the iteration process such that for a given iteration, each subsequent pixel within the input pixel array is determined based on orientation information of a previous pixel determined at another iteration preceding the given iteration.

6. The system of claim 1, wherein a number of iterations in the iteration process is determined based on orientation information of one or more pixels in one or more successive iterations.

7. The system of claim 1, wherein the imaging sensor is configured to obtain the orientation information.

8. The system of claim 7, wherein the imaging sensor is configured to be coupled to an orientation sensor, the orientation sensor comprising one or more of a gyroscope and/or accelerometer.

9. The system of claim 1, wherein the imaging sensor comprises an array of rows of photosites configured based on a CMOS imaging process wherein a time of acquiring content of a given row of photosites of the array is delayed relative to a time instance of acquiring content of another row of photosites of the array.

10. The system of claim 1, wherein the output pixel array comprises at least 64 cells.

11. A method of correcting digital image deformities, the method comprising:

obtaining an input image defined by an input pixel array, the input pixel array captured by an imaging sensor, the input pixel array including input pixels characterized by pixel positions within the input pixel array and input pixel values;

obtaining acquisition times specifying time of capture of sets of input pixels within the input pixel array;

obtaining orientation information specifying imaging sensor orientations at the acquisition times of the sets of input pixels within the input pixel array;

determining an output image defined by an output pixel array, the output pixel array determined based on the input pixel array of the input image, the output pixel array including output pixels characterized by output pixel positions within the output pixel array and output pixel values, wherein determination of a given output pixel includes;

(a) determining an acquisition time and an orientation information of the given output pixel within the output pixel array;

(b) obtaining identification of an input pixel based on the acquisition time and the orientation information of the given output pixel, the input pixel having a pixel position within the input pixel array;

(c) obtaining an input acquisition time of the input pixel based on the pixel position within the input pixel array;

(d) determining input orientation information of the input pixel based on the input acquisition time;

(e) obtaining identification of a subsequent input pixel based on the input orientation information;

(f) iterating over operations (c)-(e) for one or more other pixels of the input pixel array; and

(g) obtaining identification of a pixel within the input pixel array as an estimated input pixel based on differences in the input orientation information; and

determining the output pixel values for the given output pixel based on the input pixel values for the estimated input pixel.

12. The method of claim 11, wherein the output pixel array includes multiple cells that correspond to the input pixels in the input pixel array, the individual cells in the output pixel array being characterized by two-dimensional coordinates and identification of corresponding ones of the input pixel array.

13. The method of claim 11, wherein identification of pixels of the input pixel array corresponding to individual cells in the output pixel array includes specification of individual input pixel positions in the input pixel array.

14. The method of claim 13, wherein the individual input pixel positions are identified based on the individual orientation information.

15. The method of claim 14, wherein individual output pixels within the output pixel array are determined by obtaining identification of the estimate input pixel within the input pixel array based on the iteration process such that for a given iteration, each subsequent pixel within the input pixel array is determined based on orientation information of a previous pixel determined at another iteration preceding the given iteration.

16. The method of claim 11, wherein a number of iterations in the iteration process is determined based on orientation information of one or more pixels in one or more successive iterations.

17. The method of claim 11, wherein the imaging sensor is configured to obtain the orientation information.

18. The method of claim 17, wherein the imaging sensor is configured to be coupled to an orientation sensor, the orientation sensor comprising one or more of a gyroscope and/or accelerometer.

19. The method of claim 11, wherein the imaging sensor comprises an array of rows of photosites configured based on a CMOS imaging process wherein a time of acquiring content of a given row of photosites of the array is delayed relative to a time instance of acquiring content of another row of photosites of the array.

20. The method of claim 11, wherein the output pixel array comprises at least 64 cells.