Frame manipulation to reduce rolling shutter artifacts

• US 9,756,250 B2
• Filed: 06/13/2016
• Issued: 09/05/2017
• Est. Priority Date: 10/21/2013
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Mobbs, Paul, Campbell, Scott Patrick, Newman, David A., Rangam, Kasturi, Chawla, Sumit
• Primary Examiner(s): VILLECCO, JOHN M
• Application Number:
  US15/180,529
• Publication Number:
  US 20160301846A1
• Time in Patent Office: 449 Days
• CPC Class Codes: H04N 1/2137 with temporary storage before final recording, e.g. in a frame buffer
• CPC Class Codes: H04N 2101/00 Still video cameras
• CPC Class Codes: H04N 23/6845 by combination of a plurality of images sequentially taken
• CPC Class Codes: H04N 23/689 Motion occurring during a rolling shutter mode
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 25/531 by controlling rolling shutters in CMOS SSIS
• CPC Class Codes: H04N 25/587 acquired sequentially, e.g. using the combination of odd and even image fields
• CPC Class Codes: H04N 25/60 Noise processing, e.g. detecting, correcting, reducing or removing noise

Abstract

An image sensor of a camera system captures an image over an image capture interval of time, and waits a blanking interval of time before capturing an additional image. The captured image is provided to a frame controller, and is buffered until an image signal processor accesses the captured image. The image signal processor processes the accessed image over an image processing interval of time, producing a processed image. The image processing interval of time is selected to be greater than the image capture interval of time, but less than the sum of the image capture interval of time and the blanking interval of time. By reducing the image capture interval of time but maintaining an image processing interval of time, rolling shutter artifacts are beneficially reduced without increasing the processing resources or power required by the image signal processor to process the image.

Claims

1. A camera system, comprising:

an image sensor configured to capture a set of two or more frames over a frame set interval, wherein capturing a frame comprises capturing light during a frame capture interval to produce frame data, wherein the captured light comprises light incident upon the image sensor during the frame capture interval, and wherein the frame data is representative of the captured light;

a buffer memory for buffering the frame data associated with the set of frames;

a frame processor configured to receive the frame data associated with the set of frames from the buffer memory and combine the frame data associated with the set of frames to produce combined frame data; and

a hardware image signal processor that processes combined frame data during a frame processing interval to produce a processed frame comprising processed frame data representative of the set of frames, the frame processing interval longer than the frame capture interval and equal to or less than the frame set interval.

2. The camera system of claim 1, wherein combining the frame data associated with the set of frames comprises, for each set of corresponding pixels within the set of frames, summing pixel data corresponding to the set of corresponding pixels.

3. The camera system of claim 1, wherein combining the frame data associated with the set of frames comprises, for each set of corresponding pixels within the set of frames, averaging pixel data corresponding to the set of corresponding pixels.

4. The camera system of claim 1, wherein the frame processor is further configured to perform one or more pre-processing operations on either the frame data associated with the set of frames before combining the frame data or on the combined frame data.

5. The camera system of claim 4, wherein the pre-processing operations comprise one or more of:

noise reduction operations, defective pixel correction operations, image repair operations, and calibration operations.

6. The camera system of claim 1, wherein the buffer memory is configured to buffer the frame data associated with the set of frames until the frame processor requests the frame data associated with the set of frames.

7. The camera system of claim 1, wherein a signal-to-noise ratio (“

SNR”

) of the processed frame is greater than a SNR of the set of frames.

8. A method of capturing images, comprising:

capturing, by an image sensor, a set of two or more frames over a frame set interval, wherein capturing a frame comprises capturing light during a frame capture interval to produce frame data, wherein the captured light comprises light incident upon the image sensor during the entire frame capture interval, and wherein the frame data is representative of the captured light;

buffering, by a buffer memory, the frame data associated with the set of frames;

combining, by a frame processor configured to receive the frame data associated with the set of frames from the buffer memory, the frame data associated with the set of frames to produce combined frame data; and

processing, by a hardware image signal processor, the combined frame data during a frame processing interval to produce a processed frame comprising processed frame data representative of the set of frames, the frame processing interval longer than the frame capture interval and equal to or less than the frame set interval.

9. The method of claim 8, wherein combining the frame data associated with the set of frames comprises, for each set of corresponding pixels within the set of frames, summing pixel data corresponding to the set of corresponding pixels.

10. The method of claim 8, wherein combining the frame data associated with the set of frames comprises, for each set of corresponding pixels within the set of frames, averaging pixel data corresponding to the set of corresponding pixels.

11. The method of claim 8, wherein the frame processor is further configured to perform one or more pre-processing operations on either the frame data associated with the set of frames before combining the frame data or on the combined frame data.

12. The method of claim 11, wherein the pre-processing operations comprise one or more of:

noise reduction operations, defective pixel correction operations, image repair operations, and calibration operations.

13. The method of claim 8, wherein the buffer memory is configured to buffer the frame data associated with the set of frames until the frame processor requests the frame data associated with the set of frames.

14. The method of claim 8, wherein a signal-to-noise ratio (“

SNR”

) of the processed frame is greater than a SNR of the set of frames.

15. A non-transitory computer-readable storage medium comprising computer-executable instructions for method of capturing images, the instructions configured to, when executed, cause a camera to perform steps comprising:

capturing, by an image sensor, a set of two or more frames over a frame set interval, wherein capturing a frame comprises capturing light during a frame capture interval to produce frame data, wherein the captured light comprises light incident upon the image sensor during the frame capture interval, and wherein the frame data is representative of the captured light;

buffering, by a buffer memory, the frame data associated with the set of frames;

combining, by a frame processor configured to receive the frame data associated with the set of frames from the buffer memory, the frame data associated with the set of frames to produce combined frame data; and

processing, by a hardware image signal processor, the combined frame data during a frame processing interval to produce a processed frame comprising processed frame data representative of the set of frames, the frame processing interval longer than the frame capture interval and equal to or less than the frame set interval.

16. The non-transitory computer-readable storage medium of claim 15, wherein combining the frame data associated with the set of frames comprises, for each set of corresponding pixels within the set of frames, summing pixel data corresponding to the set of corresponding pixels.

17. The non-transitory computer-readable storage medium of claim 15, wherein combining the frame data associated with the set of frames comprises, for each set of corresponding pixels within the set of frames, averaging pixel data corresponding to the set of corresponding pixels.

18. The non-transitory computer-readable storage medium of claim 15, wherein the frame processor is further configured to perform one or more pre-processing operations on either the frame data associated with the set of frames before combining the frame data or on the combined frame data.

19. The non-transitory computer-readable storage medium of claim 18, wherein the pre-processing operations comprise one or more of:

noise reduction operations, defective pixel correction operations, image repair operations, and calibration operations.

20. The non-transitory computer-readable storage medium of claim 15, wherein a signal-to-noise ratio (“

SNR”

) of the processed frame is greater than a SNR of the set of frames.