System and method for frame capturing and processing

• US 10,148,882 B2
• Filed: 02/28/2017
• Issued: 12/04/2018
• Est. Priority Date: 10/21/2013
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Mobbs, Paul, Campbell, Scott Patrick, Newman, David A., Rangam, Kasturi, Chawla, Sumit
• Primary Examiner(s): Villecco, John
• Application Number:
  US15/444,972
• Publication Number:
  US 20170171443A1
• Time in Patent Office: 644 Days
• CPC Class Codes: H04N 1/2137 with temporary storage before final recording, e.g. in a frame buffer
• CPC Class Codes: H04N 2101/00 Still video cameras
• CPC Class Codes: H04N 23/6845 by combination of a plurality of images sequentially taken
• CPC Class Codes: H04N 23/689 Motion occurring during a rolling shutter mode
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 25/531 by controlling rolling shutters in CMOS SSIS
• CPC Class Codes: H04N 25/587 acquired sequentially, e.g. using the combination of odd and even image fields
• CPC Class Codes: H04N 25/60 Noise processing, e.g. detecting, correcting, reducing or removing noise

Abstract

Systems and methods are provided that capture and process frames of frame data. An image sensor captures frames of frame data representative of light incident upon the image sensor using a rolling shutter and outputs the frames of frame data. The image sensor captures at least one of the frames over a frame capture interval and then waits over a blanking interval before capturing another frame. A buffer receives and stores the frames output by the image sensor. An image signal processor retrieves the frames from the buffer and processes the frames over successive frame processing intervals to generate a video having a time interval per frame greater than the frame capture interval. At least one of the successive frame processing intervals is greater than the frame capture interval and is less than or equal to a sum of the frame capture interval and the blanking interval.

Claims

1. A system that captures and processes frames of frame data, comprising:

an image sensor that captures frames of frame data representative of light incident upon the image sensor using a rolling shutter and that outputs the frames of frame data, wherein the image sensor captures at least one of the frames over a frame capture interval and then waits over a blanking interval before capturing another frame;

a buffer that receives and stores the frames output by the image sensor; and

an image signal processor that retrieves the frames from the buffer and processes the frames over successive frame processing intervals to generate a video having a time interval per frame greater than the frame capture interval, wherein at least one of the successive frame processing intervals is greater than the frame capture interval and is less than or equal to a sum of the frame capture interval and the blanking interval.

2. The system of claim 1, further comprising:

a processing unit coupled to a computer-readable storage medium including instructions executable by the processing unit to;

receive a camera mode from an interface module; and

setting a frame capture rate of the image sensor according to the camera mode, wherein the frame capture interval is based on the frame capture rate.

3. The system of claim 1, wherein the frame capture interval and the blanking interval are set according to a frame processing rate.

4. The system of claim 1, wherein the blanking interval is greater than or equal to one half of the frame capture interval.

5. The system of claim 1, wherein the at least some of the frames captured by the image sensor are compressed before being stored in the buffer as compressed frames and wherein the compressed frames are decompressed before being processed by the image signal processor.

6. A method for capturing and processing frames of frame data, comprising:

capturing, by an image sensor, frames of frame data representative of light incident upon the image sensor at a frame capture rate using a Bayer color filter array and a rolling shutter, wherein at least one frame is captured over a frame capture interval before another frame is captured after a blanking interval;

storing, using a buffer, at least some of the frames captured by the image sensor;

retrieving, by an image signal processor, a subset of the frames captured by the image sensor from the buffer according to a frame processing rate that is one half or less of the frame capture rate; and

processing, by the image signal processor, the subset of the frames over successive frame processing intervals to generate a video having an output frame rate less than the frame capture rate.

7. The method of claim 6, further comprising:

receiving a camera mode from an interface module; and

responsive to receiving the camera mode, setting the frame capture rate of the image sensor according to the camera mode.

8. The method of claim 6, wherein the frame capture interval and the blanking interval are set according to the frame processing rate.

9. The method of claim 6, further comprising:

performing, by the image signal processor, clean Bayer processing and demosaicing of the retrieved frames.

10. The method of claim 6, wherein storing, using the buffer, at least some of the frames captured by the image sensor comprises compressing the at least some of the frames captured by the image sensor to produce compressed frames for storage in the buffer, and wherein retrieving, by the image signal processor, the subset of the frames captured by the image sensor from the buffer includes decompressing at least some of the compressed frames.

11. A system that captures and processes frames of frame data, comprising:

an image sensor that captures frames of frame data representative of light incident upon the image sensor at a frame capture rate using a Bayer color filter array and a rolling shutter and that outputs at least some of the frames of frame data, wherein the image sensor captures at least one of the frames over a frame capture interval and then waits over a blanking interval;

a buffer that receives and stores at least some of the frames output by the image sensor; and

an image signal processor that retrieves, from the buffer, a subset of the frames captured by the image sensor as retrieved frames according to a frame processing rate that is one half or less of the frame capture rate and that processes the retrieved frames at the frame processing rate to generate a video having an output frame rate less than the frame capture rate.

12. The system of claim 11, further comprising:

a processing unit coupled to a computer-readable storage medium including instructions executable by the processing unit to;

receive, from an interface module, a camera mode; and

responsive to receiving the camera mode, setting the frame capture rate of the image sensor according to the camera mode.

13. The system of claim 11, wherein the frames stored by the buffer include only the subset of the frames captured by the image sensor.

14. The system of claim 11, wherein the subset of the frames captured by the image sensor includes one half or less of the frames captured by the image sensor.

15. The system of claim 11, wherein the image signal processor retrieves a frame from the buffer in response to detecting that the frame is stored in the buffer.

16. The system of claim 11, wherein the frame capture interval and the blanking interval are set according to the frame capture rate.

17. The system of claim 11, wherein the buffer stores the captured frames in a circular manner.

18. The system of claim 11, further comprising a frame controller that includes the buffer, wherein the frame controller manipulates the capturing of the frames by the image sensor or the processing of retrieved frames by the image signal processor.

19. The system of claim 11, wherein the image signal processor performs clean Bayer processing and demosaicing of the retrieved frames.

20. The system of claim 11, wherein the at least some of the frames are compressed before being stored in the buffer and wherein the image signal processor retrieves the subset of the captured frames by retrieving compressed frames from the buffer and decompressing the compressed frames.