Continuous burst mode digital camera

• US 7,787,026 B1
• Filed: 11/04/2004
• Issued: 08/31/2010
• Est. Priority Date: 04/28/2004
• Status: Active Grant

• Current Assignee: MediaTek Singapore Pte. Ltd. (MediaTek, Inc.)
• Original Assignee: MediaTek Singapore Pte. Ltd. (MediaTek, Inc.)
• Inventors: Do, Hung, Flory, Kevin, Loi, Steven D., Stemper, William, Noguchi, Yasu
• Primary Examiner(s): Chan; Jason
• Assistant Examiner(s): FOSSELMAN, JOEL W
• Application Number:
  US10/981,213
• Time in Patent Office: 2,126 Days
• Field of Search: 348/231.99
• US Class Current: 348/231.99
• CPC Class Codes: H04N 1/2166 for mass storage, e.g. in document filing systems information retrieval G06F16/00
• CPC Class Codes: H04N 2101/00 Still video cameras
• CPC Class Codes: H04N 2201/3247 Data linking a set of images to one another, e.g. sequence, burst or continuous capture mode
• CPC Class Codes: H04N 23/12 with one sensor only
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/88 for colour balance, e.g. white-balance circuits or colour temperature control
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure
• CPC Class Codes: H04N 5/907 using static stores, e.g. storage tubes or semiconductor memories H04N5/91 takes precedence
• CPC Class Codes: H04N 9/8042 involving data reduction

Abstract

A camera has a continuous full-resolution burst mode wherein a sequence of full-resolution images is captured, is image processed by a pipeline of dedicated hardware image processing engines, is zoomed by a zoom engine, is compressed by a compression engine, and is stored into nonvolatile storage as a sequence of discrete files. The capturing of images and the storing of files continues at a rate of at least three frames per second until the user indicates burst mode operation is to stop or until nonvolatile storage becomes filled. Although the camera has a buffer memory into which raw sensor data is placed before image processing, the number of images that can be captured in a single burst is not limited by the size of the buffer memory. The cost of a consumer market camera having continuous burst mode capability is therefore reduced by reducing the required amount of buffer memory.

Claims

1. A digital camera, comprising:

image capture means for capturing a sequence of at least twenty images and for outputting a stream of sensor data, wherein each image is output by the image capture means as a set of at least five million sensor data values, wherein the image capture means outputs the sensor data values of the stream of sensor data at a sustained average rate of at least three of the sets per second;

nonvolatile storage; and

pipeline processing means for receiving the stream of sensor data at the sustained average rate of at least three of the sets per second and for performing color interpolation, gamma correction, color space conversion, zooming, and compression on the sensor data values of the stream of sensor data at the sustained average rate of at least three of the sets per second, generating processed sensor data values of a stream of processed sensor data and thereby outputting a stream of at least twenty files of processed sensor data values of the stream of processed sensor data, wherein each of the files corresponds with a respective one of the at least twenty images in the sequence captured by the image capture means, wherein the pipeline processing means performs zooming on some of the sensor data values of the set of at least five million sensor data values before color space conversion has been performed on all of the sensor data values of the set of at least five million sensor data values, and wherein the pipeline processing means is also for storing the stream of files in the nonvolatile storage at a sustained average rate of at least three files per second.

2. The digital camera of claim 1, wherein each of the files is an EXIF file.

3. The digital camera of claim 2, wherein the image capture means includes an image sensor and an analog to digital (A/D) converter.

13. The digital camera of claim 1, wherein the compression engine processes a first frame of the sequence of frames at the same time that the pipeline of image processing engines processes a second frame of the sequence of frames.

4. A method comprising:

(a) capturing a sequence of at least twenty images in an image sensor at a capture and readout frame rate, wherein the image sensor outputs a frame of sensor data of at least five million sensor data values for each image in the sequence, and wherein the image sensor has a maximum sustained full-resolution image frame capture and readout rate (MSFRIFCRR) of at least three frames per second, wherein the capture and readout frame rate is at least the MSFRIFCRR rate;

(b) using a digital imaging pipeline to perform image processing on each frame of sensor data at a rate at least equal to the MSFRIFCRR and outputting a frame of image processed pixel data corresponding to each frame of sensor data received, wherein the image processing includes color space conversion and zooming, and wherein the digital imaging pipeline zooms some data from each frame of sensor data before performing color space conversion on all of the data of each frame of sensor data;

(c) receiving each frame of image processed pixel data into a JPEG compression mechanism, the JPEG compression mechanism performing JPEG compression at a rate at least equal to the MSFRIFCRR rate and outputting a frame of compressed image data for each frame of image processed pixel data received, the JPEG compression mechanism outputting a sequence of at least twenty frames of compressed image data wherein there is one frame of JPEG compressed image data for each of said at least twenty images captured by the image sensor; and

(d) storing the sequence of at least twenty frames of JPEG compressed image data into nonvolatile storage at a rate at least equal to the MSFRIFCRR rate, wherein each frame of JPEG compressed image is stored in nonvolatile storage as a discrete file.

5. The method of claim 4, wherein the image sensor, the digital imaging pipeline, the JPEG compression mechanism and the nonvolatile storage are parts of a battery-powered handheld digital camera.

6. The method of claim 5, wherein the image sensor has at least five million sensors, and wherein each frame of sensor data in the sequence includes at least five million sensor data values.

7. The method of claim 4, wherein the digital imaging pipeline of (b) includes:

a Bayer-to-RGB engine, a gamma correction engine, and a zoom engine.

8. The method of claim 4, wherein the image sensor outputs a frame of sensor data for a first image of the sequence of images in a first time period, and wherein a frame of JPEG compressed image data for the same first image is stored into the nonvolatile storage in a second time period, and wherein the first time period overlaps the second time period.

9. The method of claim 4, wherein the image processing of (b) includes performing gamma correction on a first frame of image information associated with the first image of the sequence of images, the gamma correction on the first frame being performed in a first time period, and wherein the processing of (b) further includes performing zoom processing on the first frame of image information associated with the first image of the sequence of images, the zoom processing of the first frame being performed in a second time period, wherein the first time period overlaps the second time period.

10. A digital camera,

comprising:

an image sensor having at least five million sensors, wherein the image sensor captures and outputs a sequence of at least twenty full-resolution frames

of image data at a capture and readout rate of at least three frames per second;

a random access memory (RAM) buffer memory;

a pipeline of image processing engines that zooms some data of a frame of image data before performing color space conversion on all of the data of the frame of image data;

a compression engine; and

a nonvolatile storage device, wherein each frame of the sequence of full-resolution frames of image data passes into the RAM buffer, and passes from the RAM buffer, through the pipeline of image processing engines at an average rate of at least three frames per second, through the compression engine at an average rate of at least three frames per second, and is stored as a discrete file into the nonvolatile storage device, and wherein the discrete files associated with said at least twenty images are stored into the nonvolatile storage device at an average frame storage of at least three frames per second.

11. The digital camera of claim 10, wherein the sequence of at least twenty full-resolution frames of image data is digitized into a first number of bits of raw sensor image data, and wherein the RAM buffer memory has a storage capacity of a second number of bits, and wherein the second number of bits is smaller than the first number of bits.

12. The digital camera of claim 10, wherein the compression engine processes a first frame of the sequence of frames at the same time that the pipeline of image processing engines processes a second frame of the sequence of frames.

14. The digital camera of claim 10, wherein the pipeline of image processing engines comprises a chain of dedicated hardware engines, the dedicated hardware engines including:

a Bayer-to-RGB engine, a white balance engine, a gamma correction engine, and a RGB-to-YUV engine.

15. The digital camera of claim 10, further comprising:

a display, wherein the pipeline of image processing engines outputs one reduced size image frame for rendering on the display of the camera for each corresponding one of the discrete files stored into the nonvolatile storage device, wherein the reduced size image frame is rendered on the display at substantially the same time that the corresponding discrete file is stored into the nonvolatile storage device.

16. A digital camera having a continuous full-resolution burst capture mode, the digital camera comprising:

an image sensor having at least five million sensors;

a pipeline of image processing engines;

a zoom engine;

a compression engine; and

a nonvolatile storage device, wherein in the continuous full-resolution burst capture mode;

a sequence of at least twenty full-resolution images is captured by the image sensor, wherein each of the full-resolution images is image processed by the pipeline of image processing engines, is zoomed by the zoom engine, is compressed by the compression engine, and is stored into the nonvolatile storage device as a discrete file, wherein the zoom engine zooms some data of an image before the pipeline of image processing engines performs image processing on all of the data of the image, wherein the sequence of at least twenty full-resolution images is captured by the image sensor at a sustained frame rate of at least three frames per second, is image processed by the pipeline of image processing engines at the sustained frame rate of at least three frames per second, is zoomed by the zoom engine at the sustained frame rate of at least three frames per second, is compressed by the compression engine at the sustained frame rate of at least three frames per second, and wherein the discrete files are written into nonvolatile memory at the sustained rate of at least three files per second.

17. The digital camera of claim 16, wherein the digital camera is a battery-powered handheld still camera, and wherein the nonvolatile storage medium is taken from the group consisting of:

a flash memory, a hard disk, a microdrive that plugs into a compact flash slot.

18. The digital camera of claim 17, wherein the image sensor outputs for each of the full-resolution images at least five million analog Bayer sensor output values.

19. The digital camera of claim 17, wherein the image sensor is taken from the group consisting of:

a charge coupled device (CCD) image sensor, and a complementary metal oxide semiconductor (CMOS) image sensor.

20. The digital camera of claim 16, further comprising:

an analog-to-digital converter that receives raw analog sensor data from the image sensor and digitizes the raw analog sensor data into digital sensor data and supplies the digital sensor data to the pipeline of image processing engines.