Imaging systems with fixed output sizes and frame rates

• US 8,526,752 B2
• Filed: 08/23/2010
• Issued: 09/03/2013
• Est. Priority Date: 08/23/2010
• Status: Active Grant

• Current Assignee: Aptina Imaging Corporation (ON Semiconductor Corporation)
• Original Assignee: Aptina Imaging Corporation (ON Semiconductor Corporation)
• Inventors: Lin, Sheng, Zerwick, Adam, Mok, Kai Yau, Chang, Chi-Wen
• Primary Examiner(s): Ye, Lin
• Assistant Examiner(s): Morehead, III, John H
• Application Number:
  US12/861,760
• Publication Number:
  US 20120044375A1
• Time in Patent Office: 1,107 Days
• Field of Search: None
• US Class Current: 382/250
• CPC Class Codes: H04N 19/146 Data rate or code amount at the encoder output
• CPC Class Codes: H04N 19/188 the unit being a video data packet, e.g. a network abstraction layer [NAL] unit
• CPC Class Codes: H04N 19/625 using discrete cosine transform [DCT]
• CPC Class Codes: H04N 19/68 involving the insertion of resynchronisation markers into the bitstream

Abstract

An imaging system may include an image sensor and an image encoder that encodes images from the image sensor with fixed output sizes and frame rates. The image encoder may encode images from the image sensor into an image format such as a Joint Photographic Experts Group (JPEG) format. The image encoder may insert padding data between image blocks in the encoded data to compensate in real time for variations in the encoded size of an image. The amount of padding data inserted by the encoder may be calculated to ensure the encoded image has a file size close to, but not greater than, the required fixed output size. If needed, the encoder may add additional padding data after the image blocks are encoded in a blanking period before a subsequent image is encoded so that the final size of the encoded image is equal to the required output size.

Claims

1. A method of generating a data stream of encoded image frames using an image encoder in an electronic device, wherein each of the encoded image frames has a given predetermined file size and wherein the encoded image frames are transmitted in the data stream at a given predetermined frame rate, the method comprising:

with the image encoder and for a given image frame in the data stream, inserting into the data stream a first block of encoded image data during a first time period and inserting into the data stream a second block of encoded image data during a second time period that is after the first time period;

after the beginning of the first time period and prior to the beginning of the second time period, determining if the image encoder is on track to encode the given image frame at the given predetermined file size; and

with the image encoder, for the given image frame, and between the first time period and the second period, inserting a given block of padding data into the data stream, wherein the given block of padding data has a size that is varied in response to the determination of if the image encoder is on track to encode the given image frame at the given predetermined file size.

2. The method defined in claim 1 further comprising:

with the image encoder and for the given image frame, inserting into the data stream a final block of encoded image data during a third time period that is after the second time period; and

with the image encoder, for the given image frame, and after the third time period, inserting an additional block of padding data into the data stream during a blanking period before another encoded image frame is inserted into the data stream of encoded image frames by the image encoder.

3. The method defined in claim 1 the method further comprising:

with the image encoder, before the first time period, and for the given image frame, inserting into the data stream a first plurality of blocks of encoded image data;

with the image encoder, after the second time period, and for the given image frame, inserting into the data stream a second plurality of blocks of encoded image data; and

with the image encoder and for the given image frame, inserting additional blocks of padding data into the data stream, wherein each of the additional blocks of padding data is associated with a respective one of the blocks of encoded image data from the first and second pluralities of blocks of encoded image data.

4. The method defined in claim 3 wherein inserting the given block of padding data into the data stream comprises:

calculating a remaining file size value by subtracting at least the combined file size of the first plurality of blocks of encoded image data and the first block of encoded image data from the given predetermined file size.

5. The method defined in claim 4 wherein each of the encoded image frames is an encoded version of an un-encoded image frame, wherein each of the un-encoded image frames has a file size, and wherein the first plurality of blocks of encoded image data are associated with a first portion of a given un-encoded image frame, and wherein inserting the given block of padding data into the data stream comprises:

calculating a current compression ratio by dividing the combined file size of the first plurality of blocks of encoded image data and the first block of encoded image data by the file size of the first portion of the given un-encoded image frame.

10. The method defined in claim 1 further comprising:

for each of the encoded image frames, inserting blocks of encoded image data and padding data into the data stream, wherein the blocks of encoded image data and padding data for each individual encoded image frame are inserted into the data stream within a period of time determined by the given predetermined frame rate.

11. The method defined in claim 1 further comprising:

for each of the encoded image frames, inserting blocks of encoded image data and padding data into the data stream, wherein the blocks of encoded image data and padding data for each individual encoded image frame have a combined file size equal to the given predetermined file size.

12. The method defined in claim 1 further comprising:

for each of the encoded image frames, inserting blocks of encoded image data and padding data into the data stream, wherein the blocks of encoded image data and padding data for each individual encoded image frame are inserted into the data stream within a period of time determined by the given predetermined frame rate, and wherein the blocks of encoded image data and padding data for each individual encoded image frame have a combined file size equal to the given predetermined file size.

6. A method of generating a data stream of encoded image frames using an image encoder in an electronic device, wherein each of the encoded image frames has a given predetermined file size and wherein the encoded image frames are transmitted in the data stream at a given predetermined frame rate, the method comprising:

with the image encoder and for a given image frame in the data stream, inserting into the data stream a first block of encoded image data during a first time period and inserting into the data stream a second block of encoded image data during a second time period that is after the first time period;

with the image encoder, for the given image frame, and between the first time period and the second period, inserting a given block of padding data into the data stream;

with the image encoder, before the first time period, and for the given image frame, inserting into the data stream a first plurality of blocks of encoded image data;

with the image encoder, after the second time period, and for the given image frame, inserting into the data stream a second plurality of blocks of encoded image data; and

with the image encoder and for the given image frame, inserting additional blocks of padding data into the data stream, wherein each of the additional blocks of padding data is associated with a respective one of the blocks of encoded image data from the first and second pluralities of blocks of encoded image data, wherein inserting the given block of padding data into the data stream comprises;

calculating a remaining file size value by subtracting at least the combined file size of the first plurality of blocks of encoded image data and the first block of encoded image data from the given predetermined file size, wherein each of the encoded image frames is an encoded version of an un-encoded image frame, wherein each of the un-encoded image frames has a file size, and wherein the first plurality of blocks of encoded image data are associated with a first portion of a given un-encoded image frame, and wherein inserting the given block of padding data into the data stream comprises;

calculating a current compression ratio by dividing the combined file size of the first plurality of blocks of encoded image data and the first block of encoded image data by the file size of the first portion of the given un-encoded image frame, wherein the given un-encoded image frame includes the first portion that has been encoded and includes a second portion that includes a portion that has not yet been encoded and a portion associated with the second block of encoded image data and wherein inserting the given block of padding data into the data stream comprises;

calculating an estimated remaining encoded file size value by dividing the file size of the second portion of the given un-encoded image frame by the current compression ratio.

7. The method defined in claim 6 wherein inserting the given block of padding data into the data stream comprises:

determining if the image encoder is on track to encode the given image frame at the given predetermined file size by subtracting the estimated remaining encoded file size value from the remaining file size value.

8. The method defined in claim 7 wherein, when the image encoder is on track to encode the given image frame at a file size that is less than the given predetermined file size, inserting the given block of padding data into the data stream comprises inserting a sufficient amount of padding data such that the image encoder is on pace to encode the given image frame at a file size that is equal to the given predetermined file size.

9. The method defined in claim 7 wherein inserting into the data stream the first block of encoded image data comprises encoding image data with the image encoder at a first compression ratio and wherein, when the image encoder is on track to encode the given image frame at a file size that is greater than the given predetermined file size, inserting into the data stream the second block of encoded image data comprises encoding image data with the image encoder at a second compression ratio, and wherein the second compression ratio is greater than the first compression ratio.

13. A method comprising:

with an image encoder in an electronic device, producing a data stream of encoded image frames at a given frame rate, wherein producing the stream of encoded image frames comprises producing each of the image frames at a given file size; and

with the image encoder, inserting a given amount of padding data into the data stream between a first encoded image portion of a given image frame and a second encoded image portion of the given image frame, wherein the first encoded image portion of the given image frame corresponds to a first portion of a given unencoded image frame, wherein the second encoded image portion of the given image frame corresponds to a second portion of the given unencoded image frame, and wherein inserting the padding data into the data stream between the first encoded image portion of the given image frame and the second encoded image portion of the given image frame comprises;

determining the given amount of the padding data by dividing the size of the second portion of the given unencoded image frame by a current compression ratio.

14. The method defined in claim 13 further comprising:

with the image encoder and for the given image frame, inserting into the data stream a final encoded image portion of the given image frame after inserting the first and second encoded image portions of the given image frame; and

with the image encoder, for the given image frame, and after interesting into the data stream the final encoded image portion, inserting padding data into the data stream during a blanking period before another image frame in the data stream of encoded image frames is inserted into the data stream by the image encoder.

15. The method defined in claim 13 further comprising:

determining if the image encoder is on track to encode the given image frame at the given file size.

16. The method defined in claim 13, wherein the given image frame includes a plurality encoded image portions including and in addition to the first and second encoded image portions, the method further comprising:

with the image encoder and for the given image frame, inserting blocks of padding data into the data stream between at least some of the plurality of encoded image portions.

17. The method defined in claim 13 wherein the image encoder comprises a Joint Photographic Experts Group (JPEG) image encoder and wherein producing the stream of encoded image frames comprises producing a stream of image frames encoded in a JPEG compatible format.

18. An electronic device comprising:

a host subsystem configured to receive a data stream of encoded image frames at a given predetermined frame rate, wherein each of the encoded image frames has a given predetermined file size; and

an image encoder that produces the data stream of encoded image frames by, for each of the encoded image frames, inserting padding data into the data stream between at least a first encoded portion of that image frame and a second encoded portion of that image frame, wherein the image encoder determines the amount of the padding data that is inserted into the data stream between the first and second encoded portions at least partly based on the given predetermined file size, the file size of the first encoded portion, and an estimated file size of the second encoded portion.

19. The electronic device defined in claim 18 further comprising:

image sensor circuitry that converts incident light into digital signals that are provided to the image encoder.