Bit-rate reduction for multimedia data streams

• US 8,160,160 B2
• Filed: 09/08/2006
• Issued: 04/17/2012
• Est. Priority Date: 09/09/2005
• Status: Expired due to Fees

• Current Assignee: Broadcast International, Inc. (Creative Realities, Inc.)
• Original Assignee: Broadcast International, Inc. (Creative Realities, Inc.)
• Inventors: Blum, David Frederique
• Primary Examiner(s): Rao, Andy
• Application Number:
  US11/518,298
• Publication Number:
  US 20070058716A1
• Time in Patent Office: 2,048 Days
• Field of Search: 37524001-24029
• US Class Current: 375/240.29
• CPC Class Codes: H04N 19/115 Selection of the code volume for a coding unit prior to coding
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/146 Data rate or code amount at the encoder output
• CPC Class Codes: H04N 19/15 by monitoring actual compressed data size at the memory before deciding storage at the transmission buffer
• CPC Class Codes: H04N 19/154 Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion use of rate-distortion criteria H04N19/147
• CPC Class Codes: H04N 19/194 involving only two passes
• CPC Class Codes: H04N 19/523 with sub-pixel accuracy
• CPC Class Codes: H04N 19/533 Motion estimation using multistep search, e.g. 2D-log search or one-at-a-time search [OTS]
• CPC Class Codes: H04N 19/583 Motion compensation with overlapping blocks
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/63 using sub-band based transform, e.g. wavelets
• CPC Class Codes: H04N 19/80 Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation H04N19/635, H04N19/86 take precedence
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression

Abstract

Systems and methods for reducing a bit-rate in a multimedia data stream while maintaining high image quality. Two consecutive frames of the multimedia data stream are spatially filtered, temporally filtered, and combined into a pre-processed frame. In one embodiment, the two consecutive frames are consecutive fields of an interlaced data stream that are deinterlaced by the spatial and temporal filtering process. The pre-processed frame is encoded multiple times using different quantization values to produce a plurality of encoded frames. One of the encoded frames is selected based on its image quality and/or size for inclusion in an encoded data stream. In one embodiment, the pre-processed frame undergoes a motion estimation method that performs an integer level search of overlapping locations around a co-located macroblock in a reference frame followed by a sub-pixel level diamond pattern search.

Claims

1. A method for reducing a bit rate in a multimedia data stream, the method comprising:

spatially filtering two consecutive frames of a first multimedia data stream, wherein spatially filtering comprises;

splitting a frame into a plurality of sub-frames each comprising a two-dimensional array of sample values, the two-dimensional array corresponding to the spatial dimensions of the frame; and

for at least a first sub-frame;

determining an average sample value for the first sub-frame; and

replacing all of the sample values in the first sub-frame with the average sample value;

temporally filtering and combining the two consecutive frames to produce a pre-processed frame;

encoding the pre-processed frame using a plurality of scalar quantization values to produce a plurality of encoded frames;

selecting one of the encoded frames to include in a second multimedia data stream, the selection based at least in part on the visual quality of the encoded frames; and

for at least a second sub-frame;

determining a maximum sample value for the second sub-frame; and

replacing sample values in the second sub-frame that are greater than a predetermined portion of the maximum sample value with a replacement value that substantially equals the predetermined portion of the maximum sample value;

wherein the predetermined portion of the maximum sample value comprises approximately one-half the maximum sample value.

2. The method of claim 1, wherein the selection of one of the encoded frames for inclusion in the second multimedia data stream is further based at least in part on the size of the encoded frames.

3. The method of claim 2, wherein, if more than one of the encoded frames have a minimum size, the selection of one of the encoded frames for inclusion in the second multimedia data stream is further based at least in part on the smallest scalar quantization value used to encode the pre-processed frame.

4. The method of claim 1, wherein spatially filtering the two consecutive frames comprises spatially filtering the two consecutive frames in parallel.

5. The method of claim 1, further comprising reducing noise in the spatial domain in the second sub-frame.

6. The method of claim 5, wherein reducing noise in the spatial domain for at least the second sub-frame comprises:

transforming the second sub-frame to the frequency domain; and

removing frequency samples from the second sub-frame that are less than a frequency threshold.

7. The method of claim 6, wherein transforming comprises performing a discrete cosine transform on the second sub-frame.

8. The method of claim 7, further comprising:

performing an inverse discrete cosine transform on the second sub-frame; and

passing the second sub-frame through a two-dimensional Wiener filter.

9. The method of claim 1, wherein splitting the frame comprises:

low pass filtering and decimating the frame along a first spatial dimension;

high pass filtering and decimating the frame along the first spatial dimension;

low pass filtering and decimating the frame along a second spatial dimension; and

high pass filtering and decimating the frame along the second spatial dimension.

10. The method of claim 9, further comprising recombining the sub-frames by:

low pass filtering and interpolating the plurality of sub-frames along the second spatial dimension;

high pass filtering and interpolating the plurality of sub-frames along the second spatial dimension;

low pass filtering and interpolating the plurality of sub-frames along the first spatial dimension; and

high pass filtering and interpolating the plurality of sub-frames along the first spatial dimension.

11. The method of claim 1, wherein splitting the frame comprises splitting the frame into nineteen sub-frames.

12. The method of claim 1, wherein temporally filtering the two consecutive frames comprises:

creating a three-dimensional array comprising the spatially filtered two consecutive frames, a first and a second dimension of the three-dimensional array corresponding to spatial dimensions and a third dimension corresponding to a temporal dimension of the two consecutive frames; and

filtering and decimating the three-dimensional array along the first dimension, the second dimension, and the third dimension so as to split the three-dimensional array into a plurality of smaller arrays.

13. The method of claim 12, further comprising passing at least one of the plurality of smaller arrays through a Wiener filter to reduce noise in the temporal dimension.

14. The method of claim 12, further comprising performing a clear operation on one or more of the plurality of smaller arrays.

15. The method of claim 14, further comprising:

determining an average sample value for a selected smaller array of the plurality of smaller arrays; and

replacing all of the sample values in the selected smaller array with the average sample value.

16. The method of claim 14, further comprising:

determining a maximum sample value for a selected smaller array of the plurality of smaller arrays; and

replacing sample values in the selected smaller array that are greater than a predetermined portion of the maximum sample value with a replacement value that substantially equals the predetermined portion of the maximum sample value.

17. The method of claim 16, wherein the predetermined portion of the maximum sample value comprises approximately one-half the maximum value.

18. The method of claim 14, further comprising:

performing a discrete cosine transform on a selected smaller array of the plurality of smaller arrays;

removing samples from the selected smaller array that are less than a threshold value;

performing an inverse discrete cosine transform on the selected smaller array; and

passing the selected smaller array through a through a two-dimensional Wiener filter.

19. The method of claim 12, further comprising:

filtering and interpolating the plurality of smaller arrays along the first dimension, the second dimension, and the third dimension so as to recombine the plurality of smaller arrays to produce the pre-processed frame.

20. The method of claim 1, wherein the visual quality of the encoded frames is based at least in part on a peak signal-to-noise ratio (PSNR) score.

21. The method of claim 1, wherein encoding the pre-processed frame comprises:

dividing the pre-processed frame into a plurality of macroblocks;

performing motion estimation for each macroblock in the pre-processed frame, the motion estimation comprising;

searching overlapping locations around a co-located macroblock in a reference frame to determine a first estimated location having a minimum error measure at an integer pixel level; and

searching locations in a diamond pattern around the first estimated location to determine a second estimated location having a minimum error measure at a quarter and half pixel level.

22. The method of claim 21, further comprising searching quarter and half pixel level locations in a diamond pattern around the second estimated location to determine a third estimated location having a minimum error measure at the quarter and half pixel level.

23. The method of claim 21, wherein the error measure is a sum of absolute differences (SAD) between the macroblock in the pre-processed frame and the co-located macroblock in the reference frame.

24. A method for reducing a bit rate in a multimedia data stream, the method comprising:

spatially filtering two consecutive frames of a first multimedia data stream, wherein spatially filtering comprises;

splitting a frame into a plurality of sub-frames each comprising a two-dimensional array of sample values, the two-dimensional array corresponding to the spatial dimensions of the frame, wherein splitting the frame comprises;

low pass filtering and decimating the frame along a first spatial dimension;

high pass filtering and decimating the frame along the first spatial dimension;

low pass filtering and decimating the frame along a second spatial dimension; and

high pass filtering and decimating the frame along the second spatial dimension; and

for at least a first sub-frame;

determining an average sample value for the first sub-frame; and

replacing all of the sample values in the first sub-frame with the average sample value;

temporally filtering and combining the two consecutive frames to produce a pre-processed frame;

encoding the pre-processed frame using a plurality of scalar quantization values to produce a plurality of encoded frames;

selecting one of the encoded frames to include in a second multimedia data stream, the selection based at least in part on the visual quality of the encoded frames; and

recombining the sub-frames by;

low pass filtering and interpolating the plurality of sub-frames along the second spatial dimension;

high pass filtering and interpolating the plurality of sub-frames along the second spatial dimension;

low pass filtering and interpolating the plurality of sub-frames along the first spatial dimension; and

high pass filtering and interpolating the plurality of sub-frames along the first spatial dimension.

25. A method for reducing a bit rate in a multimedia data stream, the method comprising:

spatially filtering two consecutive frames of a first multimedia data stream;

temporally filtering and combining the two consecutive frames to produce a pre-processed frame, wherein temporally filtering comprises;

creating a three-dimensional array comprising the spatially filtered two consecutive frames, a first and a second dimension of the three-dimensional array corresponding to spatial dimensions and a third dimension corresponding to a temporal dimension of the two consecutive frames; and

filtering and decimating the three-dimensional array along the first dimension, the second dimension, and the third dimension so as to split the three-dimensional array into a plurality of smaller arrays;

encoding the pre-processed frame using a plurality of scalar quantization values to produce a plurality of encoded frames;

selecting one of the encoded frames to include in a second multimedia data stream, the selection based at least in part on the visual quality of the encoded frames;

performing a clear operation on one or more of the plurality of smaller arrays;

determining an average sample value for a selected smaller array of the plurality of smaller arrays; and

replacing all of the sample values in the selected smaller array with the average sample value.

26. A method for reducing a bit rate in a multimedia data stream, the method comprising:

spatially filtering two consecutive frames of a first multimedia data stream;

temporally filtering and combining the two consecutive frames to produce a pre-processed frame, wherein temporally filtering comprises;

creating a three-dimensional array comprising the spatially filtered two consecutive frames, a first and a second dimension of the three-dimensional array corresponding to spatial dimensions and a third dimension corresponding to a temporal dimension of the two consecutive frames; and

filtering and decimating the three-dimensional array along the first dimension, the second dimension, and the third dimension so as to split the three-dimensional array into a plurality of smaller arrays;

encoding the pre-processed frame using a plurality of scalar quantization values to produce a plurality of encoded frames;

selecting one of the encoded frames to include in a second multimedia data stream, the selection based at least in part on the visual quality of the encoded frames;

performing a clear operation on one or more of the plurality of smaller arrays; and

filtering and interpolating the plurality of smaller arrays along the first dimension, the second dimension, and the third dimension so as to recombine the plurality of smaller arrays to produce the pre-processed frame.