Tweening-based codec for scaleable encoders and decoders with varying motion computation capability

• US 6,907,073 B2
• Filed: 12/06/2000
• Issued: 06/14/2005
• Est. Priority Date: 12/20/1999
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: Sarnoff Corporation (SRI International, Inc.)
• Inventors: Winarsky, Norman, Kumar, Rakesh, Hanna, Keith, Sawhney, Harpreet Singh, Burt, Peter
• Primary Examiner(s): VO, TUNG T
• Application Number:
  US09/731,194
• Publication Number:
  US 20010031003A1
• Time in Patent Office: 1,651 Days
• Field of Search: H04/B00.1/66, 375/240.01, 375/240.03, 375/240.15, 375/240.16, 375/240.12, 375/240.11, 375/240.14, 375/240.09, 375/240.22, 375/240.23, 375/240.25, 375/240.13, 382/234, 382/236, 382/238, 382/250
• US Class Current: 375/240.14
• CPC Class Codes: H04N 19/107 between spatial and temporal predictive coding, e.g. picture refresh
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/137 Motion inside a coding unit, e.g. average field, frame or block difference
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/33 in the spatial domain
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/577 Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/63 using sub-band based transform, e.g. wavelets

Abstract

A scaleable video encoder has one or more encoding modes in which at least some, and possibly all, of the motion information used during motion-based predictive encoding of a video stream is excluded from the resulting encoded video bitstream, where a corresponding video decoder is capable of performing its own motion computation to generate its own version of the motion information used to perform motion-based predictive decoding in order to decode the bitstream to generate a decoded video stream. All motion computation, whether at the encoder or the decoder, is preferably performed on decoded data. For example, frames may be encoded as either H, L, or B frames, where H frames are intra-coded at full resolution and L frames are intra-coded at low resolution. The motion information is generated by applying motion computation to decoded L and H frames and used to generate synthesized L frames. L-frame residual errors are generated by performing inter-frame differencing between the synthesized and original L frames and are encoded into the bitstream. In addition, synthesized B frames are generated by tweening between the decoded H and L frames and B-frame residual errors are generated by performing inter-frame differencing between the synthesized B frames and, depending on the implementation, either the original B frames or sub-sampled B frames. These B-frame residual errors are also encoded into the bitstream. The ability of the decoder to perform motion computation enables motion-based predictive encoding to be used to generate an encoded bitstream without having to expend bits for explicitly encoding any motion information.

Claims

1. A method for encoding a video stream to generate an encoded video bitstream, comprising the steps of:

(a) encoding, into the encoded video bitstream, a first original frame/region in the video stream using intra-frame coding to generate an encoded first frame/region; and

(b) encoding, into the encoded video bitstream, a second original frame/region in the video stream using motion-based predictive coding, wherein;

the motion-based predictive coding comprises motion computation adapted to generate motion information using during the motion-based predictive coding; and

at least some of the motion information used during the motion-based predictive coding is excluded from the encoded video bitstream.

2. The invention of claim 1, wherein all of the motion information used during the motion-based predictive coding is excluded from the encoded video bitstream and the encoded video bitstream does not explicitly include any motion information.

3. The invention of claim 1, wherein step (b) comprises the steps of:

(1) decoding the encoded first frame/region to generate a decoded first frame/region;

(2) encoding the second original frame/region to generate an encoded second frame/region;

(3) decoding the encoded second frame/region to generate a decoded second frame/region;

(4) performing motion computation between the decoded second frame/region and the decoded first frame/region to generate the motion information;

(5) applying the motion information to the decoded first frame/region to generate a synthesized second frame/region;

(6) performing inter-frame differencing between the synthesized second frame/region and the second original frame/region to generate residual errors; and

(7) encoding, into the encoded video bitstream, at least some of the residual errors.

4. The invention of claim 1, further comprising the step of:

(c) encoding, into the encoded video bitstream, a third original frame/region in the video stream using tweening based on the motion information used to encode the second original frame/region.

5. The invention of claim 1, wherein the motion-based predictive coding comprises:

motion computation during which one or more motion vectors are determined for the second original frame/region; and

motion compensation based on the one or more motion vectors determined during motion computation, wherein at least one of the motion vectors used during the motion compensation is excluded from the encoded video bitstream.

6. The invention of claim 5, wherein each motion vector used during the motion compensation is excluded from the encoded video bitstream.

7. A video encoder for encoding a video stream to generate an encoded video bitstream, comprising:

(a) a frame/region type selector configured for selecting different processing paths for encoding different frames/regions into the encoded video bitstream;

(b) a first processing path configured for encoding, into the encoded video bitstream, a first original frame/region in the video stream using intra-frame coding to generate an encoded first frame/region; and

(c) a second processing path configured for encoding, into the encoded video bitstream, a second original frame/region in the video stream using motion-based predictive coding, wherein;

the motion-based predictive coding comprises motion computation adapted to generate motion information using during the motion-based predictive coding; and

the video encoder has an encoding mode in which at least some of the motion information used during the motion-based predictive coding is excluded from the encoded video bitstream.

8. The invention of claim 7, wherein the video encoder is a scaleable video encoder that can be operated at a plurality of different encoding modes, wherein:

in a first encoding mode, all of the motion information is excluded from the encoded video bitstream and the encoded video bitstream does not explicitly include any motion information; and

in a second encoding mode, at least some of the motion information is encoded into the encoded video bitstream.

9. The invention of claim 8, wherein:

in the second encoding mode, a first portion of the motion information is encoded into the encoded video bitstream and a second portion of the motion information is excluded from the encoded video bitstream; and

in a third encoding mode, all of the motion information is encoded into the encoded video bitstream.

10. The invention of claim 7, wherein:

the first processing path is configured for decoding the encoded first frame/region to generate a decoded first frame/region; and

the second processing path is configured for;

(1) encoding the second original frame/region to generate an encoded second frame/region;

(2) decoding the encoded second frame/region to generate a decoded second frame/region;

(3) performing motion computation between the decoded second frame/region and the decoded first frame/region to generate the motion information;

(4) applying the motion information to the decoded first frame/region to generate a synthesized second frame/region;

(5) performing inter-frame differencing between the synthesized second frame/region and the second original frame/region to generate residual errors; and

(6) encoding, into the encoded video bitstream, at least some of the residual errors.

11. The invention of claim 10, wherein the encoding in the first processing path and the encoding of the second original frame/region in the second processing path are based on intra-frame wavelet encoding.

12. The invention of claim 10, wherein:

the first processing path is configured for intra-frame coding the first original frame/region at a high resolution;

the decoded first frame/region is at the high resolution;

the second processing path is configured for;

(i) spatially sub-sampling the second original image/region to generate a low-resolution second frame/region having a resolution lower than the high resolution; and

(ii) intra-frame coding the low-resolution second frame/region to generate the encoded second frame/region;

the decoded second frame/region is at the low resolution; and

the synthesized second frame/region is at the high resolution.

13. The invention of claim 10, wherein the second processing path is configured for:

(i) thresholding the residual errors to generate binary data; and

(ii) encoding, into the encoded video bitstream, the at least some of the residual errors based on the binary data.

14. The invention of claim 7, further comprising a third processing path configured for encoding, into the encoded video bitstream, a third original frame/region in the video stream using tweening based on the motion information used to encode the second original frame/region.

15. The invention of claim 14, wherein:

the first processing path is configured for decoding the encoded first frame/region to generate a decoded first frame/region; and

the third processing path is configured for;

(1) temporally interpolating the motion information used to encode the second original frame/region;

(2) applying the temporally interpolated motion information to the decoded first frame/region to generate a synthesized third frame/region;

(3) generating residual errors between the synthesized third frame/region and the third original frame/region; and

(4) encoding, into the encoded video bitstream, at least some of the residual errors.

16. The invention of claim 15, wherein:

the first processing path is configured for intra-frame coding the first original frame/region at a high resolution;

the decoded first frame/region is at the high resolution;

the synthesized third frame/region is at the high resolution; and

the third processing path is configured for performing inter-frame differencing between the synthesized third frame/region and the third original frame/region to generate the residual errors.

17. A method for decoding an encoded video bitstream to generate a decoded video stream, comprising the steps of:

(a) decoding, from the encoded video bitstream, an encoded first frame/region using intra-frame decoding to generate a decoded first frame/region; and

(b) decoding, from the encoded video bitstream, an encoded second frame/region using motion-based predictive decoding, wherein at least some motion information used during the motion-based predictive decoding is generated by performing motion computation as part of the decoding method.

18. The invention of claim 17, wherein the encoded video bitstream does not explicitly include any motion information and all of the motion information used during the motion-based predictive decoding is generated as part of the method.

19. The invention of claim 17, wherein step (b) comprises the steps of:

(1) decoding, from the encoded video bitstream, the encoded second frame/region to generate a decoded second frame/region;

(2) performing the motion computation between the decoded second frame/region and the decoded first frame/region to generate the motion information;

(3) applying the motion information to the decoded first frame/region to generate a synthesized second frame/region;

(4) decoding, from the encoded video bitstream, encoded residual errors to generate decoded residual errors corresponding to the synthesized second frame/region; and

(5) performing inter-frame addition between the decoded residual errors and the synthesized second frame/region to generate an error-corrected decoded second frame/region.

20. The invention of claim 17, further comprising the step of:

(c) generating a decoded third frame/region using tweening based on the motion information used to decode the encoded second frame/region.

21. The invention of claim 17, further comprising the step of de-interlacing a decoded second frame/region generated during step (b) to generate two corresponding fields corresponding to the decoded second frame/region.

23. The invention of claim 17, wherein the motion-based predictive decoding further comprises motion compensation for the encoded second frame/region based on one or more motion vectors, wherein at least one of the motion vectors used during the motion compensation is determined during the motion computation.

24. The invention of claim 23, wherein each motion vector used during the motion compensation is determined during the motion computation.

22. A video decoder for decoding an encoded video bitstream to generate a decoded video stream, comprising:

(a) a frame/region type selector configured for selecting different processing paths for decoding different encoded frames/regions from the encoded video bitstream;

(b) a first processing path configured for decoding, from the encoded video bitstream, an encoded first frame/region in the video stream using intra-frame decoding to generate a decoded first frame/region; and

(c) a second processing path configured for decoding, from the encoded video bitstream, an encoded second frame/region in the video stream using motion-based predictive decoding, wherein the video decoder has a decoding mode in which at least some motion information used during the motion-based predictive decoding is generated by the video decoder performing motion computation.

25. The invention of claim 22, wherein the video decoder is a scaleable video decoder that can be operated at a plurality of different decoding modes, wherein:

in a first decoding mode, the encoded video bitstream does not explicitly include any motion information and all of the motion information is generated by performing the motion computation by the video decoder; and

in a second decoding mode, at least some of the motion information is decoded from the encoded video bitstream.

26. The invention of claim 25, wherein:

in the second decoding mode, a first portion of the motion information is decoded from the encoded video bitstream and a second portion of the motion information is generated by performing the motion computation by the video decoder; and

in a third decoding mode, all of the motion information is decoded from the encoded video bitstream.

27. The invention of claim 22, wherein:

the second processing path is configured for;

(1) decoding, from the encoded video bitstream, the encoded second frame/region to generate a decoded second frame/region;

(2) performing the motion computation between the decoded second frame/region and the decoded first frame/region to generate the motion information;

(3) applying the motion information to the decoded first frame/region to generate a synthesized second frame/region;

(4) decoding, from the encoded video bitstream, encoded residual errors to generate decoded residual errors corresponding to the synthesized second frame/region; and

(5) performing inter-frame addition between the decoded residual errors and the synthesized second frame/region to generate an error-corrected decoded second frame/region.

28. The invention of claim 27, wherein the decoding in the first processing path and the decoding of the second encoded frame/region in the second processing path are based on intra-frame wavelet decoding.

29. The invention of claim 27, wherein:

the decoded first frame/region is at a high resolution;

the decoded second frame/region is at a low resolution lower than the high resolution;

the synthesized second frame/region is at the high resolution; and

the error-corrected decoded second frame/region is at the high resolution.

30. The invention of claim 22, further comprising a third processing path configured for generating a decoded third frame/region using tweening based on the motion information used to decode the encoded second frame/region.

31. The invention of claim 30, wherein the third processing path is configured for:

(1) temporally interpolating the motion information used to decode the encoded second frame/region; and

(2) applying the temporally interpolated motion information to the decoded first frame/region to generate the decoded third frame/region.

32. The invention of claim 31, wherein the decoded third frame/region is not explicitly represented in the encoded video bitstream.

33. The invention of claim 31, wherein the third processing path is configured for:

(i) applying the temporally interpolated motion information to the decoded first frame/region to generate a synthesized third frame/region;

(ii) decoding, from the encoded video bitstream, encoded residual errors for an encoded third frame/region to generate decoded residual errors; and

(iii) applying the decoded residual errors to the synthesized third frame/region to generate the decoded third frame/region.

34. The invention of claim 33, wherein:

the decoded first frame/region is at a high resolution;

the synthesized third frame/region is at the high resolution; and

the third processing path is configured for performing inter-frame addition between the synthesized third frame/region and the decoded residual errors to generate the decoded third frame/region at the high resolution.

35. The invention of claim 22, wherein the second processing path is configured for de-interlacing a decoded second frame/region to generate two corresponding fields corresponding to the decoded second frame/region.