Spatial/frequency hybrid video coding facilitating the derivatives of variable-resolution images

• US 5,253,056 A
• Filed: 07/02/1992
• Issued: 10/12/1993
• Est. Priority Date: 07/02/1992
• Status: Expired due to Term

• Current Assignee: American Telephone & Telegraph Company (AT&T, Inc.), Shingo Limited Liability Company
• Original Assignee: AT&T, Inc.
• Inventors: Wong, Andria H., Puri, Atul
• Primary Examiner(s): Britton, Howard W.
• Application Number:
  US07/908,135
• Time in Patent Office: 467 Days
• Field of Search: 358/133, 358/135, 358/136
• US Class Current: 375/240.01
• CPC Class Codes: G06T 3/4084 in the transform domain, e.g. fast Fourier transform [FFT] domain scaling
• CPC Class Codes: H04N 19/186 the unit being a colour or a chrominance component
• CPC Class Codes: H04N 19/577 Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/63 using sub-band based transform, e.g. wavelets

Abstract

An adaptive technique for video encoding and decoding which facilitates the transmission, reception, storage, or retrieval of a scalable video signal. The technique allows this scaling to be performed in both the spatial and frequency domains. In a specific embodiment, video encoding is adaptively performed based upon a selection from among a multiplicity of compatible spatial-domain predictions from previously decoded images, and a selection of compatible predictions obtained from up-sampling decoded lower resolution images. The decoded lower resolution images correspond to a current temporal reference, and at least one of these lower resolution images is encoded in more than one frequency domain scales. In yet another embodiment, one lower resolution signal is encoded (without reference to any other low resolution signal) by employing the MPEG-1 standard, with a modified partitioning of the resultant bitstream into two or more frequency scale bitstreams. The decoded images from these bitstreams are employed in the spatial prediction of a higher resolution layers.

Claims

1. Apparatus for encoding digital video signals, comprising:

means for receiving a digital video input signal including a succession of digital representations related to picture elements of a video image;

means for coding a reduced resolution digital signal related to the picture elements of said video image by frequency scaling said received digital video input signal;

means for producing a temporal prediction of said video image from said reduced resolution digital signal;

means for producing a spatial prediction of said first video image based upon said temporal prediction produced from said reduced resolution digital signal; and

means for coding a second digital signal related to the picture elements of said video image, adaptively employing an estimate based upon said temporal or said spatial predictions of said video image.

2. The apparatus of claim 1 wherein said means for coding said second digital signal determines on a block basis if an estimate based upon said temporal prediction of said video image, or an estimate based upon said spatial prediction of said video image will be employed in the encoding of said second digital signal.

3. The apparatus of claim 1 wherein said frequency scaling of said received digital video input signal is performed on an 8×

8 block basis.

4. The apparatus of claim 1 wherein said frequency scaling of said received digital video input signal is performed on an 4×

4 block basis.

5. Method for encoding digital video signals, comprising the steps of:

receiving a digital video input signal including a succession of digital representations related to picture elements of a video image;

coding a reduced resolution digital signal related to the picture elements of said video image by frequency scaling said received digital video input signal;

producing a temporal prediction of said video image from said reduced resolution digital signal;

producing a spatial prediction of said video image based upon said temporal prediction produced from said reduced resolution digital signal; and

coding a second digital signal related to the picture elements of said video image, adaptively employing an estimate based upon said temporal or said spatial predictions of said video image.

6. The method of claim 5 wherein a determination as to employing an estimate based upon said temporal prediction of said video image, or an estimate based upon said spatial prediction of said video image is made on a block basis during the coding of said second digital signal.

7. The method of claim 5 wherein said frequency scaling of said received digital video input signal is performed on an 8×

8 block basis.

8. The method of claim 5 wherein said frequency scaling of said received digital video input signal is performed on an 4×

4 block basis.

9. Apparatus for encoding digital video signals, comprising:

means for receiving a digital video input signal including a succession of digital representations related to picture elements of at least two frames of a video image, each of said frames comprising a plurality of interlaced fields;

means for coding a reduced resolution digital signal related to a field of a received frame by frequency scaling said received digital video input signal;

means for coding an intermediate resolution digital signal related to said field of said received frame by frequency scaling said received digital video input signal;

means for producing a temporal prediction of said field of said received frame from said intermediate resolution digital signal;

means for producing a spatial prediction of said field of said received frame based upon said temporal prediction produced from said intermediate resolution digital signal; and

means for coding a full resolution digital signal related to the picture elements of said field of said received frame, adaptively employing an estimate based upon said temporal or said spatial predictions of said field of said received frame.

10. The apparatus of claim 9 wherein said means for coding said full resolution digital signal determines on a block basis if an estimate based upon said temporal prediction of said field of first video field, or an estimate based upon said spatial prediction of said field of said first video field will be employed in the encoding of said full resolution digital signal.

11. The apparatus of claim 9 wherein said frequency scaling of said received digital video input signal is performed on an 4×

4 block basis to code said reduced resolution digital signal, and wherein said frequency scaling of said received digital video input signal is performed on an 8×

8 block basis to code said intermediate resolution digital signal.

12. Method for encoding digital video signals, comprising the steps of:

receiving a digital video input signal including a succession of digital representations related to picture elements of at least two frames of a video image, each of said frames comprising a plurality of interlaced fields;

coding a reduced resolution digital signal related to a field of a received frame by frequency scaling said received digital video input signal;

coding an intermediate resolution digital signal related to said field of said received frame by frequency scaling said received digital video input signal;

producing a temporal prediction of said field of said received frame from said intermediate resolution digital signal;

producing a spatial prediction of said field of said received frame based upon said temporal prediction produced from said intermediate reduced resolution digital signal; and

coding a full resolution digital signal related to the picture elements of said field of said received frame, adaptively employing an estimate based upon said temporal or said spatial predictions of said field of said received frame.

13. The method of claim 12 wherein a determination as to employing an estimate based upon said temporal prediction of said field of first video field, or an estimate based upon said spatial prediction of said field of said first video field is made on a block basis during the coding of said full resolution digital signal.

14. The method of claim 12 wherein said frequency scaling of said received digital video input signal is performed on an 4×

4 block basis to code said reduced resolution digital signal, and wherein said frequency scaling of said received digital video input signal is performed on an 8×

8 block basis to code said intermediate resolution digital signal.

15. Apparatus for encoding digital video signals, comprising:

means for receiving a digital video input signal including a succession of digital representations related to picture elements of at least two frames of a video image, each of said frames comprising a plurality of interlaced odd and even fields;

means for coding a reduced resolution digital signal related to an odd field of a received frame by frequency scaling said received digital video input signal;

means for coding an intermediate resolution digital signal related to said odd field of said received frame by frequency scaling said received digital video input signal;

means for producing a temporal prediction of said odd field of said received frame from said intermediate resolution digital signal;

means for producing a spatial prediction of said odd field of said received frame based upon said temporal prediction produced from said intermediate resolution digital signal; and

means for coding a full resolution digital signal related to the picture elements of said odd field of said received frame, adaptively employing an estimate based upon said temporal or said spatial predictions of said odd field of said received frame.

16. The apparatus of claim 15 wherein said means for coding said full resolution digital signal determines on a block basis if an estimate based upon said temporal prediction of said odd field of first video field, or an estimate based upon said spatial prediction of said odd field of said first video field will be employed in the encoding of said full resolution digital signal.

17. The apparatus of claim 15 wherein said frequency scaling of said received digital video input signal is performed on a 4×

4 block basis to code said reduced resolution digital signal, and wherein said frequency scaling of said received digital video input signal is performed on an 8×

8 block basis to code said intermediate resolution digital signal.

18. Method for encoding digital video signals, comprising the steps of:

receiving a digital video input signal including a succession of digital representations related to picture elements of at least two frames of a video image, each of said frames comprising a plurality of interlaced odd and even fields;

coding a reduced resolution digital signal related to an odd field of a received frame by frequency scaling said received digital video input signal;

coding an intermediate resolution digital signal related to said odd field of said received frame by frequency scaling said received digital video input signal;

producing a temporal prediction of said odd field of said received frame from said intermediate resolution digital signal;

producing a spatial prediction of said odd field of said received frame based upon said temporal prediction produced from said second reduced resolution digital signal; and

coding a full resolution digital signal related to the picture elements of said odd field of said received frame, adaptively employing an estimate based upon said temporal or said spatial predictions of said odd field of said received frame.

19. The method of claim 18 wherein a determination as to employing an estimate based upon said temporal prediction of said odd field of first video field, or an estimate based upon said spatial prediction of said odd field of said first video field is made on a block basis during the coding of said full resolution digital signal.

20. The method of claim 18 wherein said frequency scaling of said received digital video input signal is performed on an 4×

4 block basis to code said reduced resolution digital signal, and wherein said frequency scaling of said received digital video input signal is performed on an 8×

8 block basis to code said intermediate resolution digital signal.

21. Apparatus for encoding digital video signals, comprising:

means for receiving a digital video input signal including a succession of digital representation related to picture elements of at least two frames of a video image, each of said frames comprising a plurality of interlaced odd and even fields;

means for coding a reduced resolution digital signal related to an odd field of a received frame by frequency scaling said received digital video input signal;

means for coding an intermediate resolution digital signal related to said odd field of said received frame by frequency scaling said received digital video input signal;

means for producing a temporal prediction of said odd field of said received frame from said intermediate resolution digital signal;

means for producing a spatial prediction of said odd field of said received frame based upon said temporal prediction produced from said intermediate resolution digital signal;

means for coding a full resolution digital signal related to the picture elements of said odd field of said received frame, adaptively employing an estimate based upon said temporal or said spatial predictions of said odd field of said received frame;

means for coding a reduced resolution digital signal related to the picture elements of an even field of said received frame, employing, if said received frame is not the initial frame received, a prediction of said even field of said received frame based upon a previously coded even field from a previously received frame;

means for producing a temporal prediction of said even field of said received frame from said reduced resolution digital signal related to said even field of said received frame; and

means for coding a full-resolution digital signal related to the picture elements of said even field of said received frame, adaptively employing an estimate based upon said temporal predictions of said even field of said received frame.

22. The apparatus of claim 21 wherein said means for coding said full resolution digital signal related to the picture elements of said odd field of said received frame determines on a block basis if an estimate based upon said temporal prediction of said odd field of first video field, or an estimate based upon said spatial prediction of said odd field of said first video field will be employed in the encoding of full resolution digital signal related to the picture elements of said odd field of said received frame.

23. The apparatus of claim 21 wherein said frequency scaling of said received digital video input signal is performed on an 4×

4 block basis to code said reduced resolution digital signal related to said odd field of said received frame, and wherein said frequency scaling of said received digital video input signal is performed on an 8×

8 block basis to code said intermediate resolution digital signal.

24. Method for encoding digital video signals, comprising the steps of:

receiving a digital video input signal including a succession of digital representations related to picture elements of at least two frames of a video image, each of said frames comprising a plurality of interlaced odd and even fields;

coding a reduced resolution digital signal related to an odd field of a received frame by frequency scaling said received digital video input signal;

coding an intermediate resolution digital signal related to said odd field of said received frame by frequency scaling said received digital video input signal;

producing a temporal prediction of said odd field of said received frame from said intermediate resolution digital signal;

producing a spatial prediction of said odd field of said received frame based upon said temporal prediction produced from said intermediate resolution digital signal;

coding a full resolution digital signal related to the picture elements of said odd field of said received frame, adaptively employing an estimate based upon said temporal or said spatial predictions of said odd field of said received frame;

coding a reduced resolution digital signal related to the picture elements of an even field of said received frame, employing, if said received frame is not the initial frame received, a prediction of said even field of said received frame based upon a previously coded even field from a previously received frame;

producing a temporal prediction of said even field of said received frame from said reduced resolution digital signal related to said even field of said received frame; and

coding a full-resolution digital signal related to the picture elements of said even field of said received frame, adaptively employing an estimate based upon said temporal predictions of said even field of said received frame.

25. The method of claim 24 wherein a determination as to employing an estimate based upon said temporal prediction of said odd field of first video field, or an estimate based upon said spatial prediction of said odd field of said first video field is made on a block basis during the coding of said full resolution digital signal related to the picture elements of said odd field of said received frame.

26. The method of claim 24 wherein said frequency scaling of said received digital video input signal is performed on an 4×

4 block basis to code said reduced resolution digital signal related to said odd field of said received frame, and wherein said frequency scaling of said received digital video input signal is performed on an 8×

8 block basis to code said intermediate resolution digital signal.

27. Apparatus for decoding digital video signals, comprising:

means for receiving a first and second digital signals, said first digital signal related to a reduced resolution representation of a video image, and said second digital signal related to a full resolution representation of said video image;

means for decoding from said first digital signal said reduced resolution video image, employing in said decoding, if said decoded reduced resolution image is not the initial image to be decoded by said apparatus, a prediction of said reduced resolution video image based upon a previously decoded reduced resolution image from a previous video image;

means for producing a temporal prediction of said video image from said decoded reduced resolution field;

means for producing a spatial prediction of said video image based upon said reduced resolution video field; and

means for decoding from said second digital signal said full resolution video image, employing an estimate based upon said temporal or said spatial predictions of said video image.

28. The apparatus of claim 27 wherein said means for decoding said second digital signal determines on a block basis if an estimate based upon said temporal prediction of said video image, or an estimate based upon said spatial prediction of said video image will be employed in the decoding of said second digital signal.

29. The apparatus of claim 27 wherein said first digital signal is frequency scaled on an 8×

8 block basis.

30. The apparatus of claim 27 wherein said first digital signal is frequency scaled on an 4×

4 block basis.

31. Method for decoding digital video signals, comprising the steps of:

receiving a first and second digital signals, said first digital signal related to a reduced resolution representation of a video image, and said second digital signal related to a full resolution representation of said video image;

decoding from said first digital signal said reduced resolution video image, employing in said decoding, if said decoded reduced resolution image is not the initial image to be decoded by said apparatus, a prediction of said reduced resolution video image based upon a previously decoded reduced resolution image from a previous video image;

producing a temporal prediction of said video image from said decoded reduced resolution field;

producing a spatial prediction of said video image based upon said reduced resolution video field; and

decoding from said second digital signal said full resolution video image, employing an estimate based upon said temporal or said spatial predictions of said video image.

32. The method of claim 31 wherein a determination as to employing an estimate based upon said temporal prediction of said video image, or an estimate based upon said spatial prediction of said video image is made on a block basis during the decoding of said second digital signal.

33. The method of claim 31 wherein said first digital signal is frequency scaled on an 8×

8 block basis.

34. The method of claim 31 wherein said first digital signal is frequency scaled on an 4×

4 block basis.

35. Apparatus for decoding digital video signals, comprising:

means for receiving a first, second and third digital signals, said first digital signal representing a reduced resolution field of a first frame of video, said second digital signal representing an intermediate resolution field of a first frame of video, and said third digital signal representing a full resolution field of said first frame of video;

means for decoding from said first digital signal said reduced resolution video field, employing in said decoding, if said decoded reduced resolution field is not the initial field to be decoded by said apparatus, a prediction of said reduced resolution field based upon a previously decoded reduced resolution field from a previous frame;

means for decoding from said second digital signal said intermediate resolution video field, employing in said decoding, if said decoded intermediate resolution field is not the initial field to be decoded by said apparatus, a prediction of said intermediate resolution field based upon a previously decoded field from a previous frame;

means for producing a temporal prediction of said first field from said decoded intermediate resolution field;

means for producing a spatial prediction of said received first video field based upon said decoded intermediate resolution video field; and

means for decoding from said third digital signal said full resolution video field, employing an estimate based upon said temporal or said spatial predictions of said first video field.

36. The apparatus of claim 35 wherein said means for decoding said third digital signal determines on a block basis if an estimate based upon said temporal prediction of said first video field, or an estimate based upon said spatial prediction of said first video field will be employed in the decoding of said third digital signal.

37. The apparatus of claim 35 wherein said first digital signal is frequency scaled on an 4×

4 block basis, and wherein said second digital video input signal is frequency scaled on an 8×

8 block basis.

38. Method for decoding digital video signals, comprising the steps of:

receiving a first, second and third digital signals, said first digital signal representing a reduced resolution field of a first frame of video, said second digital signal representing an intermediate resolution field of a first frame of video, and said third digital signal representing a full resolution field of said frame of video;

decoding from said first digital signal said reduced resolution video field, employing in said decoding, if said decoded reduced resolution field is not the initial field to be decoded by said apparatus, a prediction of said reduced resolution field based upon a previously decoded reduced resolution field from a previous frame;

decoding from said second digital signal said intermediate resolution video field, employing in said decoding, if said decoded intermediate resolution field is not the initial field to be decoded by said apparatus, a prediction of said intermediate resolution field based upon a previously decoded field from a previous frame;

producing a temporal prediction of said first field from said decoded intermediate resolution field;

producing a spatial prediction of said received first video field based upon said decoded intermediate resolution video field; and

decoding from said third digital signal said full resolution video field, employing an estimate based upon said temporal or said spatial predictions of said first video field.

39. The method of claim 38 wherein a determination as to employing an estimate based upon said temporal prediction of said first video field, or an estimate based upon said spatial prediction of said first video field is made on a block basis during the decoding of said third digital signal.

40. The method of claim 38 wherein said first digital signal is frequency scaled on an 4×

4 block basis, and wherein said second digital video input signal is frequency scaled on an 8×

8 block basis.

41. Apparatus for decoding digital video signals, comprising:

means for receiving a first, second and third digital signals, said first digital signal representing a reduced resolution field of an odd field of a first frame of video, said second digital signal representing an intermediate resolution field of said odd field of said first frame of video, and said third digital signal representing a full resolution odd field of said first frame of video;

means for decoding from said first digital signal said reduced resolution odd field, employing in said decoding, if said decoded reduced resolution odd field is not the initial odd field to be decoded by said apparatus, a prediction of said reduced resolution odd field based upon a previously decoded reduced resolution odd field from a previous frame;

means for decoding from said second digital signal said intermediate resolution odd field, employing in said decoding, if said decoded intermediate resolution odd field is not the initial odd field to be decoded by said apparatus, a prediction of said intermediate resolution odd field based upon a previously decoded odd field from a previous frame;

means for producing a temporal prediction of said odd field of said first frame from said decoded intermediate resolution odd field;

means for producing a spatial prediction of said odd field of said first frame based upon said decoded intermediate resolution odd field; and

means for decoding from said third digital signal said full resolution odd field, employing an estimate based upon said temporal or said spatial predictions of said odd field of said first frame.

42. The apparatus of claim 41 wherein said means for decoding said third digital signal determines on a block basis if an estimate based upon said temporal prediction of said odd field of said first frame, or an estimate based upon said spatial prediction of said odd field of said first frame will be employed in the decoding of said third digital signal.

43. The apparatus of claim 41 wherein said first digital signal is frequency scaled on an 4×

4 block basis, and wherein said second digital video input signal is frequency scaled on an 8×

8 block basis.

44. Method for decoding digital video signals, comprising the steps of:

receiving a first, second and third digital signals, said first digital signal representing a reduced resolution field of an odd field of a first frame of video, said second digital signal representing an intermediate resolution field of said odd field of said first frame of video, and said third digital signal representing a full resolution odd field of said first frame of video;

decoding from said first digital signal said reduced resolution odd field, employing in said decoding, if said decoded reduced resolution odd field is not the initial odd field to be decoded by said apparatus, a prediction of said reduced resolution odd field based upon a previously decoded reduced resolution odd field from a previous frame;

decoding from said second digital signal said intermediate resolution odd field, employing in said decoding, if said decoded intermediate resolution odd field is not the initial odd field to be decoded by said apparatus, a prediction of said intermediate resolution odd field based upon a previously decoded odd field from a previous frame;

producing a temporal prediction of said odd field of said first frame from said decoded intermediate resolution odd field;

producing a spatial prediction of said odd field of said first frame based upon said decoded intermediate resolution odd field; and

decoding from said third digital signal said full resolution odd field, employing an estimate based upon said temporal or said spatial predictions of said odd field of said first frame.

45. The method of claim 44 wherein a determination as to employing an estimate based upon said temporal prediction of said odd field of said first frame, or an estimate based upon said spatial prediction of said odd field of said first frame is made on a block basis during the decoding of said third digital signal.

46. The method of claim 44 wherein said first digital signal is frequency scaled on an 4×

4 block basis, and wherein said second digital video input signal is frequency scaled on an 8×

8 block basis.

47. Apparatus for decoding digital video signals, comprising:

means for receiving a first, second, third, fourth and fifth digital signals, said first digital signal representing a reduced resolution field of an odd field of a first frame of video, said second digital signal representing an intermediate resolution field of said odd field of said first frame of video, said third digital signal representing a full resolution odd field of said first frame of video, said fourth digital signal representing a reduced resolution field of an even field of a first frame of video, and said fifth digital signal representing a full resolution even field of said first frame of video;

means for decoding from said first digital signal said reduced resolution odd field, employing in said decoding, if said decoded reduced resolution odd field is not the initial odd field to be decoded by said apparatus, a prediction of said reduced resolution odd field based upon a previously decoded reduced resolution odd field from a previous frame;

means for decoding from said second digital signal said intermediate resolution odd field, employing in said decoding, if said decoded intermediate resolution odd field is not the initial odd field to be decoded by said apparatus, a prediction of said intermediate resolution odd field based upon a previously decoded odd field from a previous frame;

means for producing a temporal prediction of said odd field of said first frame from said decoded intermediate resolution odd field;

means for producing a spatial prediction of said odd field of said first frame based upon said decoded intermediate resolution odd field;

means for decoding from said third digital signal said full resolution odd field, employing an estimate based upon said temporal or said spatial predictions of said odd field of said first frame;

means for decoding from said fourth digital signal said reduced resolution even field, employing in said decoding, if said decoded reduced resolution even field is not the initial even field to be decoded by said apparatus, a prediction of said reduced resolution even field based upon a previously decoded even field from a previous frame;

means for producing a temporal prediction of said even field of said first frame from said decoded reduced resolution even field; and

means for decoding from said fifth digital signal said full resolution even field, employing an estimate based upon said temporal prediction of said even field of said first frame.

48. The apparatus of claim 47 wherein said means for decoding said third digital signal determines on a block basis if an estimate based upon said temporal prediction of said odd field of said first frame, or an estimate based upon said spatial prediction of said odd field of said first frame will be employed in the decoding of said third digital signal.

49. The apparatus of claim 47 wherein said first digital signal is frequency scaled on an 4×

4 block basis, and wherein said second digital video input signal is frequency scaled on an 8×

8 block basis.

50. Method for decoding digital video signals, comprising the steps of:

receiving a first, second, third, fourth and fifth digital signals, said first digital signal representing a reduced resolution field of an odd field of a first frame of video, said second digital signal representing an intermediate resolution field of said odd field of said first frame of video, said third digital signal representing a full resolution odd field of said first frame of video, said fourth digital signal representing a reduced resolution field of an even field of a first frame of video, and said fifth digital signal representing a full resolution even field of said first frame of video;

decoding from said first digital signal said reduced resolution odd field, employing in said decoding, if said decoded reduced resolution odd field is not the initial odd field to be decoded by said apparatus, a prediction of said reduced resolution odd field based upon a previously decoded reduced resolution odd field from a previous frame;

decoding from said second digital signal said intermediate resolution odd field, employing in said decoding, if said decoded intermediate resolution odd field is not the initial odd field to be decoded by said apparatus, a prediction of said intermediate resolution odd field based upon a previously decoded odd field from a previous frame;

producing a temporal prediction of said odd field of said first frame from said decoded intermediate resolution odd field;

producing a spatial prediction of said odd field of said first frame based upon said decoded intermediate resolution odd field;

decoding from said third digital signal said full resolution odd field, employing an estimate based upon said temporal or said spatial predictions of said odd field of said first frame;

decoding from said fourth digital signal said reduced resolution even field, employing in said decoding, if said decoded reduced resolution even field is not the initial even field to be decoded by said apparatus, a prediction of said reduced resolution even field based upon a previously decoded even field from a previous frame;

producing a temporal prediction of said even field of said first frame from said decoded reduced resolution even field; and

decoding from said fifth digital signal said full resolution even field, employing an estimate based upon said temporal prediction of said even field of said first frame.

51. The method of claim 50 wherein a determination as to employing an estimate based upon said temporal prediction of said odd field of said first frame, or an estimate based upon said spatial prediction of said odd field of said first frame is made on a block basis during the decoding of said third digital signal.

52. The method of claim 50 wherein said first digital signal is frequency scaled on an 4×

4 block basis, and wherein said second digital video input signal is frequency scaled on an 8×

8 block basis.