Video decoder with bit stream based enhancements

• US 6,466,624 B1
• Filed: 10/13/1999
• Issued: 10/15/2002
• Est. Priority Date: 10/28/1998
• Status: Expired due to Fees

• Current Assignee: Skymoon Ventures
• Original Assignee: Pixonics, LLC
• Inventors: Fogg, Chad Edward
• Primary Examiner(s): RAO, ANAND SHASHIKANT
• Application Number:
  US09/417,989
• Time in Patent Office: 1,098 Days
• Field of Search: 375/240--24
• US Class Current: 375/240.27
• CPC Class Codes: H04N 19/42 characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation H04N19/635 takes precedence
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/521 for estimating the reliability of the determined motion vectors or motion vector field, e.g. for smoothing the motion vector field or for correcting motion vectors
• CPC Class Codes: H04N 19/523 with sub-pixel accuracy
• CPC Class Codes: H04N 19/577 Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression
• CPC Class Codes: H04N 19/86 involving reduction of coding artifacts, e.g. of blockiness
• CPC Class Codes: H04N 19/895 in combination with error concealment

Abstract

An advanced decoder provides enhancement processing in accordance with information previously known only to the decoder and/or encoder. Preferably, advanced digital video image enhancement utilizing bitstream information is provided in an integrated manner with advanced decoding and, more preferably, provides for MPEG decoding and image enhancement. In addition to advanced edge detection and other advanced processing tools, the invention further enables the use of decoder based coding-type operations, such as reverse superresolution, as well as advanced superresolution, and 3-D display processing techniques in conjunction with video processing. The invention is therefore capable of reducing system complexity and/or bandwidth, requirements, and improving system robustness and/or the ultimate quality of the enhanced video signal, among other advantages.

Claims

1. A method for enhanced video reconstruction, comprising:

receiving encoded digital video bitstream data into a bitstream buffer, the bitstream buffer comprising a plurality of buffer portions;

receiving buffered encoded digital video bitstream data into a Bitstream Parser-Processor (BPP);

determining from the buffered encoded digital video bitstream data an initial state of a plurality of bitstream data components within the buffered encoded digital video bitstream data;

deriving a second state of the plurality of bitstream data components, the deriving comprising the BPP interacting with at least one of the plurality of buffer portions, a high-layer state storage element and at least one of a plurality of enhanced bitstream data analysis and decision elements;

receiving the second state of the plurality of bitstream data components and the buffered encoded bitstream data into a block decoder element of the plurality of enhanced bitstream data analysis and decision elements;

decoding the buffered encoded bitstream data employing the received plurality of bitstream data components;

transferring the decoded bitstream data to a frame buffer as a first video frame;

deriving a third state of the plurality of bitstream data components using the plurality of enhanced bitstream data analysis and decision elements; and

employing the third state of the plurality of bitstream data components for reconstructing the first video frame into a second video frame.

2. The method of claim 1, where the plurality of buffer portions comprises an input buffer portion, a current buffer portion and a previous buffer portion.

3. The method of claim 2, where the receiving buffered encoded digital video bitstream data into the BPP comprises receiving the buffered encoded digital video bitstream data from the input buffer portion.

4. The method of claim 2, where the BPP interacting with at least one of the plurality of buffer portions comprises the BPP interacting with the current buffer portion and/or the previous buffer portion.

5. The method of claim 1, where the plurality of enhanced bitstream data analysis and decision elements comprises one or more elements selected from the group consisting of a modified block decoder element, a motion analysis element, a spatial analysis element and a decision unit element.

6. The method of claim 1, where the BPP comprises a parser portion and a bitstream processor portion, the parser portion further comprising a motion vector reconstructor sub-portion and a coefficient block storage sub-portion and the bitstream processor portion further comprising one or more sub-portions selected from the group of sub-portions consisting of a motion vector post-processor sub-portion, a motion tracer sub-portion, a coefficient analyzer sub-portion, a coefficient counter sub-portion, a DC extractor sub-portion and a side into sequencer sub-portion.

7. The method of claim 1, where the BPP comprises a motion vector reconstructor sub-portion and a motion vector post processor sub-portion, the motion vector post processor sub-portion receiving motion vectors from the motion vector reconstructor sub-portion and determining an effect of each of the motion vectors upon sample areas in a plurality of second video frames.

8. The method of claim 1, where the decoding comprises retaining at least one of reconstructed spatial-domain data and DCT frequency-domain data for use in decoding, and outputting at feast one of prediction macroblock data, inverse-DCT result data and inverse quantized coefficient data, in addition to outputting the first video frame.

9. The method of claim 1, where the plurality of enhanced bitstream data analysis and decision elements comprises a motion analysis element, the motion analysis element further comprising a sub-block motion estimator sub-element and a gradient measurer sub-element, the motion analysis element analyzing bitstream data from the BPP and first frame data from the block decoder and/or frame buffer to derive third state data.

10. The method of claim 9, where deriving third state data comprises:

using original bitstream vectors and motion mode and coefficient count data outputted from the BPP with first video frame data from the block decoder to produce temporal filter vectors and optical flow metrics; and

determining gradient measurements from temporal filter vectors and optical flow metrics.

11. The method of claim 9, where deriving third state data comprises determining motion vectors from original bitstream vectors, the motion vectors having a granularity that is finer than that of the original bitstream vectors.

12. The method of claim 9, where deriving third state data comprises performing motion analysis using hierarchical motion estimation.

13. The method of claim 1, where the plurality of enhanced bitstream data analysis and decision elements comprises a spatial analysis element, the spatial analysis element further comprising a mask generator sub-element, a gradient measurer sub-element and an edge detector sub-element, the spatial analysis element analyzing bitstream data from the BPP and first frame data from the block decoder and/or frame buffer to derive third state data.

14. The method of claim 13, where deriving third state data comprises:

using the mask generator sub-element to create occlusion labels;

using the gradient measurer sub-element to measure gradients of surrounding pixels;

using the occlusion labels and measured gradients for edge detection; and

deriving spatial-domain analysis data.

15. The method of claim 13, where deriving third state data comprises:

performing gradient measurements of motion vector and or spatial vector data; and

performing hierarchical motion estimation temporal processing.

16. The method of claim 1, where the plurality of enhanced bitstream data analysis and decision elements comprises a decision unit element, the decision unit element further comprising a parameter control processor sub-element, a quantization Noise estimator sub-element, a residual measurer sub-element an enhancement processor sub-element and a prior frame iteration storage sub-element, the decision unit element analyzing data received from the BPP, the block decoder and/or frame buffer, the spatial analysis element and the motion analysis element to derive data for rendering.

17. The method of claim 16, where deriving data for rendering comprises:

receiving DCT data from the BPP;

receiving vector field discontinuity data the motion analyzer element;

receiving gradient detection from the spatial analyzer element; and

performing stream vector prediction, enhanced prediction and generating final edge metrics.

18. The method of claim 16, where deriving data for rendering comprises merging an inverse point spread function with optical flow measurements.

19. The method of claim 1, where deriving a third state of the plurality of bitstream data components further comprises utilizing 3-D methods for warping new image areas to create interpolated video frame data.

20. The method of claim 1, where deriving a third state of the plurality of bitstream data components further comprises:

processing intertrame dependence between adjacent groups of pictures (GOP) data blocks, where such processing includes interframe filtering; and

applying meta data multiplexed within a plurality of adjacent GOP'"'"'s for processing each of the GOP'"'"'s.