System and method for using bitstream information to process images for use in digital display systems

• US 6,157,396 A
• Filed: 02/16/1999
• Issued: 12/05/2000
• Est. Priority Date: 02/16/1999
• Status: Expired due to Fees

• Current Assignee: Skymoon Ventures
• Original Assignee: Pixonics, LLC
• Inventors: Fogg, Chad, Margulis, Neal
• Primary Examiner(s): Tung, Kee M.
• Application Number:
  US09/250,424
• Time in Patent Office: 658 Days
• Field of Search: 345/501, 345/502, 345/506, 345/507-509, 345/512, 345/202, 348/625, 348/630, 348/631
• US Class Current: 345/506
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution

Abstract

An image processing system using bitstream information for enhancing visual display quality. The processing system retains I frames and some P or B frames from a standard MPEG-2 bitstream, and stores the information across multiple Groups of Pictures (GOPs). The system uses prediction block information in conjunction with detected object edges and bitstream vector information to analyze object optical flow and to reconstruct high quality image frames. The system can also utilize supplemental bitstream information such as layer coding, instructional cues, and image key meta data to further improve the quality of the image frames and for special system functions. When an input image does not include MPEG-2 motion vector bitstream information, the system uses techniques such as optical flow, block matching, or Pel-recursion to estimate motion that tracks the image object motion.

Claims

1. An image processing apparatus for processing bitstream information to provide video stream image data to a display device, comprising:

a display input processor (DIP) coupled to a databus, said DIP comprising,

an input data connector and a first plurality of processing modules,

said first plurality including a digital input control module, a compressed input control module and an image reconstruction module, wherein at least one of said first plurality of processing modules is coupled to said input data connector, and

wherein said first plurality provides for receiving, reconstructing and processing said bitstream information to provide first outputs to said databus;

a display output processor (DOP) coupled to said databus, said DOP comprising,

a second plurality of processing modules,

including a geometric image transformation (GIT) module wherein said second plurality provides for receiving and processing said first outputs to provide second outputs to said databus; and

a buffer memory coupled to said databus, wherein said buffer memory stores said first and second outputs and provides said video stream image data to said display device.

2. The apparatus of claim 1 wherein said DOP comprises a display memory map.

3. The apparatus of claim 1 wherein said second plurality of modules comprises a post GIT filtering module.

4. The apparatus of claim 1 wherein said second plurality of modules comprises a display modulator module.

5. The apparatus of claim 1 wherein said compressed input control module comprises a decompression engine and a bitstream engine.

6. The apparatus of claim 1 wherein said image reconstruction module comprises a motion estimator.

7. The apparatus of claim 6 wherein said image reconstruction module produces image motion information for use by said DOP.

8. The apparatus of claim 1 wherein said bitstream engine tracks block information for use by said DOP.

9. The apparatus of claim 1 comprising an image modulator coupled to said DOP.

10. A method for providing video stream image data to a display device, comprising:

receiving bitstream information into a display input processor (DIP);

directing said bitstream information to a compressed input control module within said DIP;

processing said bitstream information with a decompression engine to create decompressed bitstream data that retains at least some frame information from the compressed bitstream;

processing said decompressed bitstream information with a bitstream engine using the retained frame information to create enhanced video frame information; and

directing said enhanced video frame information to a display output processor (DOP), said DOP for providing video stream image data to said display device.

11. The method of claim 10 comprising using said compressed input control module for de-multiplexing said bitstream information into audio, video and system data streams prior to the decompressing.

12. The method of claim 11 comprising directing said audio stream to an audio subsystem for decoding and playback.

13. The method of claim 10 where creating decompressed bitstream data comprises retaining I-frame information and at least some B-frame and P-frame information from multiple Groups of Pictures (GOPs) within said compressed bitstream information.

14. The method of claim 10 where creating enhanced video stream information with said bitstream engine comprises:

preserving Discrete Cosine Transform (DCT) frequency domain information and spatial domain information across multiple input frames; and

using said preserved DCT and spatial domain information to track and recognize video image flow.

15. The method of claim 10 where creating enhanced video stream information with said bitstream engine comprises:

constructing a post decompression filter map; and

using said post-decompression filter map for applying sharpening filters to non-boundary data and smoothing filters to block boundary data.

16. The method of claim 10 comprising:

retrieving supplemental information from the bitstream information using an enhanced decoder; and

using said retrieved supplemental information for tracking image flow and estimating object motion over a period of time.

17. The method of claim 16 wherein said DOP providing video stream image data to said display device comprises using the flow of video data information for producing output frames having higher resolution than each output frame'"'"'s respective input frame.

18. The method of claim 16 wherein retrieving supplemental information using said enhanced decoder comprises retrieving an image key meta data stream.

19. The method of claim 18 comprising recording image key information within said enhanced video stream for output to said DOP.

20. The method of claim 19 comprising employing said image key information to replace a first object within said enhanced video stream with a second object.

21. The method of claim 20 wherein tracking a flow of video data comprises:

tracking the prediction blocks such that a temporal path of the prediction blocks delineates a coarse trajectory of moving objects; and

processing the coarse trajectory with the bitstream engine to create a motion vector.

22. The method of claim 10 wherein processing said decompressed bitstream information with a bitstream engine comprises:

retaining image block information;

using block-based prediction techniques to compare a first frame to a second frame; and

tracking a flow of video data within said decompressed bitstream information as prescribed by the prediction blocks.

23. A method for generating video stream image data directed to a display device, comprising:

receiving bitstream information into a display input processor (DIP);

processing said bitstream information to output enhanced video frame information;

directing the enhanced video frame information to a display output processor (DOP);

processing the enhanced video frame information with a geometric transformation (GT) module to generate compensated digital image data;

filtering the compensated image data with a post GT filtering module to produce DOP output data; and

providing the DOP output data to said display device.

24. The method of claim 23 wherein providing the DOP output data to said display system comprises directing the output data to an image modulator intermediate said DOP and said display device.

25. The method of claim 24 comprising storing data corresponding to said image modulator that is characteristic of said display device at chosen pixel or screen locations.

26. The method of claim 24 wherein generating the compensated digital image data comprises providing the compensated digital image data to a frame buffer memory.

27. The method of claim 26 wherein filtering the compensated digital image data comprises retrieving the data from said frame buffer memory.

28. The method of claim 24 wherein generating the compensated digital image data comprises adjusting neighboring pixel values.

29. The method of claim 24 wherein filtering the compensated digital image data comprises retrieving the data from said frame buffer memory.

30. A method for generating enhanced video stream images, comprising:

receiving compressed bitstream information into a display input processor (DIP);

directing said bitstream information to a compressed input control module within said DIP;

processing said bitstream information into a decompressed bitstream and a supplemental data bitstream;

creating enhanced video stream information using the decompressed bitstream and the supplemental information;

providing the enhanced video stream information to a display output processor (DOP);

processing the enhanced video frame information with a geometric transformation (GT) module to generate compensated digital image data;

filtering the compensated image data with a post GT filtering module to produce DOP output data; and

providing the DOP output data to said display device whereby enhanced video stream images are displayed.