System and method for using temporal gamma and reverse super-resolution to process images for use in digital display systems

• US 6,340,994 B1
• Filed: 03/26/1999
• Issued: 01/22/2002
• Est. Priority Date: 08/12/1998
• Status: Expired due to Term

• Current Assignee: RPX Corporation
• Original Assignee: Pixonics, LLC
• Inventors: Margulis, Neal, Fogg, Chad
• Primary Examiner(s): HSIA, SHERRIE Y
• Application Number:
  US09/277,100
• Time in Patent Office: 1,033 Days
• Field of Search: H04/N.521, H04/N.514, H04/N.964, 348/625, 348/630, 348/571, 348/607, 348/606, 348/620, 348/708, 348/718, 348/719, 348/720, 348/721, 348/674, 348/675
• US Class Current: 348/625
• CPC Class Codes: G09G 2320/0276 for the purpose of adaptation to the characteristics of a display device, i.e. gamma correction
• CPC Class Codes: G09G 2340/0407 Resolution change, inclusive of the use of different resolutions for different screen areas
• CPC Class Codes: G09G 2340/0435 Change or adaptation of the frame rate of the video stream
• CPC Class Codes: G09G 3/20 for presentation of an assembly of a number of characters, e.g. a page, by composing the assembly by combination of individual elements arranged in a matrix no fixed position being assigned to or needed to be assigned to the individual characters or partial characters
• CPC Class Codes: G09G 5/02 characterised by the way in which colour is displayed details of colour display specific for CRTs G09G1/28; specific for flat matrix panels other than liquid crystal displays G09G3/2003; specific for liquid crystal displays G09G3/3607
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/86 involving reduction of coding artifacts, e.g. of blockiness
• CPC Class Codes: H04N 5/14 Picture signal circuitry for video frequency region cameras or camera modules comprising electronic image sensors, or control thereof H04N23/00
• CPC Class Codes: H04N 5/21 Circuitry for suppressing or minimising disturbance, e.g. moiré or halo
• CPC Class Codes: H04N 5/74 Projection arrangements for image reproduction, e.g. using eidophor
• CPC Class Codes: H04N 9/3179 Video signal processing therefor
• CPC Class Codes: H04N 9/3185 Geometric adjustment, e.g. keystone or convergence optical or mechanical adjustment of convergence H04N9/317; using scanning means H04N3/22; optical or mechanical adjustments of projectors not peculiar to the presence of an electronic spatial light modulator G03B21/14
• CPC Class Codes: H04N 9/64 Circuits for processing colour signals H04N9/77 takes precedence; camera processing pipelines for processing colour signals H04N23/84

Abstract

An image processing system including a display output processor using Temporal Gamma Processing (TGP) and Reverse Super-resolution (RSR) techniques to process images. TGP assures that the time-related representation of an image is as accurate as possible, and thus, based on a previous frame value and a known transfer function of the display modulation system, adjusts its output values to provide a desired output value during display of a desired frame. RSR performs a superset of the frame rate conversion process for converting between disparate input and output frame rates. RSR, improving display quality when intended display images have apparent resolution higher than can be supported by an image modulator, sequences lower resolution images at higher frame rates to simulate higher resolution outputs. RSR also spatially filters the lower resolution images into transitional images by shifting the pixel matrix of each image frame such that the weighted filter center of each sequenced frame is constant or such that motion artifacts are not generated. Additionally, RSR, to prevent motion artifacts, enables moving the viewed images at the display refresh screen rate and uses motion-adaptive filters and motion tracking information to track image objects.

Claims

1. An image processing apparatus for processing bitstream information to provide the video stream image data for use in a display system having a display device, comprising:

a databus;

a display input processor (DIP) coupled to said databus, said DIP comprising a data input connector and a first plurality of processing modules configured to receive said bitstream data through said data input connector and generate DIP outputs therefrom;

a display output processor (DOP) coupled to said databus, said DOP comprising a second plurality of processing modules configured to process said DIP outputs and provide DOP outputs therefrom, said second plurality comprising a temporal gamma processing (TGP) module; and

a buffer memory, coupled to said databus, configured for storing said DIP outputs and said DOP outputs and for providing stored DOP outputs as video stream image data to said display device.

2. The apparatus of claim 1 wherein said temporal gamma processing module is configured to use a desired intensity value and a previous frame intensity value to determine an output intensity value for each color component of said DIP inputs.

3. The apparatus of claim 2 further comprising a plurality of look-up tables, the plurality configured such that said desired intensity value determines the depth of said look-up tables and said previous frame intensity value determines the number of look-up tables.

4. The apparatus of claim 3 wherein said temporal gamma processing module is configured to use at least one of said look-up tables for performing color gamma correction.

5. The apparatus of claim 3 where in s aid temporal gamma processing module is configured to use a formula for determining the values of said DOP outputs.

6. An image processing apparatus for processing bitstream information to provide video stream image data for use in a display system having a display device, comprising:

a databus;

a display input processor (DIP) coupled to said databus, said DIP comprising a data input connector and a first plurality of processing modules configured to receive said bitstream data through said data input connector and generate DIP outputs therefrom;

a display output processor (DOP) coupled to said databus, said DOP comprising a second plurality of processing modules configured to process said DIP outputs and provide DOP outputs therefrom, said second plurality comprising a reverse super-resolution (RSR) module; and

a buffer memory, coupled to said databus, configured for storing said DIP outputs and said DOP outputs and for providing stored DOP outputs as the video stream image data to said display device.

7. The apparatus of claim 6 wherein said second plurality of processing modules further comprises a display modulator, said display modulator configured to provide modulated data to an image modulator for forming display images on said display device, said image modulator configured for a specific display device.

8. The apparatus of claim 7 wherein said RSR module is configured to improve display images that have a higher spatial resolution than can be supported by the number of pixels being modulated.

9. The apparatus of claim 8 wherein each pixel of said display modulator comprises a matrix of emitting elements modulating said display images.

10. The apparatus of claim 9 wherein said matrix of emitting elements adjusts the position of image outputs during multiple output frames without reloading pixels.

11. The apparatus of claim 8 wherein said first plurality of processing modules comprises an image reconstruction (IR) module.

12. The apparatus of claim 11 wherein said IR module is configured for providing an output frame having resolution higher than an input frame.

13. The apparatus of claim 12 wherein said IR module is further configured to use an input frame to construct intermediate frames having resolution higher than the input frame and to use motion estimation vectors to correlate the intermediate frames adjacent frames and restoring said input frames from said intermediate frames.

14. The apparatus of claim 13 wherein said IR module is configured to tag said motion estimation vectors with image error terms based on an image matching criteria.

15. The apparatus of claim 14 wherein said IR module is configured to restore said input frames from said intermediate frames using motion estimation vectors having a low value for said image error terms.

16. The apparatus of claim 14 wherein said IR module is configured to use input frames configured as pixel blocks having sub-blocks, and is further configured to track motion of said input frames on a sub-block basis if said motion estimation vectors are tagged with a low value for said error terms.

17. The apparatus of claim 14 wherein said IR module is configured to use motion estimation vectors from an input bitstream.

18. The apparatus of claim 7 wherein the data modulated by said display modulator is pixel data for forming display images on said display device.

19. The apparatus of claim 6 wherein said RSR module is further configured to receive said video frequency or image data as a sequence of input frames having a transfer rate of X frames per second and to RSR filter each input frame of said sequence of input frames and to provide therefrom a sequence of Y number of RSR frames having a transfer rate of Z frames per second where Z=X*Y, and wherein the Y number of RSR frames comprise a start RSR frame and at least one subsequent RSR frame.

20. A method for processing bitstream information to form images for use in a display system having a display device, comprising:

receiving said bitstream information into a display processor (DIP);

processing said received bitstream information to generate DIP outputs;

receiving said DIP outputs into a display output processor (DOP);

processing said DIP outputs with a temporal gamma processing (TGP) module to generate DOP outputs;

directing said DOP outputs to a buffer memory module; and

providing images based on said DOP outputs to said display device.

21. The method of claim 20 wherein processing with said TGP module comprises determining a output intensity value for each color component of said DIP outputs.

22. The method of claim 21, wherein determining said output intensity value comprises using a desired intensity value and a previous frame intensity value to determine said output intensity value for each color component of said DIP outputs, wherein the using comprises looking up a value within a plurality of look-up tables, the depth of said plurality of look-up tables being a function of the desired intensity value and the number of tables with the plurality being a function of the previous frame intensity.

23. The method of claim 22 wherein processing with said TGP module comprises using a formula for determining an output intensity value for each color component of said DIP outputs.

24. The method of claim 21 wherein processing said DIP outputs further comprises processing the outputs with a reverse super-resolution (RSR) module and a display modulator (DM) module.

25. A method for processing bitstream information to for m images for use in a display system having a display device, comprising:

receiving said bitstream information into a display processor (DIP);

processing said received bitstream information to generate DIP outputs;

receiving said DIP outputs into a display output processor (DOP), said DOP comprising a reverse super-resolution (RSR) module and a display modulator (DM) module;

processing said DIP outputs with said reverse super-resolution (RSR) module and said display modulator (DM) module;

directing said DOP outputs to a buffer memory module; and

providing images based on said DOP outputs to said display device.

26. The method of claim 25 wherein processing said DIP outputs with said RSR module comprises sequencing frames of low resolution data having a first transfer rate, input frames, and converting the each input frame into a number of RSR frames having a second transfer rate higher than the first rate, the number of RSR frames representing an RSR image block having a simulated higher resolution.

27. The method of claim 26 wherein the second transfer rate is equal to the product of the first transfer rate and the number of RSR frames.

28. The method of claim 25 wherein processing said DIP outputs with said RSR module to form RSR frames further comprises shifting a pixel matrix that represents each RSR image block by the same pixel amount, where the pixel amount can be either a number of pixels or a pixel fraction.

29. The method of claim 28 wherein shifting said pixel matrix comprises shifting such that a matrix pixel weighted center of each input frame is constant for each of the number of RSR frames formed from the input frame.

30. A method for temporal gamma processing, comprising:

providing a first color intensity value;

providing a second color intensity value;

providing a number of look-up tables, each table comprising a number of adjusted color intensity values, wherein the number of tables, L, is a function of the second color intensity value and the number of adjusted color intensity values, M, is a function of the first color intensity value,;

using the second intensity value to select one of said tables; and

using said first color intensity value to select one of said adjusted color intensity values in said selected table.

31. The method of claim 30 wherein said first color intensity value comprises a desired output frame color intensity value and said second color intensity value comprises a previous frame color intensity value which corresponds to a previous value of said desired output frame color intensity value.

32. The method of claim 30 wherein each of said tables further comprises mapping values.