Image processing using linear light values and other image processing improvements

• US 20050063586A1
• Filed: 11/12/2004
• Published: 03/24/2005
• Est. Priority Date: 08/01/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Evans, Glenn F., Spears, Stacey L., Munsil, Donald J.
• Granted Patent: US 7,158,668 B2
• Time in Patent Office: Days
• US Class Current: 382/162
• CPC Class Codes: H04N 1/4053 with threshold modulated relative to input image data or vice versa

Abstract

Strategies are described for processing image information in a linear form to reduce the amount of artifacts (compared to processing the data in nonlinear form). Exemplary types of processing operations can include, scaling, compositing, alpha-blending, edge detection, and so forth. In a more specific implementation, strategies are described for processing image information that is: a) linear; b) in the RGB color space; c) high precision (e.g., provided by floating point representation); d) progressive; and e) full channel. Other improvements provide strategies for: a) processing image information in a pseudo-linear space to improve processing speed; b) implementing an improved error dispersion technique; c) dynamically calculating and applying filter kernels; d) producing pipeline code in an optimal manner; and e) implementing various processing tasks using novel pixel shader techniques.

Claims

1. A method for processing image information in a linear form, comprising:

receiving image information in an interlaced nonlinear form;

converting the image information into a linear form by applying a transfer function to the image information;

converting the image information into a progressive form by de-interlacing the image information; and

performing processing on the image information in the progressive linear form.

2. The method of claim 1, wherein the received image information is in a luma-related color space.

3. The method of claim 2, further comprising, prior to the converting of the image information into the linear form, converting the image information into a nonlinear R′

G′

B′

color space by applying a conversion matrix to the image information.

4. The method of claim 3, wherein the converting of the image information into linear form comprises applying the transfer function to convert the image information from the R′

G′

B′

color space to a linear RGB color space.

5. The method of claim 1, wherein the processing comprises resizing of the image information.

6. The method of claim 1, wherein the processing comprises performing a compositing operation on the image information.

7. The method of claim 1, wherein the processing comprises performing alpha-blending on the image information.

8. The method of claim 1, wherein the processing comprises performing edge detection on the image information.

9. The method of claim 1, wherein the received image information is in a luma-related color space, and wherein the received image information has fewer chroma samples relative to luma samples.

10. The method of claim 9, further comprising up-sampling the received image information to increase the number of chroma samples relative to the number of luma samples.

11. The method of claim 10, wherein the up-sampling produces full channel 4:

4;

4 image information.

12. The method of claim 10, wherein the received image information is in the 4:

2;

0 format, and wherein analysis performed by the de-interlacing and the up-sampling are integrated together.

13. The method of claim 1, wherein the received image information is converted into a high precession format.

14. The method of claim 13, wherein the high precision format comprises a floating point format.

15. The method of claim 14, wherein the floating point format comprises a 16 bit floating point format.

16. The method of claim 1, further comprising quantizing and dithering the image information after it is processed to reduce its precision, to thereby provide quantized image information.

17. The method of claim 16, wherein the quantizing and dithering apply an error dispersion algorithm to reduce the amount of artifacts in the quantized image information, wherein the error dispersion algorithm distributes an error term to neighboring picture elements with respect to a picture element that is undergoing processing.

18. The method of claim 17, wherein the error dispersion algorithm is based on a Floyd-Steinberg algorithm.

19. The method of claim 17, wherein noise is added to the quantizing performed by the error dispersion algorithm, such that a value that is quantized is based on an original image value, an error term value, and a noise value, to yield a quantized value.

20. The method of claim 19, wherein an error term for a subsequent quantization operation is computed by finding the difference between the original image value and the quantized value.

21. The method of claim 19, wherein the noise that is added to the quantizing is generated by a noise generator having a repeat period that is sufficiently long so that artifacts do not appear in the quantized image information.

22. The method of claim 1, wherein the linear form comprises a pseudo-linear form.

23. The method of claim 22, wherein the transfer function that produces the pseudo-linear form applies a multiply operation.

24. The method of claim 22, wherein the transfer function that produces the pseudo-linear form has a linear portion in a region of input values near zero.

25. The method of claim 22, wherein the processing involves applying linear-type algorithms to the image information in the pseudo-linear form.

26. The method of claim 22, further comprising performing an operation to remove the pseudo-linear form of the image information subsequent to the processing.

27. The method of claim 26, further comprising, subsequent to the removing of the pseudo-linear form, converting the image information into a nonlinear form that is appropriate for prescribed output requirements.

28. The method of claim 27, wherein the removing of the pseudo-linear image form and the converting to a nonlinear form appropriate to output requirements is combined into a single operation.

29. The method of claim 1, wherein the method involves at least one scaling operation.

30. The method of claim 29, wherein the scaling operation employs a Catmull-Rom filter.

31. The method of claim 29, wherein the scaling operation involves pre-calculating a number of filter kernels, and then applying the pre-calculated filter kernels to process the image information.

32. The method of claim 31, wherein the pre-calculation also involves calculating a number of filter kernels that are required, and determining how many taps each kernel should have.

33. The method of claim 31, wherein the applying of the filter kernels involves cyclically repeating a series of the pre-calculated filter kernels when processing a row or column of the image information.

34. The method of claim 1, wherein the method defines an image processing pipeline, and wherein an optimized code program is used to implement the image processing pipeline.

35. The method of claim 34, wherein the optimized code is produced by:

inputting requirements of the image processing pipeline; and

assembling code modules from a library of code modules that can be used to implement the requirements, wherein code modules that are not required are not used in the optimized code.

36. The method of claim 1, wherein at least one operation in the method is performed using a graphical processing unit.

37. The method of claim 36, wherein all of the operations in the method are performed using the graphical processing unit.

38. The method of claim 36, wherein the graphical processing unit includes a pixel shader.

39. The method of claim 38, further comprising performing a scaling operation using the pixel shader by:

assigning weighting information to at least one texture unit;

assigning delta-shifted versions of input image information to plural other texture units; and

performing an operation using the pixel shader based on information received from the texture units, to provide an output result.

40. The method of claim 39, further comprising performing plural operations using the pixel shader to generate a final output result on the basis of plural partial output results.

41. The method of claim 40, wherein the performing of plural operations is applied to a case in which the scaling operation uses a filter having a number of taps which exceeds a number of available texture units.

42. An image processing pipeline configured to implement the method of claim 1.

43. One or more computer readable media storing machine-readable instructions configured to implement the method of claim 1.

44. A method for processing image information in a linear form, comprising:

receiving image information in a nonlinear form;

converting the image information into a high precision linear RGB format; and

performing processing on the image information in the high precision linear RGB format.

45. An image processing pipeline configured to implement the method of claim 44.

46. One or more computer readable media storing machine-readable instructions configured to implement the method of claim 44.

47. A method for quantizing and dithering original image information to produce quantized image information, comprising:

quantizing a sum that combines an original value taken from the image information, a noise value, and an error term, to produce a quantized value; and

calculating an error term for a subsequent quantizing operation by computing a difference between the quantized value and the original value.

48. An image processing pipeline configured to implement the method of claim 47.

49. One or more computer readable media storing machine-readable instructions configured to implement the method of claim 47.

50. A method for processing image information in pseudo-linear form, comprising:

receiving image information in a nonlinear form;

converting the image information into a pseudo-linear form using a transfer function; and

performing processing on the image information in the pseudo-linear form using at least one algorithm that is tailored to process linear information.

51. An image processing pipeline configured to implement the method of claim 50.

52. One or more computer readable media storing machine-readable instructions configured to implement the method of claim 50.

53. A method for performing a scaling operation on image information using a filter by:

computing, based on scaling to be performed, a number of filter kernels required to implement the filter, and the number of taps required for each filter kernel;

pre-computing and storing the number of filter kernels;

applying the filter kernels to a row or column of the image information by repeatedly sequencing through the filter kernels if there are more pixel elements in the row or column than there are pre-calculated filter kernels.

54. An image processing pipeline configured to implement the method of claim 53.

55. One or more computer readable media storing machine-readable instructions configured to implement the method of claim 53.

56. A method for generating code used to implement an image processing pipeline, comprising:

inputting requirements of the image processing pipeline; and

assembling code modules from a library of code modules that can be used to implement the requirements, wherein code modules that are not required are not used in the generated code.

57. A method of claim 56, wherein the assembling of code modules uses static analysis to generate the code used to implement the image processing pipeline, in advance of application of the image processing pipeline.

58. An apparatus configured to implement the method of claim 56.

59. One or more computer readable media storing machine-readable instructions configured to implement the method of claim 56.

60. A method for performing at least one processing task in an image processing pipeline using a pixel shader of a graphical processing unit, comprising:

assigning weighting information to at least one texture unit of the pixel shader;

assigning delta-shifted versions of input image information to plural other texture units of the pixel shader; and

performing an operation using the pixel shader based on the information from the texture units, to thereby provide an output result.

61. An image processing pipeline configured to implement the method of claim 60.

62. One or more computer readable media storing machine-readable instructions configured to implement the method of claim 60.