Enhancing dynamic ranges of images

• US 8,233,738 B2
• Filed: 07/30/2008
• Issued: 07/31/2012
• Est. Priority Date: 07/30/2007
• Status: Active Grant

• Current Assignee: Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Original Assignee: Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Inventors: Heidrich, Wolfgang, Seetzen, Helge, Ward, Gregory John, Whitehead, Lorne A., Rempel, Allan
• Primary Examiner(s): Patel, Kanjibhai
• Application Number:
  US12/183,033
• Publication Number:
  US 20090034868A1
• Time in Patent Office: 1,462 Days
• Field of Search: 382/169, 382/173, 382/232, 382260-270, 382274-275, 382/283, 382/299, 382/254, 345/63, 345/611, 345/626, 345/660, 345/687, 345/690, 348/235, 348/241, 348/252, 348/533, 353/31, 375/240.19, 375/254
• US Class Current: 382/264
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 5/90 Dynamic range modification of images or parts thereof
• CPC Class Codes: G09G 2320/0646 Modulation of illumination source brightness and image signal correlated to each other
• CPC Class Codes: G09G 2320/066 for control of contrast
• CPC Class Codes: G09G 3/3426 the different display panel areas being distributed in two dimensions, e.g. matrix
• CPC Class Codes: G09G 5/10 Intensity circuits
• CPC Class Codes: H04N 5/20 Circuitry for controlling amplitude response
• CPC Class Codes: H04N 9/3126 for spatial light modulators in series

Abstract

Methods and apparatus according to various aspects take as input image data in a lower-dynamic-range (LDR) format and produce as output enhanced image data having a dynamic range greater than that of the input image data (i.e. higher-dynamic range (HDR) image data). In some embodiments, the methods are applied to video data and are performed in real-time (i.e. processing of video frames to enhance the dynamic range of the video frames is completed at least on average at the frame rate of the video signal).

Claims

1. A method for enhancing lower-dynamic range (LDR) image data representing an image, the method comprising:

identifying at least one enhancement region within the image;

generating a brightness enhancement function, wherein generating the brightness enhancement function comprises;

generating a mask which identifies pixels inside and outside of the enhancement region;

applying a blur filter to the mask to obtain grey-scale image data;

applying the brightness enhancement function to the LDR image data to generate higher-dynamic range (HDR) image data, wherein compared to the LDR image data, the HDR image data comprises increased luminance values of enhancement-region pixels in the enhancement region and increased luminance values of one or more boundary-region pixels outside of the enhancement region, the luminance value increase of the boundary-region pixels decreasing with distance from the enhancement region.

2. A method according to claim 1 wherein identifying the at least one enhancement region within the image comprises evaluating luminance values of pixels in the LDR image data using an enhancement criterion.

3. A method according to claim 2 wherein the enhancement criterion comprises a luminance value threshold above which pixels are identified to be in the enhancement region and below which pixels are identified to be outside of the enhancement region.

4. A method according to claim 1 wherein the blur filter comprises a smoothly varying function.

5. A method according to claim 4 wherein the smoothly varying function comprises at least one of:

a Gaussian shape; and

an approximately Gaussian shape.

6. A method according to claim 1 wherein a spatial angular frequency of the blur filter is less than or equal to 1 cycle per degree.

7. A method according to claim 1 wherein a standard deviation of the blur filter is at least about 0.025 of a horizontal resolution of a display on which the HDR image data is to be displayed.

8. A method according to claim 1 comprising mapping the grey-scale image data to provide a smooth component of the brightness enhancement function, the smooth component having a range of 1-a, where a is a brightness amplification parameter greater than 1.

9. A method according to claim 8 wherein the brightness amplification parameter is in a range of 2 to 10.

10. A method according to claim 8 wherein applying the brightness enhancement function to the LDR image data comprises pixel-wise multiplication of the smooth component with the LDR image data.

11. A method according to claim 8 wherein generating the brightness enhancement function comprises generating an edge-stopping component of the brightness enhancement function based at least in part on a gradient image of the LDR image data, the edge-stopping component having pixel values which indicate whether or not application of the brightness enhancement function should increase the luminance of corresponding pixels of the LDR image data.

12. A method according to claim 11 wherein applying the brightness enhancement function to the LDR image data comprises applying the smooth component to the LDR image data only for pixels where the edge-stopping component indicates that application of the brightness enhancement function should increase the luminance of the pixels.

13. A method according to claim 12 wherein applying the smooth component to the LDR image data only for pixels where the edge-stopping component indicates that application of the brightness enhancement function should increase the luminance of the pixels comprises pixel-wise multiplication of the smooth component with the LDR image data.

14. A method according to claim 1 wherein generating the brightness enhancement function comprises generating an edge-stopping component of the brightness enhancement function based at least in part on a gradient image of the LDR image data, the edge-stopping component having pixel values which indicate whether or not application of the brightness enhancement function should increase the luminance of corresponding pixels of the LDR image data.

15. A method according to claim 14 comprising determining the gradient image of the LDR image data using a method of divided differences.

16. A method according to claim 14 comprising determining the gradient image of the LDR image data by computing differences between vertically neighbouring pixels and horizontally neighbouring pixels.

17. A method according to claim 16 wherein the vertically neighbouring pixels and horizontally neighbouring pixels are spaced apart from one another by a baseline of two or more pixels.

18. A method according to claim 14 wherein generating the edge-stopping component comprises determining whether the gradient image of the LDR image data are greater than a threshold.

19. A method according to claim 18 wherein generating the edge-stopping component comprises performing a flood fill operation using the mask as a seed and allowing the flood fill to proceed outward from the enhancement region unit the flood fill reaches pixels for which the gradient image are greater than the threshold.

20. A method according to claim 19 wherein generating the edge-stopping component comprises applying a morphological OPEN operator to a result of the flood fill operation.

21. A method according to claim 1 comprising generating the brightness enhancement function, wherein generating the brightness enhancement function comprises:

generating a mask which identifies pixels inside and outside of the enhancement region;

down-sampling the mask to obtain a down-sampled mask;

for each iteration in a loop comprising N iterations;

applying a blur filter to the down-sampled mask in an initial iteration and to a result of the previous iteration in other iterations; and

up-sampling a result of the blur filter by an up-sampling step;

wherein a result of the N^th iteration of the loop comprises grey-scale image data having the same resolution as the LDR image data.

22. A method according to claim 21 comprising mapping the grey-scale image data to provide a smooth component of the brightness enhancement function, the smooth component having a range of 1-a, where a is a brightness amplification parameter greater than 1.

23. A method according to claim 22 wherein applying the brightness enhancement function to the LDR image data comprises pixel-wise multiplication of the smooth component with the LDR image data.

24. A method according to claim 22 wherein generating the brightness enhancement function comprises generating an edge-stopping component of the brightness enhancement function based at least in part on a gradient image of the LDR image data, the edge-stopping component having pixel values which indicate whether or not application of the brightness enhancement function should increase the luminance of corresponding pixels of the LDR image data.

25. A method according to claim 24 wherein generating the edge-stopping component comprises:

down-sampling the gradient image of the LDR image data in a series of N down-sampling steps to obtain a set of N down-sampled gradient images;

for each iteration in a loop comprising N iterations;

performing a DILATION operation to the down-sampled mask in an initial iteration and to a result of the previous iteration in other iterations; and

up-sampling a result of the DILATION operation by an up-sampling step.

26. A method according to claim 25 comprising, for each DILATION operation:

identifying one of the set of N down-sampled gradient images having a resolution corresponding to that of a current subject of the DILATION operation; and

stopping the DILATION operation at pixels in the one of the set of N down-sampled gradient images having gradients greater than a threshold.

27. A method according to claim 25 wherein applying the brightness enhancement function to the LDR image data comprises applying the smooth component to the LDR image data only for pixels where the edge-stopping component indicates that application of the brightness enhancement function should increase the luminance of the pixels.

28. A method according to claim 27 wherein applying the smooth component to the LDR image data only for pixels where the edge-stopping component indicates that application of the brightness enhancement function should increase the luminance of the pixels comprises pixel-wise multiplication of the smooth component with the LDR image data.

29. A method according to claim 21 wherein generating the brightness enhancement function comprises generating an edge-stopping component of the brightness enhancement function based at least in part on a gradient image of the LDR image data, the edge-stopping component having pixel values which indicate whether or not application of the brightness enhancement function should increase the luminance of corresponding pixels of the LDR image data.

30. A method according to claim 29 wherein generating the edge-stopping component comprises:

down-sampling the gradient image of the LDR image data in a series of N down-sampling steps to obtain a set of N down-sampled gradient images;

for each iteration in a loop comprising N iterations;

performing a DILATION operation to the down-sampled mask in an initial iteration and to a result of the previous iteration in other iterations; and

up-sampling a result of the DILATION operation by an up-sampling step.

31. A method according to claim 30 comprising, for each DILATION operation:

identifying one of the set of N down-sampled gradient images having a resolution corresponding to that of a current subject of the DILATION operation; and

stopping the DILATION operation at pixels in the one of the set of N down-sampled gradient images having gradients greater than a threshold.

32. A method according to claim 1 comprising generating the brightness enhancement function, wherein generating the brightness enhancement function comprises:

down-sampling the LDR image data to obtain intermediate-resolution down-sampled image data;

further down-sampling the intermediate-resolution down-sampled image data to obtain lower-resolution down-sampled image data;

generating a lower-resolution mask from the lower-resolution down-sampled image data, the lower-resolution mask identifying pixels inside and outside of the enhancement region; and

for each iteration in a loop comprising N iterations;

applying a blur filter to the lower-resolution mask in an initial iteration and to a result of the previous iteration in other iterations; and

up-sampling a result of the blur filter by an up-sampling step;

wherein a result of the N^th iteration of the loop comprises grey-scale image data having the same resolution as the LDR image data.

33. A method according to claim 32 comprising mapping the grey-scale image data to provide a smooth component of the brightness enhancement function, the smooth component having a range of 1-a, where a is a brightness amplification parameter greater than 1.

34. A method according to claim 33 wherein generating the brightness enhancement function comprises generating an edge-stopping component of the brightness enhancement function based at least in part on a gradient image of the LDR image data, the edge-stopping component having pixel values which indicate whether or not application of the brightness enhancement function should increase the luminance of corresponding pixels of the LDR image data.

35. A method according to claim 34 wherein generating the edge-stopping component comprises:

down-sampling the gradient image of the LDR image data in a series of N down-sampling steps to obtain a set of N down-sampled gradient images;

for each iteration in a first loop comprising M iterations, where M<

N;

performing a DILATION operation to the lower-resolution mask in an initial iteration and to a result of the previous iteration in other iterations; and

up-sampling a result of the DILATION operation by an up-sampling step;

wherein a result of the first loop comprises intermediate-resolution edge-stopping data;

for each iteration in a second loop comprising N-M iterations;

performing the DILATION operation to the intermediate-resolution edge-stopping data in an initial iteration and to a result of the previous iteration in other iterations; and

up-sampling a result of the DILATION operation by an up-sampling step;

wherein a result of the second loop comprises full-resolution edge-stopping data having a resolution corresponding to that of the LDR image data.

36. A method according to claim 35 comprising determining first modulator driving values for a first modulator of a dual modulator display used to display the HDR image data based at least in part on the intermediate-resolution down-sampled image data and based at least in part on the intermediate-resolution edge-stopping data.

37. A method according to claim 32 wherein generating the brightness enhancement function comprises generating an edge-stopping component of the brightness enhancement function based at least in part on a gradient image of the LDR image data, the edge-stopping component having pixel values which indicate whether or not application of the brightness enhancement function should increase the luminance of corresponding pixels of the LDR image data.

38. A method according to claim 37 wherein generating the edge-stopping component comprises:

down-sampling the gradient image of the LDR image data in a series of N down-sampling steps to obtain a set of N down-sampled gradient images;

for each iteration in a first loop comprising M iterations, where M<

N;

performing a DILATION operation to the lower-resolution mask in an initial iteration and to a result of the previous iteration in other iterations; and

up-sampling a result of the DILATION operation by an up-sampling step;

wherein a result of the first loop comprises intermediate-resolution edge-stopping data;

for each iteration in a second loop comprising N-M iterations;

performing the DILATION operation to the intermediate-resolution edge-stopping data in an initial iteration and to a result of the previous iteration in other iterations; and

up-sampling a result of the DILATION operation by an up-sampling step;

wherein a result of the second loop comprises full-resolution edge-stopping data having a resolution corresponding to that of the LDR image data.

39. A method according to claim 38 comprising determining first modulator driving values for a first modulator of a dual modulator display used to display the HDR image data based at least in part on the intermediate-resolution down-sampled image data and based at least in part on the intermediate-resolution edge-stopping data.

40. A method according to claim 32 comprising determining first modulator driving values for a first modulator of a dual modulator display used to display the HDR image data based at least in part on the intermediate-resolution down-sampled image data.

41. A non-transitory computer-readable medium comprising a series of instructions which, when executed by a suitably configured processor, cause the processor to perform a method for enhancing lower-dynamic range (LDR) image data representing an image, the method comprising:

identifying at least one enhancement region within the image;

generating a brightness enhancement function, wherein generating the brightness enhancement function comprises;

generating a mask which identifies pixels inside and outside of the enhancement region;

applying a blur filter to the mask to obtain grey-scale image data;

applying the brightness enhancement function to the LDR image data to generate higher-dynamic range (HDR) image data, wherein compared to the LDR image data, the HDR image data comprises increased luminance values of enhancement-region pixels in the enhancement region and increased luminance values of one or more boundary-region pixels outside of the enhancement region, the luminance value increase of the boundary-region pixels decreasing with distance from the enhancement region.

42. A non-transitory computer-readable medium according to claim 41 wherein generating the brightness enhancement function comprises generating an edge-stopping component of the brightness enhancement function based at least in part on a gradient image of the LDR image data, the edge-stopping component having pixel values which indicate whether or not application of the brightness enhancement function should increase the luminance of corresponding pixels of the LDR image data.

43. A non-transitory computer-readable medium according to claim 41 wherein the method comprises generating the brightness enhancement function, wherein generating the brightness enhancement function comprises:

generating a mask which identifies pixels inside and outside of the enhancement region;

down-sampling the mask to obtain a down-sampled mask;

for each iteration in a loop comprising N iterations;

applying a blur filter to the down-sampled mask in an initial iteration and to a result of the previous iteration in other iterations; and

up-sampling a result of the blur filter by an up-sampling step;

wherein a result of the N^th iteration of the loop comprises grey-scale image data having the same resolution as the LDR image data.

44. A non-transitory computer-readable medium according to claim 43 wherein generating the brightness enhancement function comprises generating an edge-stopping component of the brightness enhancement function based at least in part on a gradient image of the LDR image data, the edge-stopping component having pixel values which indicate whether or not application of the brightness enhancement function should increase the luminance of corresponding pixels of the LDR image data.

45. A non-transitory computer-readable medium according to claim 44 wherein generating the edge-stopping component comprises:

down-sampling the gradient image of the LDR image data in a series of N down-sampling steps to obtain a set of N down-sampled gradient images;

for each iteration in a loop comprising N iterations;

performing a DILATION operation to the down-sampled mask in an initial iteration and to a result of the previous iteration in other iterations; and

up-sampling a result of the DILATION operation by an up-sampling step.

46. A non-transitory computer-readable medium according to claim 41 wherein the method comprises generating the brightness enhancement function, wherein generating the brightness enhancement function comprises:

down-sampling the LDR image data to obtain intermediate-resolution down-sampled image data;

further down-sampling the intermediate-resolution down-sampled image data to obtain lower-resolution down-sampled image data;

generating a lower-resolution mask from the lower-resolution down-sampled image data, the lower-resolution mask identifying pixels inside and outside of the enhancement region; and

for each iteration in a loop comprising N iterations;

applying a blur filter to the lower-resolution mask in an initial iteration and to a result of the previous iteration in other iterations; and

up-sampling a result of the blur filter by an up-sampling step;

wherein a result of the N^th iteration of the loop comprises grey-scale image data having the same resolution as the LDR image data.

47. A non-transitory computer-readable medium according to claim 46 wherein generating the brightness enhancement function comprises generating an edge-stopping component of the brightness enhancement function based at least in part on a gradient image of the LDR image data, the edge-stopping component having pixel values which indicate whether or not application of the brightness enhancement function should increase the luminance of corresponding pixels of the LDR image data.

48. A non-transitory computer-readable medium according to claim 47 wherein generating the edge-stopping component comprises:

down-sampling the gradient image of the LDR image data in a series of N down-sampling steps to obtain a set of N down-sampled gradient images;

for each iteration in a first loop comprising M iterations, where M<

N;

performing a DILATION operation to the lower-resolution mask in an initial iteration and to a result of the previous iteration in other iterations; and

up-sampling a result of the DILATION operation by an up-sampling step;

wherein a result of the first loop comprises intermediate-resolution edge-stopping data;

for each iteration in a second loop comprising N-M iterations;

performing the DILATION operation to the intermediate-resolution edge-stopping data in an initial iteration and to a result of the previous iteration in other iterations; and

up-sampling a result of the DILATION operation by an up-sampling step;

wherein a result of the second loop comprises full-resolution edge-stopping data having a resolution corresponding to that of the LDR image data.

49. A non-transitory computer-readable medium according to claim 48 wherein the method comprises determining first modulator driving values for a first modulator of a dual modulator display used to display the HDR image data based at least in part on the intermediate-resolution down-sampled image data and based at least in part on the intermediate-resolution edge-stopping data.

50. A non-transitory computer-readable medium according to claim 46 wherein the method comprises determining first modulator driving values for a first modulator of a dual modulator display used to display the HDR image data based at least in part on the intermediate-resolution down-sampled image data.

51. A display for displaying a higher-dynamic range (HDR) image obtained from lower-dynamic range (LDR) image data representing an image, the display comprising:

a pair of modulators comprising;

a first modulator configured to provide a light pattern varying in response to a set of driving signals; and

a second modulator configured to modulate the light pattern to yield an output image; and

a processor configured to;

identify at least one enhancement region within the image;

generate a brightness enhancement function and wherein generating the brightness enhancement function inv olves generating an edge-stopping component of the brightness enhancement function based at least in part on a gradient image of the LDR image data, the edge-stopping component having pixel values which indicate whether or not application of the brightness enhancement function should increase the luminance of a corresponding pixels of the LDR image data and wherein the intermediate data comprises a low resolution version of the edge-stopping component;

apply the brightness enhancement function to the LDR image data to generate the HDR image data, wherein compared to the LDR image data, the HDR image data comprises increased luminance values of enhancement-region pixels in the enhancement region and increased luminance values of one or more boundary-region pixels outside of the enhancement region, the luminance value increase of the boundary-region pixels decreasing with distance from the enhancement region; and

determine the driving signals for the first modulator based at least in part on intermediate data obtained in the course of application of the brightness enhancement function to the LDR image data.

52. A display according to claim 51 wherein the intermediate data comprises a down-sampled version of the LDR image data.

54. A method according to claim 52 wherein the intermediate data comprises a down-sampled version of the LDR image data.

53. A method for displaying a higher-dynamic range (HDR) image obtained from lower-dynamic range (LDR) image data, the method comprising:

identifying at least one enhancement region within the LDR image data;

generating a brightness enhancement function and wherein generating the brightness enhancement function comprises generating an edge-stopping component of the brightness enhancement function based at least in part on a gradient image of the LDR image data, the edge-stopping component having pixel values which indicate whether or not application of brightness enhancement function should increase the luminance of corresponding pixels of the LDR image data and wherein the intermediate data comprises a low resolution version of the edge-stopping component;

applying the brightness enhancement function to the LDR image data to generate the HDR image data, wherein compared to the LDR image data, the HDR image data comprises increased luminance values of enhancement-region pixels in the enhancement region and increased luminance values of one or more boundary-region pixels outside of the enhancement region, the luminance value increase of the boundary-region pixels decreasing with distance from the enhancement region; and

determining driving signals for at least one modulator of a dual-modulator display based at least in part on intermediate data obtained in the course of application of the brightness enhancement function to the LDR image data.