Tone mapping of very large aerial image mosaic

• US 8,737,736 B2
• Filed: 12/31/2010
• Issued: 05/27/2014
• Est. Priority Date: 12/31/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Liu, Yuxiang, Schickler, Wolfgang, Rosenshein, Leon, Simons, David, Omer, Ido, Ledner, Rob
• Primary Examiner(s): Mehta, Bhavesh
• Assistant Examiner(s): Niu, Feng
• Application Number:
  US12/983,131
• Publication Number:
  US 20120170842A1
• Time in Patent Office: 1,243 Days
• Field of Search: None
• US Class Current: 382/169
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/90 Dynamic range modification of images or parts thereof

Abstract

A method for tone mapping a high dynamic range image of a large terrestrial area into a lower dynamic range image uses a globally aware, locally adaptive approach whereby local tonal balancing parameter values are derived from known tone mapping parameters for a local 3×3 matrix of image tiles and used in turn to derive a local sigmoid transfer function for pixels in the tile in the middle of the matrix. A global sigmoid transfer function is derived based on values of the tone mapping parameters applicable to the entire image. A lower dynamic range image pixel will have a local tone mapped value and a globally tone mapped value, which are combined by giving each a weighted value to provide a final low dynamitic range pixel value.

Claims

1. A method for converting a first image represented by HDR pixels having a predetermined dynamic range, said first image comprising a plurality of tiles each including a two-dimensional array of a plurality of said HDR pixels, into a second image with corresponding LDR pixels having a lower dynamic range, the method including:

performing a local tone mapping on each HDR pixel of said first image using a local transfer function for converting HDR pixels into LDR pixels having a locally tone mapped value, said local transfer function being derived from at least one image parameter of said HDR pixels in a plurality of tiles proximate to a tile containing a particular HDR pixel, wherein the number of the plurality of tiles proximate to said tile containing the particular HDR pixel depends on content of a scene that has been captured by said first image;

performing a global tone mapping on each HDR pixel using a global transfer function for converting HDR pixels into LDR pixels having a globally tone mapped value, said global transfer function being derived from an average value of said at least one image parameter of all said HDR pixels;

deriving a final value for each LDR pixel by weighting said locally and globally tone mapped pixel values using a first weighting factor applied to each said locally tone mapped LDR pixel value and a second weighting factor applied to each said globally tone mapped LDR pixel value and combining said weighted locally and globally tone mapped LDR pixel values; and

storing one or more derived final values in a storage device.

2. The method as in claim 1, wherein said at least one image parameter includes a shadow percentile value and a highlight percentile value and said transfer functions comprise sigmoid tone mapping functions embodied in a look-up table of a computer processor.

3. The method as in claim 2, wherein said first image and second images are color images comprising pixels with red (R), green (G), blue (B), and infrared (IR) components, each said component being represented by 16 bits of data in said first HDR image and 8 bits of data in said second LDR image.

4. The method as in claim 3, wherein

said plural tiles proximate to said tile define a neighborhood of said tiles and each said neighborhood has a shadow percentile value (ShadowValue) and a highlight percentile value (HighlightValue) for each component of said HDR image comprising said neighborhood, and

the method further includes:

determining neighborhood values of the following tonal balancing parameters for all of said neighborhoods in said image

RGBTileShadowValue=min(ShadowValueR,ShadowValueG,ShadowValueB)

FCIRTileShadowValue=min(ShadowValueR,ShadowValueG,ShadowValueIR)

RGBTileHighlightValue=max(HighlightValueR,HighlightValueG, HighlightValueB)

FCIRTileHighlightValue=max(HighlightValueR,HighlightValueG, HighlightValuelR),

where the notation “

min(A,B,C)”

indicates that a tonal balancing parameter is taken as the minimum value of the included percentile values for each pixel in said respective tile, and the notation “

max(A,B,C)”

indicates that a tonal balancing parameter is taken as the maximum value of the included percentile values for each pixel in said respective tile;

assigning to a predetermined tile in each said neighborhood said neighborhood values of said tonal balancing parameters; and

deriving from said neighborhood tonal balancing parameters plural transfer functions for tone mapping each said component of each HDR pixel in said predetermined tile to an LDR pixel.

5. The method as in claim 4, wherein said tiles are square, said neighborhood comprises a square matrix of at least nine tiles, and said predetermined tile is the center tile of each said matrix.

6. The method as in claim 5, wherein,

for a pixel in said first image located at a point X,Y in one of four equal quadrants of said center tile and having an HDR value for each image component of S_X,Y, with the neighbor transfer functions for the image components of said center tile being f₁₁, and the neighborhood transfer functions for the image components of each of said three tiles adjacent to said quadrant respectively being f₁₂, f₂₂, and f₂₁ viewed in a clockwise direction around said center tile, the locally tone mapped values for said components of said HDR pixel at X,Y are determined by the following algorithm

R1=(X2−

X)/(X2−

X1)×

f₁₁+(X−

X1)/(X2−

X1)×

f₂₁

R2=(X2−

X)/(X2−

X1)×

f₁₂+(X−

X1)/(X2−

X1)×

f₂₂

P_X,Y=(Y2−

Y)/(Y2−

Y1)×

R1+(Y−

Y1)/(Y2−

Y1)×

R2, and

wherein X,Y are the coordinates of said pixel in question and the center of said center tile is the origin of orthogonal X-Y axes, with X1,Y2, X2,Y2, and X2,Y1 are the respective coordinates of the centers of said three adjacent tiles viewed in a clockwise direction around said center tile.

7. The method as in claim 6, wherein each said transfer function for locally tone mapping said HDR pixel at X,Y to an LDR image pixel is derived by:

subtracting said RGBTileShadowValue from each of said HDR pixel'"'"'s R,G,B, components and FCIRTileShadowValue from said pixel'"'"'s IR component;

obtaining a floating point value between 0 and 1 by multiplying each respective pixel component value by a scale factor=1÷

the derived 16-bit dynamic range for said center tile, wherein each of said pixel'"'"'s R, G, and B components have a dynamic range=RGBTileHighlightValue−

RGBTileShadowValue and said pixel'"'"'s IF component has a dynamic range=FCIRTileHighlightValue−

FCIRTileShadowValue;

applying a gamma value (γ

) as a power-law function in the form y=x^1/γ

, wherein x is said floating point value; and

multiplying said gamma-corrected floating point value (y) by 255, wherein any resulting value below RGBTileShadowValue or FCIRTileShadowValue will be mapped to 0, and any resulting value above RGBTileHighlightValue or FCIRTileHIghlightValue will be mapped to 255.

8. The method as in claim 7, wherein said global transfer function for globally tone mapping a said HDR pixel in said first image to an LDR pixel is derived by:

determining an average global shadow percentile value (ShadowValue) and an average global highlight percentile value (HighlightValue) for all of said neighborhoods in said region R;

deriving global tonal balancing parameters RGBGlobShadowValue=min(ShadowValueR,ShadowValueG,ShadowValueB), FCIRGlobShadowValue=min(ShadowValueR,ShadowValueG,ShadowValueIR), and RGBGlobHighlightValue=max(HighlightValueR,HighlightValueG,HighlightValueB), based on said average global shadow and highlight percentile values, where the notation “

min(A,B,C)”

indicates that a tonal balancing parameter is taken as the minimum value of the included percentile values for each pixel in said respective tile, and the notation “

max(A,B,C)”

indicates that a tonal balancing parameter is taken as the maximum value of the included percentile values for each pixel in said respective tile;

subtracting said RGBGlobShadowValue from each of said pixel'"'"'s R,G,B, components and FCIRTileShadowValue from said pixel'"'"'s IR component;

obtaining a floating point value between 0 and 1 by multiplying each respective pixel component value by a scale factor=1÷

the derived 16-bit dynamic range for said pixel, wherein each of said pixel'"'"'s R, G, and B components have a dynamic range=RGBGlobHighlightValue−

RGBGlobShadowValue and said pixel'"'"'s IF component has a dynamic range=FCIRGlobHighlightValue−

FCIRGlobShadowValue;

applying a gamma value (γ

) as a power-law function in the form y=x^1/γ

, wherein x is said floating point value; and

multiplying said gamma-corrected floating point value (y) by 255, wherein any resulting value below RGBGlobShadowValue or FCIRGlobShadowValue will be mapped to 0, and any resulting value above RGBGlobHighlightValue or FCIRGlobHIghlightValue will be mapped to 255.

9. The method as in claim 8, wherein said first weighting factor is a local weight factor and said second weighting factor is (1−

local weight factor), and said final LDR pixel value is determined for each image component.

10. The method as in claim 3, wherein said first weighting factor is a local weight factor and said second weighting factor is (1−

local weight factor), and said final LDR pixel value is determined for each image component.

11. The method as in claim 1, wherein said first and second weighting factors have a sum-to-one relationship.

12. The method as in claim 1, wherein

said at least one image parameter includes a shadow percentile value and a highlight percentile value and said transfer functions comprise sigmoid tone mapping functions embodied in a look-up table of a computer processor,

said first image and second images are color images comprising pixels with red (R), green (G), blue (B), and infrared (IR) components, each said component being represented by 16 bits of data in said first HDR image and 8 bits of data in said second LDR image,

said plural tiles proximate to said tile define a neighborhood of said tiles and each said neighborhood has a shadow percentile value and a highlight percentile value for each component of said HDR image comprising said neighborhood, and

the method further includes:

determining neighborhood values of the following tonal balancing parameters for all of said neighborhoods in said image by;

setting a RGB shadow percentile tonal balancing parameter for each respective tile to the minimum value of the shadow percentile value of each of the R, G and B components;

setting an IR shadow percentile tonal balancing parameter for each respective tile to the minimum value of the shadow percentile value of each of the R, G and IR component;

setting a RGB highlight percentile tonal balancing parameter for each respective tile to the maximum value of the highlight percentile value of each of the R, G and B component; and

setting an IR highlight percentile tonal balancing parameter for each respective tile to the maximum value of the highlight percentile value of each of the R, G and IR component;

assigning to a predetermined tile in each said neighborhood said neighborhood values of said tonal balancing parameters; and

deriving from said neighborhood tonal balancing parameters plural transfer functions for tone mapping each said component of each HDR pixel in said predetermined tile to an LDR pixel.

13. A computer system for converting a first image represented by HDR pixels having a predetermined dynamic range, said first image comprising a plurality of tiles each including a two-dimensional array of a plurality of said HDR pixels, into a second image with corresponding LDR pixels having a lower dynamic range, the system comprising:

a processor module with executable instructions in an operating system module stored on a computer storage medium for carrying out operations that;

perform a local tone mapping on each HDR pixel of said first image using a local transfer function for converting HDR pixels into LDR pixels having a locally tone mapped value, said local transfer function being derived from at least one image parameter of said HDR pixels in a plurality of tiles proximate to a tile containing a particular HDR pixel, wherein the number of the plurality of tiles near said tile containing the particular HDR pixel depends on content of a scene of said first image, each tile including a plurality of pixels,

perform a global tone mapping on each HDR pixel using a global transfer function for converting HDR pixels into LDR pixels having a globally tone mapped value, said global transfer function being derived from an average value of said at least one image parameter of all said HDR pixels, and

derive a final value for each LDR pixel by weighting said locally and globally tone mapped pixel values using a first weighting factor applied to each said locally tone mapped LDR pixel value and a second weighting factor applied to each said globally tone mapped LDR pixel value and combining said weighted locally and globally tone mapped LDR pixel values; and

a storage module storing one or more derived final values in a storage device.

14. The system as in claim 13, further comprising a display component for rendering said second image in human-readable form.

15. The system as in claim 14, wherein said first image and second images are color images comprising pixels with red (R), green (G), blue (B), and infrared (IR) components, each said component being represented by 16 bits of data in said first HDR image and 8 bits of data in said second LDR image.

16. The system as in claim 15, wherein:

said tiles are square, said neighborhood comprises a square matrix of at least nine said tiles, and said predetermined tile is in the center of said neighborhood;

each said neighborhood has a shadow percentile value (ShadowValue) and a highlight percentile value (HighlightValue) for each component of said HDR image comprising said neighborhood,

neighborhood values of the following tonal balancing parameters are determined for all of said neighborhoods in said image

RGBTileShadowValue=min(ShadowValueR,ShadowValueG,ShadowValueB)

FCIRTileShadowValue=min(ShadowValueR,ShadowValueG,ShadowValueIR)

RGBTileHighlightValue=max(HighlightValueR,HighlightValueG, HighlightValueB)

FCIRTileHighlightValue=max(HighlightValueR,HighlightValueG, HighlightValuelR),

where the notation “

min(A,B,C)”

indicates that a tonal balancing parameter is taken as the minimum value of the included percentile values for each pixel in said respective tile, and the notation “

max(A,B,C)”

indicates that a tonal balancing parameter is taken as the maximum value of the included percentile values for each pixel in said respective tile;

said neighborhood values of said tonal balancing parameters is assigned to said center tile in said neighborhood; and

said plural transfer functions for tone mapping each said component of each HDR pixel in said center tile to an LDR pixel are derived from said neighborhood tonal balancing parameters.

17. The system as in claim 16, wherein,

for a pixel in said first image located at a point X,Y in one of four equal quadrants of said center tile and having an HDR value for each image component of S_X,Y, with the neighbor transfer functions for the image components of said center tile being f₁₁, and the neighborhood transfer functions for the image components of each of said three tiles adjacent to said quadrant respectively being f₁₂, f₂₂, and f₂₁ viewed in a clockwise direction around said center tile, the locally tone mapped values for said components of said HDR pixel at X,Y is determined by the following algorithm

R1=(X2−

X)/(X2−

X1)×

f₁₁+(X−

X1)/(X2−

X1)×

f₂₁

R2=(X2−

X)/(X2−

X1)×

f₁₂+(X−

X1)/(X2−

X1)×

f₂₂

P_X,Y=(Y2−

Y)/(Y2−

Y1)×

R1+(Y−

Y1)/(Y2−

Y1)×

R2, and

wherein X,Y are the coordinates of said pixel in question and the center of said center tile is the origin of orthogonal X-Y axes, with X1,Y2, X2,Y2, and X2,Y1 are the respective coordinates of the centers of said three adjacent tiles viewed in a clockwise direction around said center tile.

18. The system as in claim 17, wherein each said transfer function for locally tone mapping said HDR pixel at X,Y to an LDR image pixel is derived by:

subtracting said RGBTileShadowValue from each of said HDR pixel'"'"'s R,G,B, components and FCIRTileShadowValue from said pixel'"'"'s IR component;

obtaining a floating point value between 0 and 1 by multiplying each respective pixel component value by a scale factor=1÷

the derived 16-bit dynamic range for said center tile, wherein each of said pixel'"'"'s R, G, and B components have a dynamic range=RGBTileHighlightValue−

RGBTileShadowValue and said pixel'"'"'s IF component has a dynamic range=FCIRTileHighlightValue−

FCIRTileShadowValue;

applying a gamma value (γ

) as a power-law function in the form y=x^1/γ

, wherein x is said floating point value; and

multiplying said gamma-corrected floating point value (y) by 255, wherein any resulting value below RGBTileShadowValue or FCIRTileShadowValue will be mapped to 0, and any resulting value above RGBTileHighlightValue or FCIRTileHIghlightValue will be mapped to 255.

19. The system as in claim 18, wherein said global transfer function for globally tone mapping a said HDR pixel in said first image to an LDR pixel is derived by:

determining an average global shadow percentile value (ShadowValue) and an average global highlight percentile value (HighlightValue) for all of said neighborhoods in said region R;

deriving global tonal balancing parameters RGBGlobShadowValue=min(ShadowValueR,ShadowValueG,ShadowValueB), FCIRGlobShadowValue=min(ShadowValueR,ShadowValueG,ShadowValueIR), RGBGlobHighlightValue=max(HighlightValueR,HighlightValueG,HighlightValueB), and FCIRGlobHighlightValue=max(HighlightValueR,HighlightValueG,HighlightValueB), based on said average global shadow and highlight percentile values, where the notation “

min(A,B,C)”

indicates that a tonal balancing parameter is taken as the minimum value of the included percentile values for each pixel in said respective tile, and the notation “

max(A,B,C)”

indicates that a tonal balancing parameter is taken as the maximum value of the included percentile values for each pixel in said respective tile;

subtracting said RGBGlobShadowValue from each of said pixel'"'"'s R,G,B, components and FCIRTileShadowValue from said pixel'"'"'s IR component;

obtaining a floating point value between 0 and 1 by multiplying each respective pixel component value by a scale factor=1÷

the derived 16-bit dynamic range for said pixel, wherein each of said pixel'"'"'s R, G, and B components have a dynamic range=RGBGlobHighlightValue−

RGBGlobShadowValue and said pixel'"'"'s IF component has a dynamic range=FCIRGlobHighlightValue−

FCIRGlobShadowValue;

applying a gamma value (γ

) as a power-law function in the form y=x^1/γ

, wherein x is said floating point value; and

multiplying said gamma-corrected floating point value (y) by 255, wherein any resulting value below RGBGlobShadowValue or FCIRGlobShadowValue will be mapped to 0, and any resulting value above RGBGlobHighlightValue or FCIRGlobHIghlightValue will be mapped to 255.

20. The system as in claim 15, wherein said first weighting factor is a local weight factor and said second weighting factor is (1−

local weight factor), and said final LDR pixel value is determined for each image component.

21. The system as in claim 13, wherein said at least one image parameter includes a shadow percentile value and a highlight percentile value and said transfer functions comprise sigmoid tone mapping functions embodied in a look-up table of a computer processor.

22. The system as in claim 13, wherein

said at least one image parameter includes a shadow percentile value and a highlight percentile value and said transfer functions comprise sigmoid tone mapping functions embodied in a look-up table of a computer processor,

said first image and second images are color images comprising pixels with red (R), green (G), blue (B), and infrared (IR) components, each said component being represented by 16 bits of data in said first HDR image and 8 bits of data in said second LDR image,

said tiles are square, said neighborhood comprises a square matrix of at least nine said tiles, and said predetermined tile is in the center of said neighborhood,

each said neighborhood has a shadow percentile value and a highlight percentile value for each component of said HDR image comprising said neighborhood, neighborhood values of the following tonal balancing parameters are determined for all of said neighborhoods in said image by:

a RGB shadow percentile tonal balancing parameter for each respective tile to the minimum value of the shadow percentile value of each of the R, G and B components;

an IR shadow percentile tonal balancing parameter for each respective tile to the minimum value of the shadow percentile value of each of the R, G and IR component;

a RGB highlight percentile tonal balancing parameter for each respective tile to the maximum value of the highlight percentile value of each of the R, G and B component; and

an IR highlight percentile tonal balancing parameter for each respective tile to the maximum value of the highlight percentile value of each of the R, G and IR component,

said neighborhood values of said tonal balancing parameters is assigned to said center tile in said neighborhood, and

said plural transfer functions for tone mapping each said component of each HDR pixel in said center tile to an LDR pixel are derived from said neighborhood tonal balancing parameters.

23. A computer usable storage device with a computer program stored thereon, the computer program upon execution by a processor performs a method for converting a first image represented by HDR pixels having a predetermined dynamic range, said first image comprising a plurality of tiles each including a two-dimensional array of a plurality of said pixels, into a second image with corresponding LDR pixels having a lower dynamic range, said first image and second image being color images comprising pixels with red (R), green (G), blue (B), and infrared (IR) components, the method comprising:

performing a local tone mapping on each HDR pixel of said first image using a local transfer function for converting HDR pixels into LDR pixels having a locally tone mapped value, said local transfer function being derived from at least one image parameter of said HDR pixels in plural tiles proximate to a said tile containing a particular HDR pixel;

performing a global tone mapping on each HDR pixel using a global transfer function for converting HDR pixels into LDR pixels having a globally tone mapped value, said global transfer function being derived from an average value of said at least one image parameter of all said HDR pixels;

deriving a final value for each LDR pixel by weighting said locally and globally tone mapped pixel values using a first weighting factor applied to each said locally tone mapped LDR pixel value and a second weighting factor applied to each said globally tone mapped LDR pixel value and combining said weighted locally and globally tone mapped LDR pixel values;

determining neighborhood values of the following tonal balancing parameters for all of said neighborhoods in said image by;

setting a RGB shadow percentile tonal balancing parameter for each respective tile to the minimum value of the shadow percentile value of each of the R, G and B components;

setting an IR shadow percentile tonal balancing parameter for each respective tile to the minimum value of the shadow percentile value of each of the R, G and IR components;

setting a RGB highlight percentile tonal balancing parameter for each respective tile to the maximum value of the highlight percentile value of each of the R, G and B components; and

setting an IR highlight percentile tonal balancing parameter for each respective tile to the maximum value of the highlight percentile value of each of the R, G and IR components;

assigning to a predetermined tile in each said neighborhood said neighborhood values of said tonal balancing parameters; and

deriving from said neighborhood tonal balancing parameters plural transfer functions for tone mapping each said component of each HDR pixel in said predetermined tile to an LDR pixel; and

storing said final LDR pixel values in a storage device for outputting said second image,

wherein said at least one image parameter includes a shadow percentile value and a highlight percentile value and said transfer functions comprise sigmoid tone mapping functions embodied in a look-up table of a computer processor,

each said component being represented by 16 bits of data in said first HDR image and 8 bits of data in said second LDR image, and

wherein aid plural tiles proximate to said tile define a neighborhood of said tiles and each said neighborhood has a shadow percentile value and a highlight percentile value for each component of said HDR image comprising said neighborhood.