Enhancing a digital image

• US 9,799,106 B2
• Filed: 12/16/2015
• Issued: 10/24/2017
• Est. Priority Date: 12/16/2015
• Status: Active Grant

• Current Assignee: Dropbox, Inc.
• Original Assignee: Dropbox, Inc.
• Inventors: Baek, Jongmin
• Primary Examiner(s): Allison, Andrae S
• Application Number:
  US14/971,733
• Publication Number:
  US 20170178299A1
• Time in Patent Office: 678 Days
• Field of Search: None
• CPC Class Codes: G06T 1/0007 Image acquisition
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20012 Locally adaptive
• CPC Class Codes: G06T 2207/20028 Bilateral filtering
• CPC Class Codes: G06T 2207/20112 Image segmentation details
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06T 5/94 based on local image properties, e.g. for local contrast enhancement
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/136 involving thresholding
• CPC Class Codes: G06T 7/168 involving transform domain methods

Abstract

One or more embodiments of an image enhancement system enable a computing device to generate an enhanced digital image. In particular, a computing device can enhance a digital image including, for example, a photograph of a whiteboard, document, chalkboard, or other object having a uniform background. The computing device can determine modifications to apply to the digital image by minimizing an energy heuristic that both causes pixels of the digital image to change to a uniform color (e.g., white) and preserves gradients from the digital image. The computing device can further generate an enhanced digital image by applying the determined modifications to the digital image.

Claims

1. A method for enhancing digital images, comprising:

accessing an input image comprising a background and a foreground;

determining, by at least one processor, modifications to enhance the input image by minimizing an energy heuristic comprising a sum of a background parameter and a foreground parameter, wherein;

the background parameter causes background pixels of the input image to change to a uniform color; and

the foreground parameter preserves gradients of foreground pixels from the input image; and

generating, by the at least one processor, an output image with an enhanced foreground by applying the determined modifications to the input image.

2. The method as recited in claim 1, wherein the input image comprises a digital photograph of a whiteboard with written text.

3. The method as recited in claim 1, wherein the uniform color comprises a white pixel value.

4. The method as recited in claim 1, wherein the one or more foreground parameters that preserve gradients from the input image comprise:

a first foreground parameter that preserves horizontal gradients of foreground pixels from the input image; and

a second foreground parameter that preserves vertical gradients of foreground pixels from the input image.

5. The method as recited in claim 1, further comprising converting the input image to grayscale.

6. The method as recited in claim 5, wherein converting the input image to grayscale comprises:

identifying one or more minimum red green blue (RGB) pixel values across the input image; and

normalizing the one or more minimum RGB pixel values to zero.

7. The method as recited in claim 1, wherein generating the output image with the enhanced foreground further comprises blending color from the input image to the output image.

8. The method as recited in claim 7, wherein blending color from the original color image to the output image comprises adding color to the enhanced foreground based on fractional values of gradients from the input image.

9. The method as recited in claim 1, wherein determining the modifications to enhance the input image comprises:

converting the sum of the background parameter and the foreground parameter to a frequency domain; and

solving for one or more zero values for the converted sum of the background parameter and the foreground parameter.

10. The method as recited in claim 9, wherein converting the sum of the background parameter and the foreground parameter to the frequency domain comprises performing a Fast Fourier Transform (FFT).

11. The method as recited in claim 1, further comprising filtering the output image to fill in hollow portions of the enhanced foreground.

12. The method as recited in claim 11, wherein filtering the output image to fill in hollow portions of the enhanced foreground comprises applying a cross-bilateral filter to the output image with respect to the input image.

13. The method as recited in claim 12, further comprising:

generating, based on the determined modifications to enhance the input image, a binarization map comprising pixel values to apply to the input image; and

wherein generating the output image with the enhanced foreground by applying the determined modifications to the input image comprises masking the binarization map to the input image.

14. The method as recited in claim 1, wherein the energy heuristic comprises a screened Poisson equation.

15. The method of claim 1, wherein the input image comprises a digital photograph of a document.

16. The method of claim 1, further comprising minimizing the energy heuristic in accordance with one or more received user inputs identifying a weighting factor to apply to the background parameter or the foreground parameter.

17. A method for enhancing digital images, comprising:

accessing a digital image comprising a photograph of a whiteboard and textual content;

determining, by at least one processor, modifications to enhance the digital image by minimizing a sum of squared parameters, the squared parameters comprising;

a background parameter comprising an indication that background pixels of an enhanced digital image be a uniform color; and

a foreground parameter comprising a difference between gradient values of foreground pixels of the digital image and foreground pixels of the enhanced digital image; and

generating, by the at least one processor, the enhanced digital image having enhanced textual content by causing pixels of the digital image to change in accordance with the determined modifications to the digital image.

18. The method as recited in claim 17, further comprising:

converting the sum of squared parameters to a frequency domain; and

wherein calculating the minimum value of the sum of squared parameters comprises solving for one or more zero values for the converted sum of squared parameters.

19. A computing device comprising:

at least one processor; and

a non-transitory computer-readable medium storing instructions thereon that, when executed by the at least one processor, cause the computing device to;

access an input image comprising a background and a foreground;

determine modifications to enhance the input image by minimizing an energy heuristic comprising a sum of a background parameter and a foreground parameter, wherein;

the background parameter causes background pixels of the input image to change to a uniform color; and

the foreground parameter preserves gradients of foreground pixels from the input image; and

generate an output image with an enhanced foreground by applying the determined modifications to the input image.

20. The computing device of claim 19, wherein the instructions further cause the computing device to:

generate a trimap based on an output of the energy heuristic, the trimap classifying portions of the output image as foreground, background, and unknown;

for each pixel within the output image, determine a pixel value based on neighboring foreground and background portions of the output image; and

generate, based on the determined pixel value for each pixel within the output image, a binarization map;

wherein generating the output image with the enhanced foreground by applying the determined modifications to the input image comprises masking the generated binarization map against the input image.