Enhancing a digital image

• US 10,643,314 B2
• Filed: 03/28/2019
• Issued: 05/05/2020
• Est. Priority Date: 12/16/2015
• Status: Active Grant

• Current Assignee: Dropbox, Inc.
• Original Assignee: Dropbox, Inc.
• Inventors: Baek, Jongmin
• Primary Examiner(s): Allison, Andrae S
• Application Number:
  US16/368,518
• Publication Number:
  US 20190220959A1
• Time in Patent Office: 404 Days
• Field of Search: None
• CPC Class Codes: G06T 1/0007 Image acquisition
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20012 Locally adaptive
• CPC Class Codes: G06T 2207/20028 Bilateral filtering
• CPC Class Codes: G06T 2207/20112 Image segmentation details
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06T 5/94 based on local image properties, e.g. for local contrast enhancement
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/136 involving thresholding
• CPC Class Codes: G06T 7/168 involving transform domain methods

Abstract

One or more embodiments of an image enhancement system enable a computing device to generate an enhanced digital image. In particular, a computing device can enhance a digital image including, for example, a photograph of a whiteboard, document, chalkboard, or other object having a uniform background. The computing device can determine modifications to apply to the digital image by minimizing an energy heuristic that both causes pixels of the digital image to change to a uniform color (e.g., white) and preserves gradients from the digital image. The computing device can further generate an enhanced digital image by applying the determined modifications to the digital image.

Claims

1. A method comprising:

accessing an input image depicting an object comprising foreground content and a background, the foreground content comprising a non-shadow foreground portion and a shadow foreground portion, and the background comprising a non-shadow background portion and a shadow background portion; and

generating, by at least one processor, an enhanced output image based on modifying the input image, wherein modifying the input image comprises;

modifying the non-shadow background portion and the shadow background portion of the background to be a uniform background color to create a uniform background within the enhanced output image; and

modifying the non-shadow foreground portion and the shadow foreground portion of the foreground content to be a uniform foreground color within the enhanced output image.

2. The method of claim 1, wherein identifying the shadow background portion and the non-shadow background portion comprises identifying gradient values for pixels within the input image.

3. The method of claim 2, wherein:

identifying the shadow background portion comprises identifying at least a first group of pixels having gradient values within a first range of gradient values; and

identifying the non-shadow background portion comprises identifying at least a second group of pixels having gradient values within a second range of gradient values.

4. The method of claim 3, wherein modifying the non-shadow background portion and the shadow background portion to be the uniform background color to create the uniform background comprises:

changing the first group of pixels having gradient values within the first range to the uniform background color; and

changing the second group of pixels having gradient values within the second range to the uniform background color.

5. The method of claim 3, wherein:

the first range of gradient values indicates a first non-gradient portion of the input image;

the second range of gradient values indicates a second non-gradient portion of the input image; and

the first range of gradient values is different from the second range of gradient values.

6. The method of claim 3, wherein the first range of gradient values and the second range of gradient values are based on a background parameter within an energy heuristic that defines a difference between gradient values corresponding to non-gradient portions.

7. The method of claim 1, wherein the uniform background color used to create the uniform background is white.

8. The method of claim 1, wherein:

identifying the shadow foreground portion comprises identifying at least a first group of pixels having gradient values within a first range of gradient values; and

identifying the non-shadow foreground portion comprises identifying at least a second group of pixels having gradient values within a second range of gradient values.

9. The method of claim 1, wherein modifying the non-shadow foreground portion and the shadow foreground portion comprises:

minimizing an energy heuristic corresponding to the background and the foreground content; and

determining gradients of foreground pixels of the foreground content based on minimizing the energy heuristic.

10. A system comprising:

at least one processor; and

at least one non-transitory computer-readable storage medium storing instructions that, when executed by the at least one processor, cause the system to;

access an input image depicting an object comprising foreground content and a background, the foreground content comprising a non-shadow foreground portion and a shadow foreground portion, and the background comprising a non-shadow background portion and a shadow background portion; and

generating an enhanced output image based on modifying the input image, wherein modifying the input image comprises;

modifying the non-shadow background portion and the shadow background portion of the background to be a uniform background color to create a uniform background within the enhanced output image; and

modifying the non-shadow foreground portion and the shadow foreground portion of the foreground content to be a uniform foreground color within the enhanced output image.

11. The system of claim 10, wherein:

identifying the shadow background portion comprises identifying at least a first group of pixels having gradient values within a first range of gradient values; and

identifying the non-shadow background portion comprises identifying at least a second group of pixels having gradient values within a second range of gradient values.

12. The system of claim 11, wherein modifying the non-shadow background portion and the shadow background portion to be the uniform background color to create the uniform background comprises:

changing the first group of pixels having gradient values within the first range to the uniform background color; and

changing the second group of pixels having gradient values within the second range to the uniform background color.

13. The system of claim 11, wherein:

the first range of gradient values indicates a first non-gradient portion of the input image;

the second range of gradient values indicates a second non-gradient portion of the input image; and

the first range of gradient values is different from the second range of gradient values.

14. The system of claim 11, wherein the first range of gradient values and the second range of gradient values are based on a background parameter within an energy heuristic that defines a difference between gradient values corresponding to non-gradient portions.

15. The system of claim 10, wherein modifying the non-shadow foreground portion and the shadow foreground portion further comprises:

minimizing an energy heuristic corresponding to the background and the foreground content; and

determining gradients of foreground pixels of the foreground content based on minimizing the energy heuristic.

16. A non-transitory computer readable medium storing instructions thereon that, when executed by at least one processor, cause a computer device to:

access an input image depicting an object comprising foreground content and a background, the foreground content comprising a non-shadow foreground portion and a shadow foreground portion, and the background comprising a non-shadow background portion and a shadow background portion; and

generate an enhanced output image based on modifying the input image, wherein modifying the input image comprises;

modifying the non-shadow background portion and the shadow background portion of the background to be a uniform background color to create a uniform background within the enhanced output image; and

modifying the non-shadow foreground portion and the shadow foreground portion of the foreground content to be a uniform foreground color within the enhanced output image.

17. The non-transitory computer readable medium of claim 16, wherein:

identifying the shadow background portion comprises identifying at least a first group of pixels having gradient values within a first range of gradient values; and

identifying the non-shadow background portion comprises identifying at least a second group of pixels having gradient values within a second range of gradient values.

18. The non-transitory computer readable medium of claim 17, wherein modifying the non-shadow background portion and the shadow background portion to be the uniform background color to create the uniform background comprises:

changing the first group of pixels having gradient values within the first range to the uniform background color; and

changing the second group of pixels having gradient values within the second range to the uniform background color.

19. The non-transitory computer readable medium of claim 18, wherein:

the first range of gradient values indicates a first non-gradient portion of the input image;

the second range of gradient values indicates a second non-gradient portion of the input image; and

the first range of gradient values is different from the second range of gradient values.

20. The non-transitory computer readable medium of claim 16, wherein modifying the non-shadow foreground portion and the shadow foreground portion comprises:

minimizing an energy heuristic corresponding to the background and the foreground content; and

determining gradients of foreground pixels of the foreground content based on minimizing the energy heuristic.