REMOVING ARTIFACTS FROM DOCUMENT IMAGES

• US 20170309003A1
• Filed: 04/26/2016
• Published: 10/26/2017
• Est. Priority Date: 04/26/2016
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Bako, Steve Christopher, Darabi, Aliakbar, Wang, Jue, Shechtman, Elya, Sunkavalli, Kalyan K.
• Granted Patent: US 9,852,498 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 10/507 Summing image-intensity values; Histogram projection analysis
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 10/60 relating to illumination properties, e.g. using a reflectance or lighting model
• CPC Class Codes: G06V 30/40 Document-oriented image-based pattern recognition

Abstract

Techniques for removing artifacts, such as shadows, from document images are described. A shadow map is generated for a digital image by first determining local background colors using clusters of local pixel intensities. Then, a global reference background color is selected from all pixel intensities of the digital image. Next, a per-pixel scaling factor is determined that maps the local background colors to the global reference background color, which applies localized adjustment to the digital image to remove local shadow.

Claims

1. In a digital image editing environment, a method for removing artifacts from digital images, the method comprising:

accessing a digital image;

calculating a shadow map for the digital image, the calculating including;

clustering pixel intensities of multiple blocks of the digital image into two or more clusters;

determining local background colors for each of the multiple blocks of the digital image based on the two or more clusters of each of the multiple blocks;

clustering pixel intensities representing the entire digital image into two or more clusters;

determining global background colors for the digital image based on the two or more clusters representing the entire digital image;

assigning a background color of the digital image as a global reference background color based at least in part on the determined global background colors; and

generating a per-pixel scaling factor to normalize the local background colors by the global reference background color, the per-pixel scaling factor mapping the local background colors of the digital image to the global reference background color; and

applying localized adjustment to pixels of the digital image to remove artifacts from the digital image using the per-pixel scaling factor.

2. A method as described in claim 1, wherein the digital image is of a paper document.

3. A method as described in claim 1, wherein at least one of the clusters for each block of the digital image is based on text color intensities, and another cluster for each block of the digital image is based on background color intensities.

4. A method as described in claim 1, wherein the blocks of the digital image at least partially overlap one another.

5. A method as described in claim 1, wherein the determining of the local background colors for blocks of the digital image further comprises generating a histogram of color intensities within each respective said block.

6. A method as described in claim 5, wherein the histogram of color intensities is generated from a random sample of pixel color intensities from each respective said block.

7. A method as described in claim 5, wherein the histogram of color intensities is generated from pixel color intensities sampled at a specific stride within each respective said block.

8. A method as described in claim 1, wherein the clustering of the pixel intensities of the blocks includes using a Gaussian Mixture Model (GMM).

9. A method as described in claim 1, further comprising:

receiving a user input defining a mask of a region of the digital image;

omitting the region in calculating the shadow map; and

after calculating the shadow map, interpolating the shadow map to compensate for the masked region.

10. A method as described in claim 1, further comprising:

automatically detecting an object within the digital image;

omitting an area in the digital image encompassing the picture in calculating the shadow map; and

after calculating the shadow map, interpolating the shadow map to compensate for the area encompassing the picture.

11. A computing device comprising:

one or more processors; and

one or more computer-readable media having instructions stored thereon that, responsive to execution by the one or more processors, causes the one or more processors to implement a shadow removal module of an image editing application configured to perform operations comprising;

accessing a digital image;

dividing the digital image into multiple local blocks;

clustering pixel intensities of each of the local blocks into two or more clusters, the two or more clusters corresponding at least to a graphics color and a background color within each of the local blocks;

clustering pixel intensities of the entire digital image into two or more clusters corresponding to at least to a graphics color and a global background color of the digital image;

determining a global reference background color from color intensities in the cluster corresponding to the global background color of the digital image by comparing the color intensities to a representative background color;

generating a per-pixel scaling factor that maps pixels of the background colors in the local blocks to the global reference background color; and

applying localized adjustment of color intensity to the pixels of the background colors in the local blocks to remove artifacts from the digital image using the per-pixel scaling factor.

12. A computing device as recited in claim 11, wherein the representative background color is a cluster mean of the global background color cluster of the digital image.

13. A computing device as recited in claim 11, wherein the representative background color is a most frequently occurring color intensity from the digital image.

14. A computing device as recited in claim 11, wherein the clustering pixel intensities of the multiple local blocks and the clustering pixel intensities of the entire digital image are performed using GMMs fit with Expectation-Maximization (EM) and initialized with k-means clustering.

15. A computing device as recited in claim 11, wherein the clustering pixel intensities of the multiple local blocks and the clustering pixel intensities of the entire digital image further comprise forming an additional cluster corresponding to another text color.

16. A computing device as recited in claim 11, wherein the background color within each of the local blocks and the global background color of the digital image correspond to clusters having higher color intensities than the text color within each of the local blocks and the text color of the entire digital image, respectively.

17. In a digital image editing environment, a method implemented by a computing device, the method comprising:

accessing, by the computing device, a digital image in an image editing application;

dividing, by the computing device, the digital image into multiple local blocks;

generating, by the computing device, histograms for each of the multiple local blocks, the histograms representing a sample of a number of pixels of particular color intensities in each of the multiple local blocks;

for each of the histograms of the multiple local blocks, using a Gaussian Mixture Model (GMM) by the computing device to cluster the histograms into two or more clusters;

determining, by the computing device, pixels of the digital image that make up background intensities by upsampling pixels from one of the clusters of each of the multiple local blocks;

generating, by the computing device, an aggregated histogram for multiple blocks of the digital image, the aggregated histogram representing a number of pixels of particular color intensities in all of the multiple blocks combined;

using, by the computing device, the GMM to cluster the aggregated histogram into two or more clusters;

determining, by the computing device, a global reference background color from the particular color intensities of the multiple blocks by comparing the each of the particular color intensities of the multiple blocks to a representative background color of the digital image;

generating, by the computing device, a shadow map that comprises a per-pixel scaling factor that maps the pixels of the digital image that make up background intensities to the global reference background color; and

applying, by the computing device, localized adjustment to the pixels of the digital image that make up background intensities to remove artifacts from the digital image using the per-pixel scaling factor.

18. A method as recited in claim 17, wherein the multiple local blocks at least partially overlap one another.

19. A method as recited in claim 17, wherein the one cluster of each of the multiple local blocks used to determine background intensities is a cluster with a highest cluster mean amongst the two or more clusters for each of the respective multiple blocks.

20. A method as recited in claim 17, wherein the representative background color is a cluster mean of a cluster having highest color intensities amongst the two or more clusters of the aggregated histogram.