ENHANCING DOCUMENTS PORTRAYED IN DIGITAL IMAGES

• US 20180024974A1
• Filed: 07/24/2017
• Published: 01/25/2018
• Est. Priority Date: 07/22/2016
• Status: Active Grant

• Current Assignee: Dropbox, Inc.
• Original Assignee: Dropbox, Inc.
• Inventors: Welinder, Nils Peter, Belhumeur, Peter N., Xiong, Ying, Baek, Jongmin, Kozlov, Simon, Berg, Thomas, Kriegman, David J.
• Granted Patent: US 10,628,519 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/126 Job scheduling, e.g. queuing, determine appropriate device
• CPC Class Codes: G06F 40/103 Formatting, i.e. changing of presentation of documents automatic justification G06F40/189; automatic line break hyphenation G06F40/191
• CPC Class Codes: G06F 40/106 Display of layout of documents; Previewing
• CPC Class Codes: G06F 40/123 Storage facilities
• CPC Class Codes: G06F 40/166 Editing, e.g. inserting or deleting
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/197 Version control for software G06F8/71
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/20056 Discrete and fast Fourier transform, [DFT, FFT]
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30176 Document
• CPC Class Codes: G06T 2210/22 Cropping
• CPC Class Codes: G06T 5/60 using machine learning, e.g. neural networks
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/90 Dynamic range modification of images or parts thereof
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06V 10/10 : Image acquisition document image scanning and transmission H04N1/00; control of digital cameras H04N23/60
• CPC Class Codes: G06V 10/242 : by image rotation, e.g. by 90 degrees
• CPC Class Codes: G06V 10/30 : Noise filtering
• CPC Class Codes: G06V 10/44 : Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 20/46 : Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 30/40 : Document-oriented image-based pattern recognition
• CPC Class Codes: G06V 30/413 : Classification of content, e.g. text, photographs or tables
• CPC Class Codes: G06V 30/414 : Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: G06V 30/418 : Document matching, e.g. of document images

Abstract

The present disclosure is directed toward systems and methods that efficiently and effectively generate an enhanced document image of a displayed document in an image frame captured from a live image feed. For example, systems and methods described herein apply a document enhancement process to a displayed document in an image frame that result in an enhanced document image that is cropped, rectified, un-shadowed, and with dark text against a mostly white background. Additionally, systems and method described herein determine whether a stored digital content item includes a displayed document. In response to determining that a stored digital content item does include a displayed document, systems and methods described herein generate an enhanced document image of a displayed document included in the stored digital content item.

Claims

1. A computing device comprising:

at least one processor; and

a non-transitory computer-readable medium storing instructions thereon that, when executed by the at least one processor, cause the computing device to;

detect a displayed document within a live image feed associated with the computing device;

based on detecting the displayed document within the live image feed, generate an enhanced document image corresponding to the displayed document; and

provide, for presentation on a display of the computing device, the enhanced document image.

2. The computing device as recited in claim 1, wherein the non-transitory computer-readable medium further comprises instructions thereon that, when executed by the at least one processor, cause the computing device to:

capture an image frame from the live image feed, the image frame comprising the displayed document; and

wherein generating the enhanced document comprises modifying the image frame with respect to the displayed document within the image frame.

3. The computing device as recited in claim 2, wherein modifying the image frame comprises:

detecting, without receiving user input, portions of the image frame that are not part of the displayed document; and

cropping the image frame to remove the portions of the image frame that are not part of the displayed document.

4. The computing device as recited in claim 3, wherein generating the enhanced document further comprises altering the displayed document within the cropped imaged frame.

5. The computing device of claim 4, wherein altering the displayed document comprises at least one of:

rectifying the displayed document, converting the displayed document to grayscale, or denoising the displayed document.

6. The computing device as recited in claim 4, wherein altering the displayed document comprises correcting a background of the displayed document.

7. The computing device as recited in claim 6, wherein correcting the background of the displayed document comprises:

creating a subsampled version of the displayed document; and

optimizing the subsampled version of the displayed document by solving an objective function that penalizes deviations from white within the subsampled version and penalizes deviations in gradient within the subsampled version to generate an optimized subsampled version.

8. The computing device as recited in claim 7, wherein the non-transitory computer-readable medium further comprises instructions thereon that, when executed by the at least one processor, cause the computing device to:

perform a Fourier Domain transfer of the subsampled version of the displayed document;

solve the objective function in the Fourier Domain; and

perform an inverse Fourier Domain transfer to generate the optimized subsampled version of the displayed document.

9. The computing device as recited in claim 7, wherein the non-transitory computer-readable medium further comprises instructions thereon that, when executed by the at least one processor, cause the computing device to upsample the optimized subsampled version of the displayed document to generate a tri-map version of the displayed document that identifies background pixels, foreground pixels, and unknown pixels.

10. The computing device as recited in claim 9, wherein the non-transitory computer-readable medium further comprises instructions thereon that, when executed by the at least one processor, cause the computing device to assign each of the unknown pixels as either a background pixel or a foreground pixel by estimating a background color of each of the unknown pixels.

11. The computing device as recited in claim 10, wherein the non-transitory computer-readable medium further comprises instructions thereon that, when executed by the at least one processor, cause the computing device to convert the enhanced document image to a document file format.

12. A non-transitory computer-readable medium storing instructions thereon that, when executed by at least one processor, cause a computer system to:

detect a displayed document within a live image feed associated with the computing device;

based on detecting the displayed document within the live image feed, generate an enhanced document image corresponding to the displayed document; and

provide, for presentation on a display of the computer system, the enhanced document image.

13. The non-transitory computer-readable medium recited in claim 12, further comprising instructions that, when executed by the at least one processor, cause the computer system to:

receive user input indicating one or more edits to the enhanced document image;

modify the enhanced document image in accordance with the one or more edits; and

convert the enhanced document image to a document file format.

14. The non-transitory computer-readable medium recited in claim 12, wherein generating the enhanced document image comprises altering a border of the displayed document to create a rectangular enhanced document image.

15. The non-transitory computer-readable medium recited in claim 12, wherein generating the enhanced document comprises:

converting the displayed document from a color version to a grayscale version; and

recoloring the displayed document prior to providing the enhanced document image.

16. A method comprising:

receiving, at an online content management system and from a client device, a digital content item;

determining, by at least one processor, that the digital content item comprises a displayed document;

associating metadata with the digital content item that designates the digital content item as comprising the displayed document;

based on the metadata associated with the digital content item, providing a selectable graphical element associated with the digital content item within a graphical user interface on the client device, wherein the selectable graphical element indicates that the digital content item comprises the displayed document; and

in response to receiving an indication of a user selection of the selectable graphical element, generating an enhanced document image for the displayed document.

17. The method as recited in claim 16, wherein generating an enhanced document image for the displayed document within the digital content item comprises one or more of:

cropping the digital content item with respect to the displayed document;

rectifying the displayed document;

converting the displayed document to grayscale;

or denoising the displayed document.

18. The method as recited in claim 16, further comprising:

providing the enhanced document image to the client device; and

storing the enhanced document image on the online content management system in an account associated with the client device.

19. The method as recited in claim 18, further comprising:

training an image classifier; and

wherein determining that the digital content item comprises the displayed document comprises analyzing the digital content item using the trained image classifier.

20. The method as recited in claim 19, wherein building and training the image classifier comprises:

building a convolutional neural network comprising a plurality of layers; and

training, utilizing a training data set, the convolutional neural network to determine whether an image includes a displayed document.