On-device artificial intelligence systems and methods for document auto-rotation

• US 10,616,443 B1
• Filed: 02/11/2019
• Issued: 04/07/2020
• Est. Priority Date: 02/11/2019
• Status: Active Grant

• Current Assignee: Open Text Sa Ulc (Open Text Corporation)
• Original Assignee: Open Text Sa Ulc (Open Text Corporation)
• Inventors: Lund, Christopher Dale
• Primary Examiner(s): Barnes, Ted W
• Application Number:
  US16/272,863
• Time in Patent Office: 421 Days
• Field of Search: None
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06V 10/242 by image rotation, e.g. by 90 degrees
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 30/153 using recognition of characters or words
• CPC Class Codes: G06V 30/19173 Classification techniques
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: H04N 1/3877 Image rotation
• CPC Class Codes: H04N 1/40012 Conversion of colour to monochrome

Abstract

An auto-rotation module having a single-layer neural network on a user device can convert a document image to a monochrome image having black and white pixels and segment the monochrome image into bounding boxes, each bounding box defining a connected segment of black pixels in the monochrome image. The auto-rotation module can determine textual snippets from the bounding boxes and prepare them into input images for the single-layer neural network. The single-layer neural network is trained to process each input image, recognize a correct orientation, and output a set of results for each input image. Each result indicates a probability associated with a particular orientation. The auto-rotation module can examine the results, determine what degree of rotation is needed to achieve a correct orientation of the document image, and automatically rotate the document image by the degree of rotation needed to achieve the correct orientation of the document image.

Claims

1. A method, comprising:

converting, by an auto-rotation module on a user device, a document image to a monochrome image having black and white pixels;

segmenting, by the auto-rotation module, the monochrome image into bounding boxes, the segmenting including generating a bounding box defining a connected segment of black pixels in the monochrome image;

determining, by the auto-rotation module, textual snippets from the bounding boxes;

preparing, by the auto-rotation module, the textual snippets for input to a single-layer neural network;

processing the textual snippets by the auto-rotation module using the single-layer neural network, wherein the single-layer neural network is trained to recognize a correct orientation of each textual snippet and output a set of results for the each textual snippet, each result of the set of results indicating a probability of a degree of rotation needed to achieve the correct orientation of the each textual snippet;

examining, by the auto-rotation module, results from the processing, the examining comprising comparing the probability with a threshold;

determining, by the auto-rotation module based on a majority of the results from the examining, a correct orientation of the document image, the majority of the results indicating a degree of rotation needed to achieve the correct orientation of the document image; and

automatically rotating, by the auto-rotation module, the document image by the degree of rotation to achieve the correct orientation of the document image.

2. The method according to claim 1, wherein the bounding box defines a set of coordinates in the monochrome image that contains the connected segment of black pixels.

3. The method according to claim 1, wherein determining textual snippets from the bounding boxes comprises:

determining a width-to-height ratio or size for each snippet of the monochrome image defined by a bounding box;

determining whether the width-to-height ratio or size for the each snippet falls outside of a predetermined range; and

responsive to the width-to-height ratio or size for the each snippet falling outside of the predetermined range, discarding the each snippet.

4. The method according to claim 1, wherein preparing the textual snippets for input to a single-layer neural network comprises scaling all the textual snippets to an input size of the single-layer neural network.

5. The method according to claim 4, wherein the input size is 24×

24 pixels.

6. The method according to claim 1, wherein the single-layer neural network is configured for outputting, for each textual snippet, a probability that a zero degree of rotation is needed to achieve the correct orientation of the each textual snippet, a probability that a 90 degree of rotation is needed to achieve the correct orientation of the each textual snippet, a probability that a 180 degree of rotation is needed to achieve the correct orientation of the each textual snippet, and a probability that a 270 degree of rotation is needed to achieve the correct orientation of the each textual snippet.

7. The method according to claim 1, wherein each textual snippet of the textual snippets has a high likelihood of containing at least a letter.

8. An apparatus, comprising:

a processor;

a non-transitory computer-readable medium; and

stored instructions translatable by the processor for implementing an auto-rotation module on the apparatus, the auto-rotation module having a single-layer neural network and configured for;

converting a document image to a monochrome image having black and white pixels;

segmenting the monochrome image into bounding boxes, the segmenting including generating a bounding box defining a connected segment of black pixels in the monochrome image;

determining textual snippets from the bounding boxes;

preparing the textual snippets for input to the single-layer neural network;

processing the textual snippets using the single-layer neural network, wherein the single-layer neural network is trained to recognize a correct orientation of each textual snippet and output a set of results for the each textual snippet, each result of the set of results indicating a probability of a degree of rotation needed to achieve the correct orientation of the each textual snippet;

examining results from the processing, the examining comprising comparing the probability with a threshold;

determining, based on a majority of the results from the examining, a correct orientation of the document image, the majority of the results indicating a degree of rotation needed to achieve the correct orientation of the document image; and

automatically rotating the document image by the degree of rotation to achieve the correct orientation of the document image.

9. The apparatus of claim 8, wherein the bounding box defines a set of coordinates in the monochrome image that contains the connected segment of black pixels.

10. The apparatus of claim 8, wherein determining textual snippets from the bounding boxes comprises:

determining a width-to-height ratio or size for each snippet of the monochrome image defined by a bounding box;

determining whether the width-to-height ratio or size for the each snippet falls outside of a predetermined range; and

responsive to the width-to-height ratio or size for the each snippet falling outside of the predetermined range, discarding the each snippet.

11. The apparatus of claim 8, wherein preparing the textual snippets for input to a single-layer neural network comprises scaling all the textual snippets to an input size of the single-layer neural network.

12. The apparatus of claim 11, wherein the input size is 24×

24 pixels.

13. The apparatus of claim 8, wherein the single-layer neural network is configured for outputting, for each textual snippet, a probability that a zero degree of rotation is needed to achieve the correct orientation of the each textual snippet, a probability that a 90 degree of rotation is needed to achieve the correct orientation of the each textual snippet, a probability that a 180 degree of rotation is needed to achieve the correct orientation of the each textual snippet, and a probability that a 270 degree of rotation is needed to achieve the correct orientation of the each textual snippet.

14. The apparatus of claim 8, wherein each textual snippet of the textual snippets has a high likelihood of containing at least a letter.

15. A computer program product comprising a non-transitory computer-readable medium storing instructions translatable by a processor of a user device for implementing an auto-rotation module having a single-layer neural network on the user device, the instructions when translated by the processor perform:

converting a document image to a monochrome image having black and white pixels;

segmenting the monochrome image into bounding boxes, the segmenting including generating a bounding box defining a connected segment of black pixels in the monochrome image;

determining textual snippets from the bounding boxes;

preparing the textual snippets for input to the single-layer neural network;

processing the textual snippets using the single-layer neural network, wherein the single-layer neural network is trained to recognize a correct orientation of each textual snippet and output a set of results for the each textual snippet, each result of the set of results indicating a probability of a degree of rotation needed to achieve the correct orientation of the each textual snippet;

examining results from the processing, the examining comprising comparing the probability with a threshold;

determining, based on a majority of the results from the examining, a correct orientation of the document image, the majority of the results indicating a degree of rotation needed to achieve the correct orientation of the document image; and

automatically rotating the document image by the degree of rotation to achieve the correct orientation of the document image.

16. The computer program product of claim 15, wherein the bounding box defines a set of coordinates in the monochrome image that contains the connected segment of black pixels.

17. The computer program product of claim 15, wherein determining textual snippets from the bounding boxes comprises:

determining a width-to-height ratio or size for each snippet of the monochrome image defined by a bounding box;

determining whether the width-to-height ratio or size for the each snippet falls outside of a predetermined range; and

responsive to the width-to-height ratio or size for the each snippet falling outside of the predetermined range, discarding the each snippet.

18. The computer program product of claim 15, wherein preparing the textual snippets for input to a single-layer neural network comprises scaling all the textual snippets to an input size of the single-layer neural network.

19. The computer program product of claim 18, wherein the input size is 24×

24 pixels.

20. The computer program product of claim 15, wherein the single-layer neural network is configured for outputting, for each textual snippet, a probability that a zero degree of rotation is needed to achieve the correct orientation of the each textual snippet, a probability that a 90 degree of rotation is needed to achieve the correct orientation of the each textual snippet, a probability that a 180 degree of rotation is needed to achieve the correct orientation of the each textual snippet, and a probability that a 270 degree of rotation is needed to achieve the correct orientation of the each textual snippet.