Image document processing in a client-server system including privacy-preserving text recognition

• US 9,847,974 B2
• Filed: 04/28/2016
• Issued: 12/19/2017
• Est. Priority Date: 04/28/2016
• Status: Expired due to Fees

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Kompalli, Pramod Sankar
• Primary Examiner(s): Chawan, Sheela C
• Application Number:
  US15/140,795
• Publication Number:
  US 20170317983A1
• Time in Patent Office: 600 Days
• Field of Search: 382100, 382155, 382162, 382168, 382173, 382177, 382181, 382232, 382254, 382276, 382309, 382312, 382325
• CPC Class Codes: G06F 18/24133 Distances to prototypes
• CPC Class Codes: G06F 21/6209 to a single file or object, e.g. in a secure envelope, encrypted and accessed using a key, or with access control rules appended to the object itself
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 3/088 Non-supervised learning, e.g. competitive learning
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/19173 Classification techniques
• CPC Class Codes: G06V 30/40 Document-oriented image-based pattern recognition

Abstract

Disclosed are devices and methods for processing an image document in a client-server environment such that privacy of text information contained in the image document is preserved. Specifically, in a client-server environment, an image document can be processed using a local computerized device of a client to create an obfuscated document by identifying word images in the image document and scrambling those word images. The obfuscated document can be received by a server of a service provider over a network (e.g., the Internet) and processed by previously trained software (e.g., a previously trained convolutional neural network (CNN)) to recognize specific words represented by the scrambled images in the obfuscated document without having to reconstruct the image document. Since the image document is neither communicated over the network, nor reconstructed and stored on the server, privacy concerns are minimized.

Claims

1. A computerized device comprising:

a network interface connected to a network; and

a processor in communication with the network interface and performing the following;

analyzing an image document to identify at least one text region within the image document and at least one word image contained in the at least one text region;

for each word image, randomly selecting a shuffling pattern, resizing the word image to a predetermined size so that a height and width of the word image are equal to a height and width of a grid of cells that is associated with the shuffling pattern, overlaying the grid of cells onto the word image such that the cells contain portions of the word image, and shuffling positions of the cells within the grid according to the shuffling pattern to move the portions of the word image and create a corresponding scrambled image; and

replacing all word images in the image document with corresponding scrambled images to generate an obfuscated document,

the network interface communicating the obfuscated document to a computer server over the network, wherein the computer server is capable of individually evaluating the scrambled images using trained software to recognize specific words.

2. The computerized device of claim 1, the image document comprising any of a digital photograph received from a digital camera and a scanned-in document received from a scanner.

3. The computerized device of claim 1, the grid of cells having a first number of columns and a second number of rows, the first number of columns and the second number of rows being sufficient to ensure that a word contained in the word image is incomprehensible to a user that views the scrambled image.

4. The computerized device of claim 1, the shuffling pattern being randomly selected from a set of shuffling patterns.

5. A computer server comprising:

a network interface receiving, from a computerized device over a network, an obfuscated document,

the obfuscated document created by the computerized device from an image document comprising at least one word image, and

the obfuscated document comprising at least one scrambled image, each scrambled image corresponding to a single word image in the image document;

a processor in communication with the network interface and processing the obfuscated document,

the processing comprising evaluating each specific scrambled image individually to recognize a specific word represented by the specific scrambled image, and

the processing being performed without reconstructing the image document; and

a memory storing multiple trained convolutional neural networks,

each trained convolutional neural network having been initially developed using a database of images and then fine-tuned using scrambled word images acquired by scrambling specific word images from a specific vocabulary set using a specific shuffling pattern,

each word in the specific vocabulary set being associated with a corresponding output class number, and

the processor processing the obfuscated document by executing a selected trained convolutional neural network to produce a specific output class number for the specific scrambled image and, thereby to recognize the specific word associated with the specific output class number and represented by the specific scrambled image.

6. The computer server of claim 5, the processor further selecting the selected trained convolutional neural network based on a specific vocabulary set code and a specific shuffling pattern code associated with the specific scrambled image in the obfuscated document.

7. The computer server of claim 5, the processor further evaluating the specific scrambled image to determine the specific vocabulary set and the specific shuffling pattern and selecting the selected trained convolutional neural network based on the specific vocabulary set and the specific shuffling pattern.

8. The computer server of claim 5, the specific vocabulary set comprising multiple different words and multiple instances of each of the different words depicted in different fonts.

9. The computer server of claim 5, the processor generating a response indicating the specific output class number and communicating the response to the computerized device.

10. A method comprising:

analyzing, by a processor of a computerized device, an image document to identify at least one text region and at least one word image contained in the at least one text region;

for each word image, performing, by the processor, the following;

randomly selecting a shuffling pattern, resizing the word image to a predetermined size so that a height and width of the word image are equal to a height and width of a grid of cells that is associated with the shuffling pattern, overlaying of the grid of cells onto the word image such that the cells contain portions of the word image, and shuffling of positions of the cells within the grid according to the shuffling patterning to move the portions of the word image and create a corresponding scrambled image;

replacing, by the processor, all word images in the image document with corresponding scrambled images to generate an obfuscated document; and

,

communicating, by a network interface of the computerized device, the obfuscated document to a computer server over a network, wherein the computer server is capable of individually evaluating the scrambled images using trained software to recognize specific words.

11. The method of claim 10, the image document comprising any of a digital photograph received from a digital camera and a scanned-in document received from a scanner.

12. The method of claim 10, the grid of cells having a first number of columns and a second number of cells, the first number of columns and the second number of rows being sufficient to ensure that a word contained in the word image is incomprehensible to a user that views the scrambled image.

13. The method of claim 10, the shuffling pattern being randomly selected from a set of shuffling patterns.

14. A method comprising:

storing, in a memory of a computer server, multiple trained convolutional neural networks,

each trained convolutional neural network having been initially developed using a database of images and then fine-tuned using scrambled word images acquired by scrambling specific word images from a specific vocabulary set using a specific shuffling pattern, and

each word in the specific vocabulary set being associated with a corresponding output class number;

receiving, by a network interface of the computer server from a computerized device over a network, an obfuscated document,

the obfuscated document created by the computerized device from an image document comprising at least one word image, and

the obfuscated document comprising at least one scrambled image, each scrambled image corresponding to a single word image in the image document; and

,

processing, by a processor of the computer server, the obfuscated document without reconstructing the image document, the processing comprising evaluating each specific scrambled image individually to recognize a specific word represented by the specific scrambled image and the processing of the obfuscated document further comprising executing a selected trained convolutional neural network to produce a specific output class number for the specific scrambled image and, thereby to recognize the specific word associated with the specific output class number and represented by the specific scrambled image.

15. The method of claim 14, the processing of the obfuscated document comprising selecting the selected trained convolutional neural network based on a specific vocabulary set code and a specific shuffling pattern code associated with the specific scrambled image in the obfuscated document.

16. The method of claim 14, the processing of the obfuscated document comprising evaluating the specific scrambled image to determine the specific vocabulary set and the specific shuffling pattern and selecting the selected trained convolutional neural network based on the specific vocabulary set and the specific shuffling pattern.

17. The method of claim 14, the specific vocabulary set comprising multiple different words and multiple instances of each of the different words depicted in different fonts.

18. The method of claim 14, further comprising generating a response indicating the specific output class number and communicating the response to the computerized device.