SYSTEM AND METHOD FOR CAPTURING AND PROCESSING BUSINESS DATA

• US 20100054605A1
• Filed: 11/11/2009
• Published: 03/04/2010
• Est. Priority Date: 05/18/2005
• Status: Active Grant

• Current Assignee: Mobilescan Incorporated
• Original Assignee: scanR, Inc.
• Inventors: Mutz, Andrew H., Molnar, Joseph, Ferreira, Paulo, Nieuwland, Dan
• Granted Patent: US 7,933,453 B2
• Time in Patent Office: Days
• US Class Current: 382/190
• CPC Class Codes: G06V 10/95 structured as a network, e.g. client-server architectures
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/274 Syntactic or semantic context, e.g. balancing
• CPC Class Codes: G06V 30/416 Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors
• CPC Class Codes: H04M 1/2755 by optical scanning
• CPC Class Codes: H04M 2250/52 including functional features of a camera constructional features concerning the integration of cameras in portable phones H04M1/0264

Abstract

A method and a system for interpreting information in a document are provided, with the system receiving an image of a document from a remote source and converting it into multiple sets of blocks of characters. Tags indicating likely meaning of blocks are assigned to them. At least some of the blocks have an associated score representing the probability that the characters in the block correctly represent the characters in the original image. The system selects one set from multiple sets based on the scores associated to certain blocks determined by accessing remote information over the Internet.

Claims

1. A system for use in interpreting information in a document, comprising:

a mobile device having a camera, the mobile device performs actions, including;

receiving a user input identifying a type of information to be captured using the camera;

automatically adjusting capturing information of the mobile device and camera based on the user input of identified type of information;

capturing an image identified by the user and corresponding to the identified type of information to be captured;

providing the captured image over a network; and

a server device that performs actions, including;

receiving the captured image;

performing an image enhancement activity on the received image based on a value of a capture flag (CF);

based further on the value of the CF, optionally performing an optical character recognition (OCR) or handwriting recognition (ICR) operation on the received image to generate a result with textual information; and

if the result is determined to include textual information, performing a classification of the result with textual information to interpret the included textual information.

2. The system of claim 1, wherein the mobile device performs actions, further including:

based on the selected type of information, generating the corresponding CF;

after capturing the image, pre-processing the captured image in accordance with the CF, wherein at least a portion of the pre-processing comprises tagging the image; and

wherein the pre-processing further comprises at least one of removing a surrounding field of the captured image, selecting at least a portion of the image based on a motion blurring technique, performing a deskewing operation on the image, or removing barrel distortion from the captured image.

3. The system of claim 2, wherein the CF includes additional information that identifies a type of hardware or software associated with the mobile device or camera.

4. The system of claim 1, wherein performing an image enhancement activity further comprises:

based on the CF, if the received image is determined at least one of a business card, a receipt, a generic document, or handwritten notes that have not been pre-processed at the mobile device, then, performing the following actions at the server device;

converting the image into a grayscale image;

applying a segmentation algorithm to the grayscale image to separate bright areas from dark background areas;

applying a multiple erosion-dilution filter to the resulting segmented image to remove pepper noise or to smooth outside edges;

forming a board masking image; and

masking the image.

5. The system of claim 1, wherein performing an image enhancement activity further comprises:

based on the CF, if the received image is determined at least one of a business card, a receipt, a generic document, or handwritten notes that has been pre-processed at the mobile device, then, performing the following actions at the server device;

performing a shadow and lightning gradient correction to the image;

performing a background and foreground segmentation on the corrected image; and

removing background noise from the segmented image.

6. The system of claim 1, wherein performing an image enhancement activity further comprises:

based on the CF, if the received image is determined a white board image, then, performing the following actions at the server device;

performing a shadow and lightness gradient correction to the captured image;

performing a background and foreground segmentation on the image;

performing contrast detection and correction on the segmented image;

smoothing at least a portion of the image signal in a vicinity of an average background level; and

suppressing a saturation of pixels in a vicinity of a white background on the image.

7. The system of claim 1, wherein the classification of the textual information further comprises:

interpreting the textual information to isolate at least one of a name of a business, a name of a person, a telephone number, or an email address;

performing the interpretation on a plurality of duplicate versions of the textual information, where each resulting interpretation includes a metric indicating a quality of the interpretation; and

selecting the resulting interpretation having a highest metric.

8. A computer-readable storage medium that includes data and instructions, wherein the execution of the instructions on a server device provides for interpreting information in a document by enabling actions, comprising:

receiving, from a remote device, a captured image and a value of a capture flag (CF) that includes at least an indication of a type of professional information within the captured image, and information about a capturing device from which the image was captured;

performing an image enhancement activity on the received image based on a value of a capture flag (CF);

based further on the value of the CF, optionally performing an optical character recognition (OCR) or handwriting recognition (ICR) operation on the received image to generate a result with textual information; and

if the result is determined to include textual information, performing a classification of the result with textual information to interpret the included textual information.

9. The computer-readable storage medium of claim 8, wherein the type of professional information includes one of a business card, a receipt, a form, a printed document, a handwritten note, a whiteboard image, or a graphic indicia.

10. The computer-readable storage medium of claim 8, wherein performing an image enhancement activity further comprises:

examining the CF; and

based on examination of the CF, if the received image is determined at least one of a business card, a receipt, a generic document, or handwritten notes that have not been pre-processed at the mobile device, then, performing the following actions at the server device;

converting the image into a grayscale image;

applying a segmentation algorithm to the grayscale image to separate bright areas from dark background areas;

applying a multiple erosion-dilution filter to the resulting segmented image to remove pepper noise or to smooth outside edges;

forming a board masking image; and

masking the image.

11. The computer-readable storage medium of claim 8, wherein performing an image enhancement activity further comprises:

examining the CF; and

based on examination of the CF, if the received image is determined at least one of a business card, a receipt, a generic document, or handwritten notes that has been pre-processed at the mobile device, then, performing the following actions at the server device;

performing a shadow and lightning gradient correction to the image;

performing a background and foreground segmentation on the corrected image; and

removing background noise from the segmented image.

12. The computer-readable storage medium of claim 8, wherein performing an image enhancement activity further comprises:

examining the CF; and

based on examination of the CF, if the received image is determined a white board image, then, performing the following actions at the server device;

performing a shadow and lightness gradient correction to the image;

performing a background and foreground segmentation on the corrected image;

performing contrast detection and correction on the segmented image;

smoothing at least a portion of the image signal in a vicinity of an average background level; and

suppressing a saturation of pixels in a vicinity of a white background on the image.

13. The computer-readable storage medium of claim 8, wherein performing a classification further comprises:

receiving the textual information from the OCR or ICR operation;

generating a set of tokenizations, wherein a token is a set of characters that can be manipulated as a single entity, and wherein the set of tokenizations represent possible combinations of tokens using character alternatives from the OCR or ICR operations;

generating from the set of tokenizations, a tagged set of tokenizations, the tags representing possible interpretations of the tokens;

performing a syntactic parsing on each tokenization in the set of tokenizations based on a determined grammar to obtain multiple tokenizations with a single tag assigned to each token; and

selecting as a final tokenization one tokenization with tags based on an aggregate score from the multiple tokenizations as interpreted textual content.

14. The computer-readable storage medium of claim 8, wherein a grammar dictionary is employed to interpret the text information.

15. A system that is configured to interpret information in a document, comprising:

a memory for receiving and storing a captured image, and for storing computer-executable instructions; and

a processor that executes a plurality of computer-executable instructions on the captured image to perform actions, including;

receiving, from a remote device, the captured image and a value of a capture flag (CF) that includes at least an indication of a type of professional information within the captured image, and information about a capturing device from which the image was captured;

performing an image enhancement activity on the received image based on a value of a capture flag (CF);

based further on the value of the CF, optionally performing an optical character recognition (OCR) or handwriting recognition (ICR) operation on the received image to generate a result with textual information; and

if the result is determined to include textual information, performing a classification of the result with textual information to interpret the included textual information.

16. The system of claim 15, wherein the remote device is a mobile handheld device having a camera that is configured to perform actions, including:

receiving an input selection from a user that identifies the type of professional information;

adjusting the camera or the mobile handheld device based on the selection of type of professional information;

capturing the image based on the adjustments; and

pre-processing the captured image, wherein the pre-processing is further adjusted for the selected type of professional information.

17. The system of claim 16, wherein the pre-processing of the captured image further comprises at least one of removing a surrounding field of the image, selecting at least a portion of the image based on a motion blurring technique, performing a deskewing operation on the image, or removing barrel distortion from the captured image.

18. The system of claim 15, wherein performing an image enhancement activity further comprises:

based on the CF, if the received image is determined at least one of a business card, a receipt, a generic document, or handwritten notes that have not been pre-processed at the mobile device, then, performing the following actions at the server device;

converting the image into a grayscale image;

applying a segmentation algorithm to the grayscale image to separate bright areas from dark background areas;

applying a multiple erosion-dilution filter to the resulting segmented image to remove pepper noise or to smooth outside edges;

forming a board masking image; and

masking the image.

19. The system of claim 15, wherein performing an image enhancement activity further comprises:

based on the CF, if the received image is determined at least one of a business card, a receipt, a generic document, or handwritten notes that has been pre-processed at the mobile device, then, performing the following actions at the server device;

performing a shadow and lightning gradient correction to the image;

performing a background and foreground segmentation on the corrected image; and

removing background noise from the segmented image.

20. The system of claim 15, wherein the classification of the textual information further comprises:

interpreting the textual information to isolate at least one of a name of a business, a name of a person, a telephone number, or an email address;

performing the interpretation on a plurality of duplicate versions of the textual information, where each resulting interpretation includes a metric indicating a quality of the interpretation; and

selecting a resulting interpretation having a highest metric.