Method and apparatus for populating electronic forms from scanned documents

• US 20040181749A1
• Filed: 03/24/2004
• Published: 09/16/2004
• Est. Priority Date: 01/29/2003
• Status: Active Grant

• Current Assignee: Zhigu Holdings Limited
• Original Assignee: Microsoft Corporation
• Inventors: Chellapilla, Kumar H., Herley, Cormac E., Viola, Paul A., Kristjansson, Trausti T.
• Granted Patent: US 7,305,129 B2
• Time in Patent Office: Days
• US Class Current: 715/505
• CPC Class Codes: G06F 40/174 Form filling; Merging
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/127 with the intervention of an operator
• CPC Class Codes: G06V 30/1444 Selective acquisition, locating or processing of specific regions, e.g. highlighted text, fiducial marks or predetermined fields
• CPC Class Codes: G06V 30/412 Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables

Abstract

A computer-implemented method and apparatus are provided for populating an electronic form from an electronic image. The method and apparatus identify a size, orientation and position of an object within the electronic image, and identify information elements from pixels within the image that correspond to the object. Fields of the electronic form are displayed to a user along with the identified information elements through a graphical user interface. The information elements are parsed into tagged groups of different information types. At least some of the fields of the electronic form are populated with the tagged groups to produce a populated form. The user is allowed to edit the populated fields through the graphical user interface.

Claims

17. A computer-readable medium comprising computer-executable instructions that, when executed by a computer, performs a method comprising:

(a) identifying a size, orientation and position of a first object having any arbitrary orientation within an electronic image;

(b) identifying information elements from pixels within the electronic image that correspond to the first object;

(c) displaying fields of the electronic form and the identified information elements to a user through a graphical user interface; and

(d) parsing the information elements into tagged groups of different information types;

(e) populating the fields of the electronic form with the tagged groups to produce a populated electronic form and allowing the user to edit the populated fields through the graphical user interface.

18. The computer-readable medium of claim 17 wherein the electronic image comprises a plurality of objects, including the first object, and the method further comprises:

(f) performing (a) through (e) for each object, such that, for each object, fields of a corresponding electronic form are at least partially populated with information elements obtained from pixels within that object.

19. The computer-readable medium of claim 18 wherein the plurality of objects comprises objects of different types.

20. The computer-readable medium of claim 17 wherein (a) comprises:

classifying each pixel within the image to produce pixel classification data;

defining an image function to process the pixel classification data;

dividing the image into sub-images based on disparities in the image function; and

processing the sub-images to determine a size, an orientation and a position for each of the objects, including the first object.

21. The computer-readable medium of claim 20 wherein classifying comprises classifying each pixel as one of a data pixel or a background pixel.

22. The computer-readable medium of claim 21 and further comprising:

defining the image function as a sum of data pixels in a certain direction;

calculating the image function in a first direction to generate a first data set;

calculating the image function in a second direction to generate a second data set; and

searching for disparities in the image function in the first direction and the image function in the second direction.

23. The computer-readable medium of claim 17 wherein:

(b) comprises identifying text blocks and two-dimensional locations of the text blocks within the first object using optical character recognition;

(c) comprises displaying the identified text blocks within an image of the first object to the user through the graphical user interface; and

(d) comprises highlighting the tagged groups of different information types within the image of the first object with visual indicators that indicate the different information types.

24. The computer-readable medium of claim 17 wherein:

(d) comprises receiving the information elements as untagged media from an untagged media store, and parsing the untagged media to identify information types of the information elements;

(e) comprises automatically populating at least one of the fields with at least one of the information elements based on the information type of that element; and

(c) comprises displaying the information elements through an object data graphical user interface and displaying the populated fields and any unpopulated fields through a form graphical user interface.

25. The computer-readable medium of claim 24 wherein (c) further comprises employing visual indicator in the object data graphical user interface to indicate that an information element is compatible with a particular field in the form.

26. The computer-readable medium of claim 24 wherein (c) further comprises indicating a status level associated with at least one information element in at least one populated field in the form.

27. The computer-readable medium of claim 26 wherein (c) further comprises prompting the user to verify and/or correct the at least one information element in the at least one populated field based on the indicated status level.

28. The computer-readable medium of claim 27 wherein (d) comprises updating other fields of the populated form based on the user verification and/or correction of the at least one information element.

29. The computer-readable medium of claim 17 and further comprising:

(f) writing side information gleaned from edits made by the user in step (e) to a side information store; and

(g) re-parsing the parsing the information elements into tagged groups of different information types in (d) and re-populating at least one other field in (e) based on the side information.

30. A system for at least partially populating electronic forms, the system comprising:

an object detection and extraction module, which processes pixels in the electronic image to identify a size, orientation and position of an object having any arbitrary orientation within the electronic image;

an optical character recognition module, which identifies information elements from pixels within the electronic image that correspond to the first object;

a graphical user interface, which simultaneously displays fields of the electronic form and the identified information elements to a user; and

a parsing module, which parses the information elements into tagged groups of different information types and at least partially populates the fields with the tagged groups to produce a populated electronic form.

31. The system of claim 30 wherein the graphical user interface allows the user to edit the populated fields and any unpopulated fields.

32. The system of claim 30 wherein the object detection and extraction module comprises:

a data pixel detection module that classifies each pixel of the image and defines an image function;

a segmentation module that is capable of dividing the image into smaller sub-images based on disparities in the image function; and

a single object extraction module that processes the sub-images to detect a number, a size, an orientation and a position of objects within the image.

34. The system of claim 30 wherein:

the optical character recognition module comprises means for identifying text blocks and two-dimensional locations of the text blocks within the first object using optical character recognition; and

the graphical user interface comprises means for displaying the identified text blocks within an image of the first object to the user through the graphical user interface and highlighting the tagged groups of different information types within the image of the first object with visual indicators that indicate the different information types.

35. The system of claim 30 wherein:

the parsing module comprises means for receiving the information elements as untagged media from an untagged media store, parsing the untagged media to identify information types of the information elements, and automatically populating at least one of the fields with at least one of the information elements based on the information type of that element; and

the graphical user interface comprises means for displaying the information elements through an object data graphical user interface and displaying the populated fields and any unpopulated fields through a form graphical user interface.

36. The system of claim 35 wherein the object data graphical user interface comprises a visual indicator to indicate that an information element is compatible with a particular field in the form.

37. The system of claim 35 wherein the form graphical user interface comprises a status level indicator associated with at least one information element in at least one populated field in the form.

38. The system of claim 37 wherein the graphical user interface comprises means for prompting the user to verify and/or correct at least one information element in at least one populated field based on the status level indicator associated with that field.

39. The system of claim 31 wherein the parsing module comprises means for updating other fields of the populated form according to edits made by the user to the populated fields and the unpopulated fields.

33-1. The system of claim 33 wherein the data pixel detection module defines the image function as a sum of data pixels in a certain direction, calculates the image function in a first direction to generate a first data set, calculates the image function in a second direction to generate a second data set, and searches for disparities in the image function in the first direction and the image function in the second direction.

40. A method for populating electronic forms from an electronic image having first and second objects of different information types, the method comprising:

identifying a size, orientation and position of the first and second objects within the electronic image and dividing the electronic image into sub-images corresponding to pixels in the electronic image associated with the size, orientation and position of each object;

performing optical character recognition on each sub-image to identify untagged information elements within the corresponding object;

for each sub-image, parsing the untagged information elements into tagged information elements;

populating fields in a first electronic form type with the tagged information elements identified from the sub-image of the first object to produce a first populated form;

populating fields in a second electronic form type with the tagged information elements identified from the sub-image of the second object to produce a second populated form; and

displaying the first and second populated forms and the untagged information elements to a user through a graphical user interface and allowing the user to edit the first and second populated forms through the graphical user interface.

41. The method of claim 40 wherein:

the first object comprises a business card and the second object comprises a purchase receipt; and

the first electronic form comprises a contact record of a software address book and the second electronic form comprises an electronic financial record of a software financial application.