Methods and apparatus for populating electronic forms from scanned documents

• US 7,305,129 B2
• Filed: 03/24/2004
• Issued: 12/04/2007
• Est. Priority Date: 01/29/2003
• Status: Active Grant

• Current Assignee: Zhigu Holdings Limited
• Original Assignee: Microsoft Corporation
• Inventors: Chellapilla, Kumar H., Herley, Cormac E., Viola, Paul A., Kristjansson, Trausti T.
• Primary Examiner(s): Werner; Brian
• Assistant Examiner(s): RASHID, DAVID
• Application Number:
  US10/808,194
• Publication Number:
  US 20040181749A1
• Time in Patent Office: 1,350 Days
• Field of Search: None
• US Class Current: 382/174
• CPC Class Codes: G06F 40/174 Form filling; Merging
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/127 with the intervention of an operator
• CPC Class Codes: G06V 30/1444 Selective acquisition, locating or processing of specific regions, e.g. highlighted text, fiducial marks or predetermined fields
• CPC Class Codes: G06V 30/412 Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables

Abstract

A computer-implemented method and apparatus are provided for populating an electronic form from an electronic image. The method and apparatus identify a size, orientation and position of an object within the electronic image, and identify information elements from pixels within the image that correspond to the object. Fields of the electronic form are displayed to a user along with the identified information elements through a graphical user interface. The information elements are parsed into tagged groups of different information types. At least some of the fields of the electronic form are populated with the tagged groups to produce a populated form. The user is allowed to edit the populated fields through the graphical user interface.

Claims

1. A computer-implemented method for populating an electronic form from an electronic image, the method comprising:

(a) identifying a size, orientation and position of a first object having any arbitrary orientation within the electronic image;

(b) identifying information elements from pixels within the electronic image that correspond to the first object, including identifying text blocks within the first object using optical character recognition;

(c) displaying simultaneously to a user fields of the electronic form in a form data area and the identified text blocks in an object data area that is outside of the form data area, which corresponds to the first object, through a graphical user interface, wherein the text blocks are selectable by the user within the object data area through the graphical user interface for insertion into respective fields of the electronic form in the form data area;

(d) parsing the information elements into tagged groups of different information types;

(e) automatically populating the fields of the electronic form with the tagged groups to produce a populated form and allowing the user to edit the populated fields through the graphical user interface; and

(f) providing a visual status indicator adjacent each field of the form data area alerting the user that the field is unfilled and unverified, filled but unverified, and filled and verified, the status being based on the automatic populating and user editing.

2. The method of claim 1 wherein (a) comprises identifying a size, orientation and position of the first object among a plurality of objects within the electronic image.

3. The method of claim 1 wherein (a) comprises:

classifying each pixel within the image to produce pixel classification data;

defining an image function to process the pixel classification data;

dividing the image into sub-images based on disparities in the image function; and

processing the sub-images to determine a size, an orientation and a position for each of the objects, including the first object.

4. The method of claim 3 and further comprising repeating the classifying pixels, defining an image function, and dividing of the image until the image contains a single object or the image can no longer be divided.

5. The method of claim 3 wherein classifying comprises classifying each pixel as one of a data pixel or a background pixel.

6. The method of claim 5 and further comprising:

defining the image function as a sum of data pixels in a certain direction;

calculating the image function in a first direction to generate a first data set;

calculating the image function in a second direction to generate a second data set; and

searching far disparities in the image function in the first direction and the image function in the second direction.

7. The method of claim 1 wherein:

(b) comprises identifying two-dimensional locations of the text blocks within the first object.

8. The method of claim 7 wherein:

(c) comprises displaying to the user the identified information elements in the object data area within an image of the first object through the graphical user interface; and

(d) comprises highlighting the tagged groups of different information types within the image of the first object in the object data area with visual indicators that indicate the different information types.

9. The method of claim 1 wherein:

(d) comprises receiving the information elements as untagged media from an untagged media store, and parsing the untagged media to identify information types of the information elements;

(c) comprises automatically populating at least one of the fields with at least one of the information elements based on the information type of that element; and

(c) comprises displaying the information elements through an object data graphical user interface and displaying the populated fields and any unpopulated fields through a form graphical user interface.

10. The method of claim 9 wherein (c) further comprises employing visual indicator in the object data graphical user interface to indicate that an information element is compatible with a particular field in the form.

11. The method of claim 1 wherein (c) further comprises prompting the user to perform at least one of verifying or correcting the at least one information element in the at least one populated field based on the status indicator associated with that field.

12. The method of claim 11 wherein (d) comprises updating other fields of the populated form based on at least one of the user verification or correction of the at least one information element.

13. The method of claim 1 and further comprising:

(g) writing side information gleaned from edits made to any of the populated fields to a side information store; and

(h) re-parsing the information elements into tagged groups of different information types in (d) and re-populating at least one other field in (e) based on the side information.

14. The method of claim 1 wherein the electronic image comprises a plurality of objects and the method further comprises:

(g) performing (a) through (e) for each object, such that, for each object, fields of a corresponding electronic form are at least partially populated with information elements obtained from pixels within that object.

15. The method of claim 14 wherein the plurality of objects comprises objects of different types.

16. A computer-readable medium comprising computer storage media and computer-executable instructions, which are stored on the computer storage media and that, when executed by a computer, perform a method comprising:

(a) identifying a size, orientation and position of a first object having any arbitrary orientation within an electronic image;

(b) identifying information elements from pixels within the electronic image that correspond to the first object, including identifying text blocks within the first object using optical character recognition;

(c) displaying simultaneously to a user fields of the electronic form in a data area and the identified text blocks in an object data area that is outside of the form data area, which corresponds to the first object, through a graphical user interface, wherein the text blocks are selectable by the user within the object data area through the graphical user interface for insertion into respective fields of the electronic form in the form data area;

(d) parsing the information elements into tagged groups of different information types;

(e) automatically populating the fields of the electronic form with the tagged groups to produce a populated electronic form and allowing the user to edit the populated fields through the graphical user interface; and

(f) providing a visual status indicator adjacent each field of the form data area alerting the user that the field is unfilled and unverified, filled but unverified, and filled and verified, the status being based on the automatic populating and user editing.

17. The computer-readable medium of claim 16 wherein the electronic image comprises a plurality of objects, including the first object, and the method further comprises:

(g) performing (a) through (e) for each object such that, for each object, fields of a corresponding electronic form are at least partially populated with information elements obtained from pixels within that object.

18. The computer-readable medium of claim 17 wherein the plurality of objects comprises objects of different types.

19. The computer readable medium of claim 16 wherein (a) comprises:

classifying each pixel within the image to produce pixel classification data;

defining an image function to process the pixel classification data;

dividing the image into sub-images based on disparities in the image function; and

processing the sub-images to determine a size, an orientation and a position for each of the objects, including the first object.

20. The computer-readable medium of claim 19 wherein classifying comprises classifying each pixel as one of a data pixel or a background pixel.

21. The computer-readable medium of claim 20 and further comprising:

defining the image function as a sum of data pixels in a certain direction;

calculating the image function in a first direction to generate a first data set;

calculating the image function in a second direction to generate a second data set; and

searching for disparities in the image function in the first direction and the image function in the second direction.

22. The computer-readable medium of claim 16 wherein:

(b) comprises identifying two-dimensional locations of the text blocks within the first object; and

(d) comprises highlighting the tagged groups of different information types within the image of the first object in the object data area with visual indicators that indicate the different information types.

23. The computer-readable medium of claim 16 wherein:

(d) comprises receiving the information elements as untagged media from an untagged media store, and parsing the untagged media to identify information types of the information elements;

(e) comprises automatically populating at least one of the fields with at least one of the information elements based on the information type of that element; and

(c) comprises displaying the information elements through an object data graphical user interface and displaying the populated fields and any unpopulated fields through a form graphical user interface.

24. The computer-readable medium of claim 23 wherein (c) further comprises employing visual indicator in the object data graphical user interface to indicate that an information element is compatible with a particular field in the form.

25. The computer-readable medium of claim 16 wherein (c) further comprises prompting the user to perform at least one or verifying or correcting the at least one information element in the at least one populated field based on the status indicator associated with that field.

26. The computer-readable medium of claim 25 wherein (d) comprises updating other fields of the populated form based on at least one of the user verification or correction of the at least one information element.

27. The computer-readable medium of claim 16 and further comprising:

(g) writing side information gleaned from edits made by the user in step (c) to a side information store; and

(h) re-parsing the parsing the information elements into tagged groups of different information types in (d) and re-populating at least one other field in (e) based on the side information.

28. A system for a least partially populating electronic forms, the system comprising:

a display device;

an object detection and extraction module, which processes pixels in the electronic image to identify a size, orientation, and position of an object having any arbitrary orientation within the electronic image;

on optical character recognition module, which identifies information elements, including text blocks, from pixels within the electronic image that correspond to the first object;

a graphical user interface, which simultaneously displays to a user, on the display device, fields of the electronic form in a form data area and the identified text blocks in an object data area that is outside of the form data area and corresponds to the first object, and wherein the text blocks are selectable by the user within the object data area through the graphical user interface for insertion into respective fields of the electronic form in the form data area; and

a parsing module, which parses the information elements into tagged groups of different information types and at least partially populates the fields with the tagged groups automatically to produce a populated electronic form, wherein the graphical user interface provides a visual status indicator adjacent each field of the form data area alerting the user that the field is unfilled and unverified, filled but unverified, and filled and verified, the status being based on the automatic populating and user editing.

29. The system of claim 28 wherein the graphical user interface allows the user to edit the populated fields and any unpopulated fields.

30. The system of claim 29 wherein the parsing module comprises means for updating other fields of the populated form according to edits made by the user to the populated fields and the unpopulated fields.

31. The system of claim 28 wherein the object detection and extraction module comprises:

a data pixel detection module that classifies each pixel of the image and defines an image function;

a segmentation module that is capable of dividing the image into smaller sub-images based on disparities in the image function; and

a single object extraction module that processes the sub-images to detect a number, a size, an orientation and a position of objects within the image.

32. The system of claim 31 wherein the data the detection module classifies each pixel as cue of a data pixel or a background pixel.

33. The system of claim 32 wherein the data pixel detection module defines the image function as a sum of data pixels in a certain direction, calculates the image function in a first direction to generate a first data set, calculates the image function in a second direction to generate a second data set, and searches for disparities in the image function in the first direction and the image function in the second direction.

34. The system of claim 28 wherein:

the optical character recognition module comprises means for identifying two-dimensional locations of the text blocks within the first object; and

the graphical user interface comprises means for displaying the identified text blocks within an image of the first object to the user through the graphical user interface and highlighting the tagged groups of different information types within the image of the first object with visual indicators that indicate the different information types.

35. The system of claim 28 wherein:

the parsing module comprises means for receiving the information elements as untagged media from an untagged media store, parsing the untagged media in identify information types of the information elements, and automatically populating at least one of the fields with at least one of the information elements based on the information type of that element; and

the graphical user interface comprises means for displaying the information elements through an object data graphical user interface and displaying the populated fields and any unpopulated fields through a form graphical user interface.

36. The system of claim 35 wherein the object data graphical user interface comprises a visual indicator to indicated that an information element is compatible with a particular field in the form.

37. The system of claim 28 wherein the graphical user interface comprises means for prompting the user to perform at least one of verifying or correcting at least one information element in at least one populated field based on the status level indicator associated with that field.