Enhanced interpretation of character arrangements

• US 9,824,269 B2
• Filed: 11/15/2016
• Issued: 11/21/2017
• Est. Priority Date: 08/21/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Zhang, Yi, Parish, Daniel, Wilson, Aaron Lamar, Fang, Shangwei
• Primary Examiner(s): Desire, Gregory M
• Application Number:
  US15/352,485
• Publication Number:
  US 20170116474A1
• Time in Patent Office: 371 Days
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/224 of printed characters having additional code marks or containing code marks
• CPC Class Codes: G06V 30/245 Font recognition
• CPC Class Codes: G06V 30/412 Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables
• CPC Class Codes: G06V 30/413 Classification of content, e.g. text, photographs or tables

Abstract

Technologies are described herein for interpreting character arrangements. An image including an arrangement of characters may be received or captured by a computing device. Techniques described herein generate data representative of the characters. Characteristics and other information interpreted from the image may be processed to determine a data type. The data representative of the characters may be arranged into a data structure based on the data type, an arrangement type and/or other information interpreted from the image. The data type may indicate one or more attributes of the arranged data such as a format, font, date, language, or currency. The data type may also indicate how data is used in a process, equation or calculation. In addition, the data type may identify an anchor that may be used to merge data generated from the image with other data generated from another image.

Claims

1. A computer-implemented method comprising computer-implemented operations for:

obtaining an image comprising an arrangement of a plurality of characters;

generating data representing individual characters of the plurality of characters;

determining an arrangement type based on the arrangement of the plurality of characters;

determining a data type of at least one character of the plurality of characters;

arranging the data representing the individual characters of the plurality of characters in a data structure based on the arrangement type and the data type of the at least one character of the plurality of characters;

displaying the data structure on an interface configured to receive an input gesture to modify the data structure, wherein the displaying of the data structure is performed in response to a camera of a computing device obtaining the image;

receiving the input gesture to modify the data structure; and

modifying the data representing the individual characters of the plurality of characters based on the input gesture.

2. The computer-implemented method of claim 1, further comprising computer-implemented operations for:

obtaining a second image comprising a second arrangement of a second plurality of characters;

generating other data representing individual characters of the second plurality of characters;

determining a second data type of at least one character of the second plurality of characters; and

arranging the other data representing the individual characters of the second plurality of characters in the data structure based on the second arrangement of the second plurality of characters and the second data type.

3. The computer-implemented method of claim 2, wherein the data type and the second data type identify an anchor, and wherein the data representing the individual characters of the plurality of characters and the other data representing the individual characters of the second plurality of characters are merged in the data structure based on the anchor.

4. The computer-implemented method of claim 1, wherein the determining the data type of the at least one character of the plurality of characters is based on a format of the at least one character of the plurality of characters, a font of the at least one character of the plurality of characters, or at least one text pattern formed from the plurality of characters.

5. The computer-implemented method of claim 1, wherein the plurality of characters comprises the at least one character and other characters, wherein the data type indicates that data representing the at least one character is a header, and wherein the arranging the data in the data structure comprises distinguishing a display of the data representing the at least one character from a display of data representing the other characters.

6. The computer-implemented method of claim 1, wherein the plurality of characters comprises the at least one character and other characters, and wherein the computer-implemented method further comprises computer-implemented operations for:

identifying a second data type of the other characters, wherein the second data type of the other characters is incompatible with the data type of the at least one character of the plurality of characters; and

providing a graphical distinction between a display of data representing the at least one character from a display of other data representing the other characters.

7. The computer-implemented method of claim 6, further comprising computer-implemented operations for generating at least one equation utilizing a data set, the data set consisting of the other data representing the other characters of the plurality of characters.

8. A computer, comprising:

a processor; and

a computer-readable storage medium in communication with the processor, the computer-readable storage medium having computer-executable instructions stored thereupon which, when executed by the processor, cause the computer to;

obtain an image comprising an arrangement of a plurality of characters,

generate data representing individual characters of the plurality of characters,

determine a data type of at least one character of the plurality of characters, wherein the data type is based on the at least one character, the arrangement of the plurality of characters, or contextual data interpreted from the image,

arrange the data representing the individual characters of the plurality of characters in a data structure based on the data type of the at least one character of the plurality of characters,

display the data structure on an interface configured to receive an input gesture to modify the data structure, wherein the display of the data structure is performed in response to a camera of the computer obtaining the image,

receive the input gesture to modify the data structure, and

modify the data representing individual characters of the plurality of characters based on the input gesture.

9. The computer of claim 8, wherein the computer-readable storage medium has further computer-executable instructions stored thereupon which, when executed by the processor, cause the computer to:

obtain a second image comprising a second arrangement of a second plurality of characters;

generate other data representing individual characters of the second plurality of characters;

determine a second data type of at least one character of the second plurality of characters; and

arrange the other data representing the individual characters of the second plurality of characters in the data structure based on the second arrangement of the second plurality of characters and the second data type.

10. The computer of claim 9, wherein the data type and the second data type identify an anchor, and wherein the data representing the individual characters of the plurality of characters and the other data representing the individual characters of the second plurality of characters are merged in the data structure based on the anchor.

11. The computer of claim 8, wherein the determining the data type of the at least one character of the plurality of characters is based on a format of the at least one character of the plurality of characters or at least one text pattern formed from the plurality of characters.

12. The computer of claim 8, wherein the data type indicates that data representing the at least one character is a header, and wherein the arranging the data representing the individual characters of the plurality of characters in the data structure comprises distinguishing a display of the data representing the at least one character from a display of data representing other characters of the plurality of characters.

13. The computer of claim 8, wherein the plurality of characters comprises the at least one character and other characters, and wherein the computer-readable storage medium has further computer-executable instructions stored thereupon which, when executed by the processor, cause the computer to:

identify a second data type of the other characters, wherein the second data type is incompatible with the data type; and

provide a graphical distinction between a display of data representing the at least one character and a display of other data representing the other characters.

14. The computer of claim 13, wherein the computer-readable storage medium has further computer-executable instructions stored thereupon which, when executed by the processor, cause the processor to generate at least one equation utilizing a data set, the data set consisting of the other data representing the other characters of the plurality of characters.

15. A computer-readable storage medium having computer-executable instructions stored thereupon which, when executed by a computer, cause the computer to:

obtain an image comprising an arrangement of a plurality of characters;

generate data representing individual characters of the plurality of characters;

determine an arrangement type based on the arrangement of the plurality of characters;

determine a data type of at least one character of the plurality of characters, wherein the data type is based on the at least one character, the arrangement of the plurality of characters, or contextual data interpreted from the image;

arrange the data representing the individual characters of the plurality of characters in a data structure based on the arrangement type and the data type of the at least one character of the plurality of characters;

display the data structure on an interface configured to receive an input gesture to modify the data structure, wherein the display of the data structure is performed as a camera of the computer is capturing the image;

receive the input gesture to modify the data structure; and

modify the data representing the individual characters of the plurality of characters based on the input gesture.

16. The computer-readable storage medium of claim 15, having further computer-executable instructions stored thereupon, which when executed by the computer, cause the computer to:

obtain a second image comprising a second arrangement of a second plurality of characters;

generate other data representing individual characters of the second plurality of characters;

determine a second arrangement type of the second plurality of characters;

determine a second data type of at least one character of the second plurality of characters; and

arrange the other data representing the individual characters of the second plurality of characters in the data structure based on the second arrangement of the second plurality of characters, the second arrangement type, and the second data type.

17. The computer-readable storage medium of claim 16, wherein the arrangement type and the second arrangement type identify an anchor, and wherein the data representing the individual characters of the plurality of characters and the other data representing the individual characters of the second plurality of characters are merged in the data structure based on the anchor.

18. The computer-readable storage medium of claim 17, wherein the plurality of characters comprises the at least one character and other characters, and wherein the computer-readable storage medium has further computer-executable instructions stored thereupon, which when executed by the computer, cause the computer to:

identify a second data type of the other characters, wherein the second data type is incompatible with the data type; and

provide a graphical distinction between a display of data representing the at least one character and a display of other data representing the other characters.