UNSTRUCTURED AND SEMISTRUCTURED DOCUMENT PROCESSING AND SEARCHING

• US 20080263032A1
• Filed: 04/19/2007
• Published: 10/23/2008
• Est. Priority Date: 04/19/2007
• Status: Active Grant

• Current Assignee: Nook Digital, LLC (f/k/a Barnesandnoble.com LLC) (Bahwan CyberTek Pvt Ltd.)
• Original Assignee: Nook Digital, LLC (f/k/a Barnesandnoble.com LLC) (Bahwan CyberTek Pvt Ltd.)
• Inventors: Wu, Jiang, Vailaya, Aditya, Rathi, Manish
• Granted Patent: US 8,504,553 B2
• Time in Patent Office: Days
• US Class Current: 707/6
• CPC Class Codes: G06F 16/334 Query execution G06F16/335 takes precedence

Abstract

A method for analyzing and indexing an unstructured or semistructured document according to one embodiment includes receiving an unstructured or semistructured document; converting the document to one or more text streams; analyzing the one or more text streams for identifying textual contents of the document; analyzing the one or more text streams for identifying logical sections of the document; associating the textual contents with the logical sections; indexing the textual contents and their association with the logical sections; and saving a result of the indexing in a data storage device.

Claims

1. A method for analyzing and indexing an unstructured or semistructured document, comprising:

receiving an unstructured or semistructured document;

converting the document to one or more text streams;

analyzing the one or more text streams for identifying textual contents of the document;

analyzing the one or more text streams for identifying logical sections of the document;

associating the textual contents with the logical sections;

indexing the textual contents and their association with the logical sections; and

saving a result of the indexing in a data storage device.

2. The method of claim 1, wherein the unstructured or semistructured document is in a printer format.

3. The method of claim 1, wherein the unstructured or semistructured document is a binary representation of dark and light areas of a scanned document.

4. The method of claim 1, wherein the unstructured or semistructured document does not contain format markers.

5. The method of claim 1, further comprising identifying a context of the unstructured document and generating meta data representing a context of at least some of the sections.

6. The method of claim 5, further comprising indexing the meta data.

7. The method of claim 5, wherein the context information is based on a word extracted from the document and matched to a term in a context-related dictionary.

8. The method of claim 1, wherein the textual contents include words in the document.

9. The method of claim 1, wherein the sections include groups of paragraphs of the document, each paragraph being individually detected by analyzing the one or more text streams.

10. The method of claim 1, further comprising associating taxonomy-related information with the textual contents, and indexing the taxonomy-related information.

11. The method of claim 1, wherein the indexing includes assigning a weight to the textual contents.

12. The method of claim 11, further comprising analyzing an index of the document, and assigning a higher weight to a textual content matching a term in the index of the document and present on a page of the document pointed to in the index in association with the term.

13. The method of claim 1, further comprising extracting a table of contents from the unstructured or semistructured document.

14. The method of claim 1, further comprising identifying physical page numbers in the unstructured or semistructured document, and mapping logical page numbers to the physical page numbers.

15. A method for analyzing an unstructured or semistructured document, comprising:

receiving an unstructured or semistructured document;

converting the document to one or more text streams;

analyzing the one or more text streams for identifying paragraphs of the document;

grouping the paragraphs into sections; and

outputting the sections, or derivative thereof, to at least one of a user, another system, and another process.

16. The method of claim 15, wherein the document is in a printer format.

17. The method of claim 15, wherein the document is a binary representation of dark and light areas of a scanned document.

18. The method of claim 15, wherein the document does not contain format markers.

19. The method of claim 15, further comprising extracting page numbers from the document, and associating the sections with the page numbers.

20. The method of claim 15, wherein the paragraphs are identified by:

determining geometry information about lines of the text streams;

placing the lines into blocks based on a proximity of the lines relative to each other; and

analyzing the blocks for joining lines into paragraphs.

21. The method of claim 15, wherein boundaries of the sections are determined at least in part based on an analysis of a table of contents of the document.

22. A method for analyzing and indexing an unstructured or semistructured document, comprising:

receiving an unstructured or semistructured document;

converting the document to one or more text streams;

analyzing the one or more text streams for identifying textual contents of the document;

identifying logical sections of the document;

associating the textual contents with the sections;

analyzing the one or more text streams for identifying context information about each section;

indexing the textual contents, the context information, and the association of the textual contents and context information with the logical sections; and

saving a result of the indexing in a data storage device.