METHOD AND APPARATUS FOR CALCULATING TOPICAL CATEGORIZATION OF ELECTRONIC DOCUMENTS IN A COLLECTION

• US 20120296637A1
• Filed: 05/15/2012
• Published: 11/22/2012
• Est. Priority Date: 05/20/2011
• Status: Abandoned Application

• Current Assignee: Ebrary
• Original Assignee: Ebrary
• Inventors: Santos, Tom J., SMILEY, Edwin Lee
• Application Number:
  US13/472,362
• Publication Number:
  US 20120296637A1
• Time in Patent Office: Days
• US Class Current: 704/9
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 40/30 Semantic analysis

Abstract

A computer implemented method calculates topical categorization of electronic documents in a collection. A processor applies a metric to categorize semantic distance between two sections of a document or between two documents. The processor executes a topic algorithm using the categorization provided by the metric to determine topic boundaries. Topics are extracted based upon the topic boundaries; and the extracted topics are compared for similarity with topics in other documents for organizational and research purposes.

Claims

1. A computer implemented method for calculating topical categorization of electronic documents in a collection, comprising:

processor application of a metric to categorize semantic distance between two sections of a document or between two documents;

said processor executing a topic algorithm using the categorization provided by said metric to automatically determine topic boundaries;

said processor extracting topics based upon said topic boundaries; and

said processor comparing said extracted topics for similarity with topics in other documents for organizational and research purposes.

2. The method of claim 1, said topic algorithm using variational behavior of semantic distances to calculate topical categorization.

3. The method of claim 1, said topic algorithm using recursive division and differential adhesion based on semantic distances to calculate topical categorization.

4. The method of claim 1, said topic algorithm using both variational behavior of semantic distances, and recursive division and differential adhesion based on semantic distances to calculate topical categorization.

5. The method of claim 1, said topic algorithm detecting topic changes within clear breaks indicated by any of sentence breaks, chapter headings and metadata.

6. The method of claim 1, said document comprising any data in the form of a sequence of meaningful tokens that can be represented digitally or that can be expressed in the form of such a sequence.

7. The method of claim 6, said document comprising any of text, musical passages, choreography, and mathematics.

8. The method of claim 1, said topic algorithm determining topic boundaries for any of:

detecting similarity and/or transitions of meaning in passages where an intended meaning is opaque to an analyzer;

detecting similarity and/or transitions and related passages in an unknown script;

analyzing similarity and/or transitions of purported extraterrestrial signals;

detecting similarity and/or transitions in technical and mathematical papers;

providing an element of search or document discovery;

detecting unexpected, more valuable results based on topics;

identifying unexpected correspondences in research;

finding related passages in an unknown script;

supporting social recommendation engines; and

detecting plagiarism.

9. The method of claim 1, said topic algorithm supplying chapter and/or heading generation for documents that do not possess chapters and/or headings.

10. The method of claim 1, said topic algorithm categorizing said topics in a multidimensional space by a distance determined relative to a canonical set, or are used as generators of a canonical set, of document topics which serve as axes in said multidimensional space.

11. The method of claim 1, said topic algorithm stochastically selecting a canonical set of document topics.

12. The method of claim 1, further comprising:

applying one or more compression algorithms to compute compression size alone.

13. The method of claim 12, said one or more compression algorithms taking into account self-compression overhead of compressing absolutely identical data.

14. The method of claim 1, said topic algorithm taking into account a scaling measure that is independent of the size of objects of comparison.

15. The method of claim 1, further comprising:

using said topic algorithm to carry out a calculation of topic boundaries pursuant to a document sketch technique.

16. The method of claim 1, further comprising:

testing the effectiveness of parameters used by said topic algorithm by variation against a non-sequitur document term of art.

17. The method of claim 1, wherein said topic algorithm is independent of a normalized compression metric.

18. The method of claim 1, further comprising:

adjusting a computed topic boundary using a topic algorithm in text documents to a nearest sentence or section bound.

19. The method of claim 1, further comprising:

using an embodied metric to isolate most typical gist sentences within topics.

20. The method of claim 1, further comprising:

creating an orthonormal basis for a multi-dimensional topic space of a linear combination of topics using a random or prescribed topic sample and normalization using orthogonalization schemes comprising any of a stabilized Gram-Schmidt process, Householder transformations, and Givens rotations.

21. The method of claim 1, further comprising:

using a Euclidean metric for search and discovery of related topics in a collection of documents.

22. The method of claim 1, further comprising:

using cosine distance and, thereafter, a Euclidean metric for search and discovery of related topics in a collection.

23. The method of claim 1, further comprising:

using range threshold, within which each component of the coordinate needs to fall, and, thereafter, a Euclidean metric for search and discovery of related topics in a collection.

24. The method of claim 1, further comprising:

using topics to determine one of a number of types of significant document transitions;

using significant document transitions to predict user behavior by assigning weights to transition types and/or by calculating a transition matrix of probabilities using said weights; and

constructing cost/benefit models for storing and retrieving sections of documents from large collections of documents.

25. An apparatus for calculating topical categorization of electronic documents in a collection, comprising:

a processor configured for applying a metric to categorize semantic distance between two sections of a document or between two documents;

said processor configured for executing a topic algorithm using the categorization provided by said metric to automatically determine topic boundaries;

said processor configured for extracting topics based upon said topic boundaries; and

said processor configured for comparing said extracted topics for similarity with topics in other documents for organizational and research purposes.

26. The apparatus of claim 25, said topic algorithm using variational behavior of semantic distances to calculate topical categorization.

27. The apparatus of claim 25, said topic algorithm using recursive division and differential adhesion based on semantic distances to calculate topical categorization.

28. The apparatus of claim 25, said topic algorithm using both variational behavior of semantic distances, and recursive division and differential adhesion based on semantic distances to calculate topical categorization.