Method of analyzing documents

• US 20060259481A1
• Filed: 05/12/2005
• Published: 11/16/2006
• Est. Priority Date: 05/12/2005
• Status: Active Grant

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Handley, John C.
• Granted Patent: US 7,451,124 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/355 Class or cluster creation or modification
• CPC Class Codes: G06F 16/358 Browsing; Visualisation therefor
• CPC Class Codes: G06F 40/30 Semantic analysis

Abstract

A method for analyzing documents is disclosed. The method compares concepts consisting of groups of terms for similarity within a corpus of document, clusters documents that contain certain concept term sets together. It may also rank the documents within each cluster according to the frequency of term co-occurrence within the concepts.

Claims

1. A method of analyzing a plurality of documents, comprising:

collecting and filtering terms from a plurality of documents;

identifying a term-frequency vector for each of the documents;

identifying a term-frequency matrix, wherein rows of the matrix comprise values for the term-frequency vectors;

projecting the term-frequency matrix onto a lower dimensional space using latent semantic analysis, to create a transformed term matrix;

developing a correlation matrix using the rows of the transformed term matrix;

creating a concept graph of connected components using a concept threshold, where each connected component is a set of terms that corresponds to a concept; and

clustering documents that contain concept term sets together.

2. The method of claim 1, wherein the filtering is performed with reference to a set of stop words.

3. The method of claim 1, wherein the method uses a processor and memory, and the plurality of documents is stored in the memory.

4. The method of claim 1, wherein the set of term-frequency vectors for the documents is maintained as rows and columns in a term-frequency matrix.

5. The method of claim 1, wherein the columns and rows of the correlation matrix correspond to terms that are the terms of the transformed term matrix, and a concept comprises highly correlated terms.

6. The method of claim 1, wherein terms of the correlation matrix comprising a correlation higher than the concept threshold become vertices of a concept graph.

7. The method of claim 1, further comprising ranking the documents within each cluster according to the number of occurrences of concept terms within a term set.

8. The method of claim 1, wherein singular value decomposition is used to express the term-frequency matrix as a product of at least three matrices.

9. The method of claim 7, wherein the at least three matrices include a document matrix, a diagonal matrix of eigenvalues, and a term matrix.

10. The method of claim 8, wherein the entries of the diagonal matrix are the eigenvalues of the term-frequency matrix in decreasing order, and wherein the rows of the term matrix comprise vectors corresponding to term frequencies in a transformed space.

11. The method of claim 9, wherein dimensionality reduction is performed by trimming the matrices in the decomposition by selecting the m greatest diagonal elements of the diagonal matrix in the composition and selecting the m corresponding rows of the term matrix and multiplying a trimmed diagonal matrix by a trimmed term matrix.

12. The method of claim 1, wherein the entries of the correlation matrix are values from about −

1 to about 1.

13. The method of claim 1, wherein the concept threshold is less than or equal to 1, and terms that have correlation higher than the concept threshold form the vertices of the concept graph.

15. The method of claim 1, further comprising identifying one or more logs that relate to a product problem based on the ranking of the log or logs.

16. The method of claim 1, further comprising identifying one or more problems based on the frequency of occurrence of terms in the concept term set.

14. A method for determining product problems from field service logs, comprising:

collecting and filtering terms from a plurality of service logs;

identifying a term-frequency vector for each of the logs;

identifying a term-frequency matrix, wherein rows of the matrix comprise values for the term-frequency vectors;

projecting the term-frequency matrix onto a lower dimensional space using latent semantic analysis, to create a transformed term matrix;

determining a correlation matrix using the rows of the transformed term matrix;

creating a concept graph of connected components using a concept threshold, where each connected component comprises a set of terms that corresponds to a concept;

clustering documents that contain concept term sets together;

for each cluster and corresponding term sets, ranking logs in each cluster by frequency of occurrence of terms in the concept term set.

17. A method analyzing a plurality of documents, comprising:

collecting and filtering terms from a plurality of documents;

identifying a term-frequency vector for each of the documents;

identifying a term-frequency matrix, wherein rows of the matrix comprise values for the term-frequency vectors;

projecting the term-frequency matrix onto a lower dimensional space using latent semantic analysis, to create a transformed term matrix;

developing a correlation matrix using the rows of the transformed term matrix;

creating a dendrogram of related concepts using a function of the correlation matrix;

identifying branches of the dendrogram corresponding to related concepts; and

clustering documents that contain concept term sets together.

18. The method of claim 17, wherein the filtering is performed with reference to a set of stop words.

19. The method of claim 17, wherein the method uses a processor and memory, and the plurality of documents is stored in the memory.

20. The method of claim 17, further comprising ranking the documents within each cluster according to the number of occurrences of concept terms within a term set.