INFERRING EMERGING AND EVOLVING TOPICS IN STREAMING TEXT

• US 20130151525A1
• Filed: 09/14/2012
• Published: 06/13/2013
• Est. Priority Date: 12/09/2011
• Status: Abandoned Application

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Ankan, Saha, Banerjee, Arindam, Kasiviswanathan, Shiva P., Lawrence, Richard D., Melville, Prem, Sindhwani, Vikas, Ting, Edison L.
• Application Number:
  US13/616,403
• Publication Number:
  US 20130151525A1
• Time in Patent Office: Days
• US Class Current: 707/737
• CPC Class Codes: G06F 16/316 Indexing structures
• CPC Class Codes: G06F 40/30 Semantic analysis

Abstract

A method, system and computer program product for inferring topic evolution and emergence in a set of documents. In one embodiment, the method comprises forming a group of matrices using text in the documents, and analyzing these matrices to identify evolving topics and emerging topics. The matrices includes a matrix X identifying a multitude of words in each of the documents, a matrix W identifying a multitude of topics in each of the documents, and a matrix H identifying a multitude of words for each of the multitude of topics. These matrices are analyzed to identify the evolving and emerging topics. In an embodiment, two forms of temporal regularizers are used to help identify the evolving and emerging topics. In another embodiment, a two stage approach involving detection and clustering is used to help identify the evolving and emerging topics.

Claims

1. A method of inferring topic evolution and emergence in a multitude of documents, comprising:

forming a group of matrices using text in the documents, said group of matrices including a first matrix X identifying a multitude of words in each of the documents, a second matrix W identifying a multitude of topics in each of the documents, and a third matrix H identifying a multitude of words for each of said multitude of topics; and

analyzing said group of matrices to identify a first group of said multitude of topics as evolving topics and a second group of said multitude of topics as emerging topics.

2. The method according to claim 1, wherein said multitude of documents comprise a sequence of streaming documents, each of the documents being associated with a timepoint t, in a defined period of time T, and wherein:

the forming the group of matrices using data in the documents includes;

forming a first sequence of matrices X(t), each of the matrices X(t) identifying a multitude of words in each of a set of the documents associated with the timepoints within a defined sliding window in the time period T;

forming a second sequence of matrices W(t), each of the matrices W(t) identifying a multitude of topics in said set of documents associated with the timepoints within said defined window; and

forming a third sequence of matrices H(t), each of the matrices H(t) identifying a multitude of words for each of the topics identified in an associated one of the matrices W(t); and

the analyzing the groups of matrices includes using a defined equation including the matrices X(t), W(t) and H(t), to identify the evolving and the emerging topics.

3. The method according to claim 2, wherein:

said defined equation includes a first regularizer μ

to enforce smooth evolution of the evolving topics via constraints on an amount of drift allowed by the evolving topics, and a second regularizer Ω

to apply a topic bandwidth for early detection of the emerging topics to extract smooth trends of candidate emerging topics; and

said defined equation is an objective function;

4. The method according to claim 2, wherein:

said defined equation includes solving an l₁ dictionary learning problem to identify evolving topics, and using a reconstruction error to identify novel documents;

the analyzing the group of matrices further includes clustering said novel documents to identify emerging topics; and

said defined equation is an objective function;

W*, H*=argmin_W,H∥

X−

WH∥

₁+λ

∥

W∥

₁ such that W, H≧

0.

5. The method according to claim 1, wherein the forming the group of matrices using text in the documents includes using the first matrix to form the second and third matrices.

6. The method according to claim 1, wherein the analyzing the group of matrices includes using a defined equation, including the first, second and third matrices, to identify the evolving topics and the emerging topics.

7. The method according to claim 6, wherein said defined equation includes a regularizer to enforce smooth evolution of the evolving topics via constraints on an amount of drift allowed by the evolving topics.

8. The method according to claim 6, wherein:

said defined equation includes a regularizer to apply a topic bandwidth for early detection of the emerging topics; and

the regularizer extracts smooth trends of candidate emerging topics.

9. The method of claim 6 wherein the analyzing the group of matrices further includes:

using said defined equation to identify novel documents based on reconstruction error; and

clustering the novel documents to identify emerging topics and used for updating the evolving topics.

10. The method according to claim 9, wherein:

the using the defined equation includes applying a threshold on the reconstruction errors obtained from a solution of said defined equation to identify the novel documents; and

the clustering the novel documents includes using a given clustering algorithm to cluster the novel documents.