INTEGRATING AND EXTRACTING TOPICS FROM CONTENT OF HETEROGENEOUS SOURCES

• US 20150066904A1
• Filed: 08/29/2013
• Published: 03/05/2015
• Est. Priority Date: 08/29/2013
• Status: Active Grant

• Current Assignee: Hewlett Packard Enterprise Development LP (Hewlett-Packard Enterprise Company)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Asur, Sitaram, Ghosh, Rumi
• Granted Patent: US 9,176,969 B2
• Time in Patent Office: Days
• US Class Current: 707/722
• CPC Class Codes: G06F 16/14 Details of searching files based on file metadata
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Examples relate to integrating and extracting topics from content of heterogeneous sources. Observed words are identified in documents, which are received from the heterogeneous sources. Next, document metadata and source metadata are obtained from the heterogeneous sources. The document metadata is used to calculate word topic probabilities for the observed words, and the source metadata is used to calculate source topic probabilities for the observed words. A latent topic is then determined for one of the documents based on the observed words, the word topic probabilities, and the source topic probabilities.

Claims

1. A system for integrating and extracting topics from content of heterogeneous sources, the system comprising:

a processor to;

identify a plurality of observed words in documents that are received from the heterogeneous sources;

obtain document metadata and source metadata from the heterogeneous sources;

use the document metadata to calculate a plurality of word topic probabilities for the plurality of observed words;

use the source metadata to calculate a plurality of source topic probabilities for the plurality of observed words; and

determine a latent topic for one of the documents based on the plurality of observed words, the plurality of word topic probabilities, and the plurality of source topic probabilities.

2. The system of claim 1, wherein the processor is further to use a global vocabulary and a global Dirichlet prior parameter to determine a plurality of global word topic probabilities, wherein the latent topic is further based on the plurality of global word topic probabilities.

3. The system of claim 1, wherein the latent topic is determined using a Discriminative Dirichlet Allocation (DDA) modeling technique.

4. The system of claim 3, wherein the DDA modeling technique comprises:

in response to determining that a number of occurrences of the related observed words assigned to the latent topic has reached a dynamic threshold, adjusting the word topic probability based on pre-determined user-defined features; and

adjusting the word topic probability of the observed word based on the source topic probability associated with the observed word.

5. The system of claim 4, wherein adjusting the word topic probability of the observed word comprises using Gibbs sampling to apply a bicriterion that maximizes the plurality of word topic probabilities and uses the dynamic threshold to monitor the number of occurrences of the related observed words.

6. The system of claim 1, wherein the heterogeneous sources comprise a news source, a blog source, a social media source, a document repository source, an online retailer source, an email source, and a discussion forum source.

7. A method, implemented at least in part by a computing device, for integrating and extracting topics from content of heterogeneous sources, the method comprising:

identifying a plurality of observed words in documents that are received from the heterogeneous sources;

presenting document metadata and source metadata from the heterogeneous sources;

using the document metadata to calculate a plurality of word topic probabilities for the plurality of observed words;

using the source metadata to calculate a plurality of source topic probabilities for the plurality of observed words; and

using a Discriminative Dirichlet Allocation (DDA) modeling technique to determine a latent topic for one of documents based on the plurality of observed words, the plurality of word topic probabilities, and the plurality of source topic probabilities.

8. The method of claim 7, further comprising using a global vocabulary and a global Dirichlet prior parameter to determine a plurality of global word topic probabilities, wherein the latent topic is further based on the plurality of global word topic probabilities.

9. The method of claim 7, wherein the DDA modeling technique comprises:

in response to determining that a number of occurrences of the related observed words assigned to the latent topic has reached a dynamic threshold, adjusting the word topic probability based on pre-determined user-defined features; and

adjusting the word topic probability of the observed word′

based on the source topic probability associated with the observed word.

10. The method of claim 9, wherein adjusting the word topic probability of the observed word comprises using Gibbs sampling to apply a bicriterion that maximizes the plurality of word topic probabilities and uses the dynamic threshold to monitor the number of occurrences of the related observed words.

11. The method of claim 7, wherein the heterogeneous sources comprise a news source, a blog source, a social media source, a document repository source, an online retailer source, an email source, and a discussion forum source.

12. A non-transitory machine-readable storage medium encoded with instructions executable by a processor for integrating and extracting topics from content of heterogeneous sources, the machine-readable storage medium comprising instructions to:

identify a plurality of observed words in documents that are received from the heterogeneous sources;

obtain document metadata and source metadata from the heterogeneous sources;

use the document metadata to calculate a plurality of word topic probabilities for the plurality of observed words;

use a global vocabulary and a global Dirichlet prior parameter to determine a plurality of global word topic probabilities;

use the source metadata to calculate a plurality of source topic probabilities for the plurality of observed words; and

determine a latent topic for one of the documents based on the plurality of observed words, the plurality of word topic probabilities, the plurality of global word topic probabilities, and the plurality of source topic probabilities.

13. The non-transitory machine-readable storage medium of claim 12, wherein the latent topic is determined using a Discriminative Dirichlet Allocation (DDA) modeling technique, and wherein the DDA modeling technique comprises:

in response to determining that a number of occurrences of the related observed words assigned to the latent topic has reached a dynamic threshold, adjusting the word topic probability based on pre-determined user-defined features; and

adjusting the word topic probability of the observed word based on the source topic probability associated with the observed word.

14. The non-transitory machine-readable storage medium of claim 13, wherein adjusting the word topic probability of the observed word comprises using Gibbs sampling to apply a bicriterion that maximizes the plurality of word topic probabilities and uses the dynamic threshold to monitor the number of occurrences of the related observed words.

15. The non-transitory machine-readable storage medium of claim 12, wherein the heterogeneous sources comprise a news source, a blog source, a social media source, a document repository source, an online retailer source, an email source, and a discussion forum source.