Parameter Inference Method, Calculation Apparatus, and System Based on Latent Dirichlet Allocation Model

• US 20140129510A1
• Filed: 01/13/2014
• Published: 05/08/2014
• Est. Priority Date: 07/13/2011
• Status: Active Grant

• Current Assignee: Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.)
• Original Assignee: Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.)
• Inventors: Vladislav, Kopylov, Wen, Liufei, Shi, Guangyu
• Granted Patent: US 9,213,943 B2
• Time in Patent Office: Days
• US Class Current: 706/52
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06N 5/048 Fuzzy inferencing

Abstract

A parameter inference method to solve a problem that precision of a Latent Dirichlet Allocation model is poor is provided. The method includes: calculating a Latent Dirichlet Allocation model according to a preset initial first hyperparameter, a preset initial second hyperparameter, a preset initial number of topics, a preset initial count matrix of documents and topics, and a preset initial count matrix of topics and words to obtain probability distributions; obtaining the number of topics, a first hyperparameter, and a second hyperparameter that maximize log likelihood functions of the probability distributions; and determining whether the number of topics, the first hyperparameter, and the second hyperparameter converge, and if not, putting the number of topics, the first hyperparameter, and the second hyperparameter into the Latent Dirichlet Allocation model until the optimal number of topics, an optimal first hyperparameter, and an optimal second hyperparameter that maximize the log likelihood functions of the probability distributions.

Claims

1. A parameter inference method based on a Latent Dirichlet Allocation model, comprising:

calculating a Latent Dirichlet Allocation model according to a preset initial first hyperparameter, a preset initial second hyperparameter, a preset initial number of topics, a preset initial global count matrix of documents and topics, and a preset initial global count matrix of topics and words, to obtain a probability distribution between documents and topics and a probability distribution between topics and words;

obtaining, by using an expectation maximization algorithm, a first hyperparameter, a second hyperparameter, and the number of topics that maximize log likelihood functions of the probability distributions; and

determining whether the first hyperparameter, the second hyperparameter, and the number of topics converge, and when the first hyperparameter and the second hyperparameter do not converge, putting the first hyperparameter, the second hyperparameter, and the number of topics into the Latent Dirichlet Allocation model for calculation until an optimal first hyperparameter, an optimal second hyperparameter, and an optimal number of topics that maximize the log likelihood functions of the probability distributions converge, and outputting the probability distributions, the optimal first hyperparameter, the optimal second hyperparameter, and the optimal number of topics that are eventually obtained.

2. The method according to claim 1, wherein calculating the Latent Dirichlet Allocation model according to the preset initial first hyperparameter, the preset initial second hyperparameter, the preset initial number of topics, the preset initial global count matrix of documents and topics, and the preset initial global count matrix of topics and words, to obtain the probability distribution between documents and topics and the probability distribution between topics and words comprises:

segmenting, by a master calculation apparatus, a document set into a plurality of document subsets, and setting the initial first hyperparameter, the initial second hyperparameter, the initial number of topics, the initial global count matrix of documents and topics, and the initial global count matrix of topics and words;

sending, by the master calculation apparatus, each document subset of a plurality of document subsets, the initial first hyperparameter, the initial second hyperparameter, the initial number of topics, the initial global count matrix of documents and topics, and the initial global count matrix of topics and words to each slave calculation apparatus of a plurality of slave calculation apparatuses correspondingly such that each slave calculation apparatus of the plurality of slave calculation apparatuses solves the Latent Dirichlet Allocation model and updates a local count matrix of documents and topics and a local count matrix of topics and words according to a received initial global count matrix of documents and topics, a received initial global count matrix of topics and words, the received initial first hyperparameter, the received initial second hyperparameter, and the received initial number of topics;

receiving, by the master calculation apparatus, the local count matrix of documents and topics and the local count matrix of topics and words that are returned by each slave calculation apparatus, and performing merging to obtain a global count matrix of documents and topics and a global count matrix of topics and words;

determining, by the master calculation apparatus, whether a process of solving, by each slave calculation apparatus, the Latent Dirichlet Allocation model and updating the local count matrix of documents and topics and the local count matrix of topics and words converges, and when the process of solving the Latent Dirichlet Allocation model and updating the local count matrix of documents and topics and the local count matrix of topics and words does not converge, sending the global count matrix of documents and topics and the global count matrix of topics and words to each slave calculation apparatus correspondingly for calculation, and continuing to update the local count matrix of documents and topics and the local count matrix of topics and words until the process of solving the Latent Dirichlet Allocation model and updating the local count matrix of documents and topics and the local count matrix of topics and words converges, and outputting the global count matrix of documents and topics and global count matrix of topics and words that are eventually obtained; and

calculating, by the master calculation apparatus, the probability distribution between documents and topics and the probability distribution between topics and words according to the global count matrix of documents and topics and the global count matrix of topics and words.

3. The method according to claim 2, wherein determining whether the first hyperparameter, the second hyperparameter, and the number of topics converge, and when the first hyperparameter, the second hyperparameter, and the number of topics do not converge, putting the first hyperparameter, the second hyperparameter, and the number of topics into the Latent Dirichlet Allocation model for calculation comprises determining, by the master calculation apparatus, whether the first hyperparameter, the second hyperparameter, and the number of topics converge, and when the first hyperparameter, the second hyperparameter, and the number of topics do not converge, sending the first hyperparameter, the second hyperparameter, and the number of topics to each slave calculation apparatus such that each slave calculation apparatus puts the first hyperparameter, the second hyperparameter, and the number of topics into the Latent Dirichlet Allocation model for local calculation.

4. A parameter inference calculation apparatus based on a Latent Dirichlet Allocation model, comprising:

a Latent Dirichlet Allocation model calculation unit configured to calculate a Latent Dirichlet Allocation model according to a preset initial first hyperparameter, a preset initial second hyperparameter, the preset initial number of topics, a preset initial global count matrix of documents and topics, and a preset initial global count matrix of topics and words, to obtain a probability distribution between documents and topics and a probability distribution between topics and words;

a parameter estimation unit configured to obtain, by using an expectation maximization algorithm, a first hyperparameter, a second hyperparameter, and the number of topics that maximize log likelihood functions of the probability distributions; and

a determination and output unit configured to determine whether the first hyperparameter, the second hyperparameter, and the number of topics converge, and when the first hyperparameter, the second hyperparameter, and the number of topics do not converge, put the first hyperparameter, the second hyperparameter, and the number of topics into the Latent Dirichlet Allocation model for calculation until an optimal first hyperparameter, an optimal second hyperparameter, and the optimal number of topics that maximize the log likelihood functions of the probability distributions converge, and output the probability distributions, the optimal first hyperparameter, the optimal second hyperparameter, and the optimal number of topics that are eventually obtained.

5. A parameter inference calculation system based on a Latent Dirichlet Allocation model, comprising:

a master calculation apparatus configured to;

segment a document set into a plurality of document subsets;

set an initial first hyperparameter, an initial second hyperparameter, the initial number of topics, an initial global count matrix of documents and topics, and an initial global count matrix of topics and words;

send each document subset of a plurality of document subsets, the initial first hyperparameter, the initial second hyperparameter, the initial number of topics, the initial global count matrix of documents and topics, and the initial global count matrix of topics and words to each slave calculation apparatus of a plurality of slave calculation apparatuses correspondingly;

receive a local count matrix of documents and topics and a local count matrix of topics and words that are returned by each slave calculation apparatus, and perform merging to obtain a global count matrix of documents and topics and a global count matrix of topics and words;

determine whether a process of solving, by each slave calculation apparatus, the Latent Dirichlet Allocation model and updating the local count matrix of documents and topics and the local count matrix of topics and words converges, and when the process of solving the Latent Dirichlet Allocation model and updating the local count matrix of documents and topics and the local count matrix of topics and words does not converge, send the global count matrix of documents and topics and the global count matrix of topics and words to each slave calculation apparatus correspondingly for calculation, and continue to update the local count matrix of documents and topics and the local count matrix of topics and words until the process of calculating the Latent Dirichlet Allocation model and updating the local count matrix of documents and topics and the local count matrix of topics and words converges, and output the global count matrix of documents and topics and global count matrix of topics and words that are eventually obtained;

obtain, through calculation, a probability distribution between documents and topics and a probability distribution between topics and words according to the global count matrix of documents and topics and the global count matrix of topics and words;

obtain, by using an expectation maximization algorithm, a first hyperparameter, a second hyperparameter, and the number of topics that maximize log likelihood functions of the probability distributions;

determine whether the first hyperparameter, the second hyperparameter, and the number of topics converge, and when the first hyperparameter, the second hyperparameter, and the number of topics do not converge, send the first hyperparameter, the second hyperparameter, and the number of topics to each slave calculation apparatus until an optimal first hyperparameter, an optimal second hyperparameter, and the optimal number of topics that maximize the log likelihood functions of the probability distributions converge, and output the probability distributions, the optimal first hyperparameter, the optimal second hyperparameter, and the optimal number of topics that are eventually obtained; and

a plurality of slave calculation apparatuses configured to;

receive the document subset, the initial first hyperparameter, the initial second hyperparameter, the initial number of topics, the initial global count matrix of documents and topics, and the initial global count matrix of topics and words that are sent by the master calculation apparatus, calculate the Latent Dirichlet Allocation model, update the local count matrix of documents and topics and the local count matrix of topics and words, and return the local count matrix of documents and topics and the local count matrix of topics and words to the master calculation apparatus;

receive the global count matrix of documents and topics, the global count matrix of topics and words, the first hyperparameter, the second hyperparameter, and the number of topics that are sent by the master calculation apparatus and put them into the Latent Dirichlet Allocation model for local calculation.

6. The calculation system according to claim 5, wherein the master calculation apparatus comprises:

an initialization unit configured to;

segment the document set into the plurality of document subsets; and

set the initial first hyperparameter, the initial second hyperparameter, the initial number of topics, the initial global count matrix of documents and topics, and the initial global count matrix of topics and words;

a parameter estimation unit configured to;

construct the log likelihood functions of the probability distributions;

obtain, by using the expectation maximization algorithm, the first hyperparameter, the second hyperparameter, and the number of topics that maximize the log likelihood functions of the probability distributions;

determine whether the first hyperparameter, the second hyperparameter, and the number of topics converge, and when the first hyperparameter, the second hyperparameter, and the number of topics do not converge, send, through a first data transceiver unit, the first hyperparameter, the second hyperparameter, and the number of topics to the slave calculation apparatus, until the optimal first hyperparameter, the optimal second hyperparameter, and the optimal number of topics that maximize the log likelihood functions of the probability distributions converge, and output the probability distributions, the optimal first hyperparameter, the optimal second hyperparameter, and the optimal number of topics that are eventually obtained;

a Latent Dirichlet Allocation model calculation unit configured to;

merge a plurality of local count matrixes of documents and topics and local count matrixes of topics and words that are received through the first data transceiver unit to obtain the global count matrix of documents and topics and the global count matrix of topics and words;

determine whether a process of solving, by each slave calculation apparatus, the Latent Dirichlet Allocation model and updating the local count matrix of documents and topics and the local count matrix of topics and words converges, and when the process of solving the Latent Dirichlet Allocation model and updating the local count matrix of documents and topics and the local count matrix of topics and words does not converge, send the global count matrix of documents and topics and the global count matrix of topics and words to each slave calculation apparatus correspondingly for calculation, and continue to update the local count matrix of documents and topics and the local count matrix of topics and words until the process of calculating the Latent Dirichlet Allocation model and updating the local count matrix of documents and topics and the local count matrix of topics and words converges, and output the global count matrix of documents and topics and global count matrix of topics and words that are eventually obtained; and

obtain, through calculation, the probability distribution between documents and topics and the probability distribution between topics and words according to the global count matrix of documents and topics and the global count matrix of topics and words; and

the first data transceiver unit configured to;

send each document subset of a plurality of document subsets, the initial first hyperparameter, the initial second hyperparameter, the initial number of topics, the initial global count matrix of documents and topics, and the initial global count matrix of topics and words to each slave calculation apparatus of a plurality of slave calculation apparatuses correspondingly;

receive the local count matrix of documents and topics and the local count matrix of topics and words that are returned by each slave calculation apparatus;

send the global count matrix of documents and topics and the global count matrix of topics and words to each slave calculation apparatus; and

send the first hyperparameter, the second hyperparameter, and the number of topics to each slave calculation apparatus.

7. The calculation system according to claim 6, wherein the slave calculation apparatus comprises:

a second data transceiver unit configured to;

receive the document subset, the initial first hyperparameter, the initial second hyperparameter, the initial number of topics, the initial global count matrix of documents and topics, and the initial global count matrix of topics and words that are sent by the master calculation apparatus;

send the local count matrix of documents and topics and the local count matrix of topics and words to the master calculation apparatus;

receive the global count matrix of documents and topics and the global count matrix of topics and words that are sent by the master calculation apparatus; and

receive the first hyperparameter, the second hyperparameter, and the number of topics that are sent by the master calculation apparatus; and

a Latent Dirichlet Allocation model local calculation unit configured to;

calculate the Latent Dirichlet Allocation model and perform update to obtain the local count matrix of documents and topics and the local count matrix of topics and words according to the document subset, the initial first hyperparameter, the initial second hyperparameter, the initial number of topics, the initial global count matrix of documents and topics, and the initial global count matrix of topics and words that are received through the second data transceiver unit;

put the global count matrix of documents and topics and the global count matrix of topics and words that are received through the second data transceiver unit into the Latent Dirichlet Allocation model for local calculation; and

put the first hyperparameter, the second hyperparameter, and the number of topics that are received through the second data transceiver unit into the Latent Dirichlet Allocation model for local calculation.