KEYWORD EXTRACTION METHOD, APPARATUS AND SERVER

• US 20190163690A1
• Filed: 01/31/2019
• Published: 05/30/2019
• Est. Priority Date: 11/10/2016
• Status: Active Grant

• Current Assignee: Tencent Technology Company Limited (Tencent Holdings Limited)
• Original Assignee: Tencent Technology Company Limited (Tencent Holdings Limited)
• Inventors: Bao, Xiao
• Granted Patent: US 10,878,004 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing
• CPC Class Codes: G06F 16/3346 using probabilistic model
• CPC Class Codes: G06F 18/211 Selection of the most significant subset of features
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 40/216 using statistical methods
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/30 Semantic analysis

Abstract

A keyword extraction method is provided. A candidate keyword from target text is extracted by a server. For each candidate keyword, each effective feature corresponding to the candidate keyword is obtained by the server. Calculation is performed by the server according to each effective feature corresponding to the candidate keyword and a weighting coefficient respectively corresponding to each effective feature, to obtain a probability that the candidate keyword belongs to a target keyword, and the candidate keyword is determined as the target keyword of the target text based on the probability.

Claims

1. A keyword extraction method, comprising:

extracting, by a server, a candidate keyword from target text;

obtaining, by the server for each candidate keyword, each effective feature corresponding to the candidate keyword; and

obtaining, by the server, a probability that the candidate keyword belongs to a target keyword according to each effective feature corresponding to the candidate keyword and a weighting coefficient respectively corresponding to each effective feature, and determining the candidate keyword as the target keyword of the target text based on the probability.

2. The method according to claim 1, wherein the performing comprises:

inputting, by the server, each effective feature corresponding to the candidate keyword to a keyword evaluation model, obtaining the probability that the candidate keyword belongs to the target keyword by using the keyword evaluation model according to each effective feature corresponding to the candidate keyword and the weighting coefficient respectively corresponding to each effective feature.

3. The method according to claim 2, further comprising:

inputting, by the server, each effective feature corresponding to the candidate keyword to a feature processing model, and performing, by using the feature processing model, predetermined processing on each effective feature corresponding to the candidate keyword, to obtain the processed effective feature corresponding to the candidate keyword, wherein

the processed effective feature corresponding to the candidate keyword is used as an input of the keyword evaluation model, and the predetermined processing comprises at least one of continuous feature discretization processing and feature combination processing.

4. The method according to claim 3, further comprising:

constructing, by the server, a training sample of a gradient boosting decision tree (GBDT) algorithm;

training, by the server, the training sample of the GBDT algorithm by using the GBDT algorithm, to obtain a GBDT model, the feature processing model being based on the GBDT model, wherein

the training sample of the GBDT algorithm comprises a first positive sample and a first negative sample, the first positive sample being a word belonging to the target keyword in sample text and being represented by an effective feature corresponding to the word belonging to the target keyword in the sample text, the first negative sample being a word not belonging to the target keyword in the sample text and being represented by an effective feature corresponding to the word not belonging to the target keyword in the sample text,

the GBDT model comprises a plurality of decision trees, each leaf node of a decision tree corresponding to a processed effective feature.

5. The method according to any one of claim 4, further comprising:

constructing, by the server, a training sample of a logistic regression (LR) algorithm according to a prediction result of each training sample of the GBDT algorithm in each of the plurality of decision trees; and

training, by the server, the training sample of the LR algorithm by using the LR algorithm, to obtain an LR model, the feature processing model being based on the LR model, wherein

the training sample of the LR algorithm comprises a second positive sample and a second negative sample, the second positive sample being the word belonging to the target keyword in the sample text and being represented by a processed effective feature corresponding to the word belonging to the target keyword in the sample text, the second negative sample being the word not belonging to the target keyword in the sample text and being represented by a processed effective feature corresponding to the word not belonging to the target keyword in the sample text.

6. The method according to claim 2, further comprising:

testing, by the server, the keyword evaluation model by using a test sample; and

obtaining, by the server, a quality evaluation parameter of the keyword evaluation model according to a test result, wherein

the quality evaluation parameter comprises at least one of precision, a recall rate, and ranking precision.

7. The method according to claim 1, wherein a type of the effective feature comprises at least one of:

a term frequency feature, a location feature, a semantic feature, and a linguistic feature,

the term frequency feature indicating a frequency at which a given word appears in text;

the location feature indicating a location at which the given word appears in the text;

the semantic feature indicating correlation between the given word and a subject of the text; and

the linguistic feature indicating a basic attribute of the given word.

8. The method according to claim 1, further comprising:

obtaining, by the server, a plurality of candidate features and a plurality of marked samples, each of the plurality of marked samples being text in which the target keyword has been marked;

performing statistics collection, by the server for each candidate feature, on a sample ratio of each feature value of the candidate feature; and

in response to a relationship between the sample ratio and a feature value of the candidate feature satisfying a preset condition, determining, by the server, the candidate feature as an effective feature, wherein

the sample ratio is a positive sample ratio or a negative sample ratio, the positive sample ratio being a ratio of a quantity of positive samples in the plurality of marked samples to a quantity of the plurality of marked samples, the negative sample ratio being a ratio of a quantity of negative samples in the plurality of marked samples to the quantity of the plurality of marked samples, a positive sample being a marked sample in which the target keyword satisfying the feature value of the candidate feature exists, and a negative sample being a marked sample in which the target keyword satisfying the feature value of the candidate feature does not exist.

9. The method according to claim 1, wherein the determining comprises:

determining the candidate keyword as the target keyword of the target text based on the probability being greater than a probability threshold.

19. The server, comprising:

one or more processors; and

a memory,

the memory storing one or more programs, the one or more programs being configured to be executed by the one or more processors, to implement the keyword extraction method according to claim 1.

10. A keyword extraction apparatus, comprising:

at least one memory configured to store computer program code; and

at least one processor configured to access the at least one memory and operate according to the computer program code, the computer program code including;

extraction code configured to cause the at least one processor to extract a candidate keyword from target text;

first obtaining code configured to cause the at least one processor to obtain, for each candidate keyword, each effective feature corresponding to the candidate keyword; and

first determining code configured to cause the at least one processor to obtain a probability that the candidate keyword belongs to a target keyword according to each effective feature corresponding to the candidate keyword and a weighting coefficient respectively corresponding to each effective feature, and determine the candidate keyword as the target keyword of the target text based on the probability.

11. The apparatus according to claim 10, wherein the first determining code further causes the at least one processor to:

input each effective feature corresponding to the candidate keyword to a keyword evaluation model, obtain the probability that the candidate keyword belongs to the target keyword by using the keyword evaluation model according to each effective feature corresponding to the candidate keyword and the weighting coefficient respectively corresponding to each effective feature, and determine the candidate keyword as the target keyword of the target text based on the probability.

12. The apparatus according to claim 11, wherein the computer program code further comprises:

processing code configured to cause the at least one processor to input each effective feature corresponding to the candidate keyword to a feature processing model, and perform, by using the feature processing model, predetermined processing on each effective feature corresponding to the candidate keyword, to obtain the processed effective feature corresponding to the candidate keyword, wherein

the processed effective feature corresponding to the candidate keyword is used as input of the keyword evaluation model, and the predetermined processing comprises at least one of continuous feature discretization processing and feature combination processing.

13. The apparatus according to claim 12, wherein the computer program code further comprises:

first construction code configured to cause the at least one processor to construct a training sample of a gradient boosting decision tree (GBDT) algorithm;

first training code configured to cause the at least one processor to train the training sample of the GBDT algorithm by using the GBDT algorithm, to obtain a GBDT model, the feature processing model being based on the GBDT model, wherein

the training sample of the GBDT algorithm comprises a first positive sample and a first negative sample;

the first positive sample is a word belonging to the target keyword in sample text, the first positive sample is represented by an effective feature corresponding to the word belonging to the target keyword in the sample text;

the first negative sample is a word not belonging to the target keyword in the sample text, and the first negative sample is represented by an effective feature corresponding to the word not belonging to the target keyword in the sample text; and

the GBDT model comprises a plurality of decision trees, and each leaf node of the decision tree corresponds to a processed effective feature.

14. The apparatus according to claim 13, wherein the computer program code further comprises:

second construction code configured to cause the at least one processor to construct a training sample of a logistic regression (LR) algorithm according to a prediction result of each training sample of the GBDT algorithm in each of the plurality of decision trees; and

second training code configured to cause the at least one processor to train the training sample of the LR algorithm by using the LR algorithm, to obtain an LR model, the feature processing model being based on the LR model, wherein

the training sample of the LR algorithm comprises a second positive sample and a second negative sample; and

the second positive sample is a word belonging to the target keyword in the sample text, the second positive sample is represented by a processed effective feature corresponding to the word belonging to the target keyword in the sample text; and

the second negative sample is a word not belonging to the target keyword in the sample text, and the second negative sample is represented by a processed effective feature corresponding to the word not belonging to the target keyword in the sample text.

15. The apparatus according to claim 11, wherein the computer program code further comprises:

testing code configured to cause the at least one processor to test the keyword evaluation model by using a test sample; and

second obtaining code configured to cause the at least one processor to obtain a quality evaluation parameter of the keyword evaluation model according to a test result, wherein

the quality evaluation parameter comprises at least one of precision, a recall rate, and ranking precision.

16. The apparatus according to claim 10, wherein a type of the effective feature comprises at least one of:

a term frequency feature, a location feature, a semantic feature, and a linguistic feature,

the term frequency feature indicating a frequency at which a given word appears in text;

the location feature indicating a location at which the given word appears in the text;

the semantic feature indicating correlation between the given word and a subject of the text; and

the linguistic feature indicating a basic attribute of the given word.

17. The apparatus according to claim 10, further comprising:

third obtaining code configured to cause the at least one processor to obtain a plurality of candidate features and a plurality of marked samples, each of the plurality of marked samples being text in which the target keyword has been marked;

statistic collecting code configured to cause the at least one processor to perform statistics collection, for each candidate feature, on a sample ratio of each feature value of the candidate feature and;

second determining code configured to cause the at least one processor to;

in response to a relationship between the sample ratio and a feature value of the candidate feature satisfying a preset condition, determine the candidate feature as an effective feature, wherein

the sample ratio is a positive sample ratio or a negative sample ratio, the positive sample ratio being a ratio of a quantity of positive samples in the plurality of marked samples to a quantity of the plurality of marked samples, the negative sample ratio being a ratio of a quantity of negative samples in the plurality of marked samples to the quantity of the plurality of marked samples, a positive sample being a marked sample in which the target keyword satisfying the feature value of the candidate feature exists, and a negative sample being a marked sample in which the target keyword satisfying the feature value of the candidate feature does not exist.

18. The apparatus according to claim 10, wherein the first determining code causes the at least one processor to determine the candidate keyword as the target keyword of the target text based on the probability being greater than a probability threshold.

20. A non-transitory computer readable media, storing a plurality of instructions, the plurality of instructions, when executed by one or more processors in a server, cause the one or more processors to perform:

extracting a candidate keyword from target text;

obtaining, for each candidate keyword, each effective feature corresponding to the candidate keyword; and

performing calculation according to each effective feature corresponding to the candidate keyword and a weighting coefficient respectively corresponding to each effective feature, to obtain a probability that the candidate keyword belongs to a target keyword, and determining the candidate keyword as the target keyword of the target text based on the probability.