METHOD AND DEVICE FOR PROCESSING DATA

• US 20140372457A1
• Filed: 06/03/2014
• Published: 12/18/2014
• Est. Priority Date: 06/17/2013
• Status: Abandoned Application

• Current Assignee: Tencent Technology Shenzhen Company Limited (Tencent Holdings Limited)
• Original Assignee: Tencent Technology Shenzhen Company Limited (Tencent Holdings Limited)
• Inventors: Yang, Yi, Wu, Haijun, Yu, Tao, Zou, Yongqiang, Lu, Ke, Chen, Zheng, Li, Luxin, Wu, Jiaxu, Cui, Jingbing, Xin, Diaoqin, Zou, Zan
• Application Number:
  US14/294,989
• Publication Number:
  US 20140372457A1
• Time in Patent Office: Days
• US Class Current: 707/752
• CPC Class Codes: G06F 16/2462 Approximate or statistical queries
• CPC Class Codes: G06Q 10/04 Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem" market predictions or forecasting for commercial activities G06Q30/0202

Abstract

A method and device for processing data in the field of data process are disclosed. The method includes: sorting samples according to primary keys, wherein the primary key includes a feature serial number and a sample serial number, and wherein a column value corresponding to the primary key is used as a feature value for the sample; acquiring a statistic of each feature in each category by taking the primary key and the feature value as an input key-value pair and calculating with a first algorithm model, and outputting the feature serial number and the statistic as an output key-value pair; and acquiring a contribution value of each feature to the category by performing calculation on the output key-value pair with a second algorithm model, and selecting a feature based on the contribution value. The device includes a sorting module, a first processing module and a second processing module.

Claims

1. A method for processing data, comprising:

sorting samples from the data according to a primary key, wherein the primary key comprises a feature serial number and a sample serial number;

acquiring a statistic of each feature in each category by taking the primary key and the feature value as an input key-value pair, and performing a calculation with a first algorithm model, to obtain the feature serial number and the statistic as an output key-value pair; and

acquiring a contribution value of each feature to the category by performing a calculation on the output key-value pair with a second algorithm model, and selecting a feature based on the contribution value.

2. The method according to claim 1, wherein a column value corresponding to the primary key is used as a feature value of each sample.

3. The method according to claim 1, wherein the sorting samples according to the primary key comprises:

sorting the samples according to the feature serial numbers, and then sorting the samples with the same feature serial number according to the sample serial numbers, if the primary key is sequentially spliced by the feature serial number and the sample serial number;

or

sorting the samples according to the sample serial numbers, and then sorting the samples with the same sample serial number according to the feature serial numbers, if the primary key is sequentially spliced by the sample serial number and the feature serial number.

4. The method according to claim 1, wherein the acquiring a statistic of each feature in each category by calculating with a first algorithm model comprises:

at least one of performing statistics on the feature values for the samples in each category and performing statistics on the number of occurrence of the feature in the samples in each category by using the first algorithm model.

5. The method according to claim 4, wherein the performing statistics on the feature values for the samples in each category comprises at least one of the following steps:

for each category, calculating a sum of the feature values for all the samples belonging to the category; and

for each category, calculating a sum of the feature values squared for all the samples belonging to the category.

6. The method according to claim 4, wherein the performing statistics on the number of occurrence of the feature in the samples in each category comprises:

for each category, recording for each feature, the number of times that a feature value thereof being a non-zero value in all the samples in the category, as the number of occurrence of the feature in the samples in the category.

7. The method according to claim 1, wherein the acquiring a contribution value of each feature to the category by performing calculation on the output key-value pair with a second algorithm model comprises:

performing at least one of statistics on the feature values for the samples in all the categories and statistics on the numbers of occurrence of the features in the samples in all the categories by using the second algorithm model, and

calculating the contribution value of each feature to the category according to the result of the statistics.

8. The method according to claim 1, wherein the selecting a feature based on the contribution value comprises:

determining a specified number of the contribution values in a descending order of the contribution values, and selecting the features corresponding to the determined contribution values from all the features.

9. A device for processing data, comprising:

a sorting module, configured to sort samples from the data according to a primary key, wherein the primary key comprises a feature serial number and a sample serial number;

a first processing module, configured to acquire a statistic of each feature in each category by taking the primary key and the feature value as an input key-value pair calculating with a first algorithm model, and output the feature serial number and the statistic as an output key-value pair; and

a second processing module, configured to acquire a contribution value of each feature to the category by performing calculation on the output key-value pair with a second algorithm model, and select a feature based on the contribution value.

10. The device according to claim 9, wherein a column value corresponding to the primary key is used as a feature value of each sample.

11. The device according to claim 10, wherein the sorting module comprises:

a first sorting unit, configured to sort the samples according to the feature serial numbers, and then sort the samples with the same feature serial number according to the sample serial numbers, in the case where the primary key is sequentially spliced by the feature serial number and the sample serial number;

or

a second sorting unit, configured to sort the samples according to the sample serial numbers, and then sort the samples with the same sample serial number according to the feature serial numbers, in the case where the primary key is sequentially spliced by the sample serial number and the feature serial number.

12. The device according to claim 10, wherein the first processing module comprises:

a statistics unit, configured for performing at least one of statistics on the feature values for the samples in each category and statistics on the number of occurrence of the feature in the samples in each category by using the first algorithm model.

13. The device according to claim 12, wherein the statistics unit is configured for, for each category, at least one of the following:

calculating a sum of the feature values for all the samples belonging to the category; and

calculating a sum of square of the feature values for all the samples belonging to the category.

14. The device according to claim 12, wherein the statistics unit is configured for:

for each category, recording for each feature, the number of times that a feature value thereof being a non-zero value in all the samples in the category, as the number of occurrence of the feature in the samples in the category.

15. The device according to claim 10, wherein the second processing module comprises:

a calculation unit, configured for performing at least one of statistics on the feature values for the samples in all the categories and statistics on the numbers of occurrence of the features in the samples in all the categories by using the second algorithm model, and calculating the contribution value of each feature to the category according to the result of the statistics.

16. The device according to claim 10, wherein the second processing module comprises:

a selection unit, configured for determining a specified number of the contribution values in a descending order of the contribution values, and selecting the features corresponding to the determined contribution values from all the features.