Processing Repetitive Data
First Claim
1. A method performed by one or more processors configured with computer-executable instructions, the method comprising:
- processing a data structure of comparison data to be same or substantially same as a data structure of data in a repetition database, the repetition database being formed by an internal memory mapping after data in a database is processed according to a preset data structure;
comparing the comparison data with the data in the repetition database to determine whether the comparison data is repetitive data; and
in response to a result that the comparison data is not repetitive data, storing the comparison data in the database.
1 Assignment
0 Petitions
Accused Products
Abstract
The present disclosure introduces a method, an apparatus, and a system of processing repetitive data. In an example embodiment, the data structure of the comparison data to be compared is processed as having a same data structure of the data in the repetition database. The repetition database is formed by an internal memory mapping after processing data in a database according to a preset data structure. The processed comparison data is compared with data in the repetition database to determine whether the comparison data is repetitive data. After it is determined that the comparison data is not repetitive data, the comparison data is written into the database. The techniques described herein improve the efficiency of the servers for eliminating repetitive data and save the server resources.
18 Citations
20 Claims
-
1. A method performed by one or more processors configured with computer-executable instructions, the method comprising:
-
processing a data structure of comparison data to be same or substantially same as a data structure of data in a repetition database, the repetition database being formed by an internal memory mapping after data in a database is processed according to a preset data structure; comparing the comparison data with the data in the repetition database to determine whether the comparison data is repetitive data; and in response to a result that the comparison data is not repetitive data, storing the comparison data in the database. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)
-
-
12. An apparatus comprising:
-
a processing module that processes a data structure of comparison data to be same or substantially same as a data structure of data in a repetition database, the repetition database being formed by an internal memory mapping after data in a database is processed according to a preset data structure; a comparison module that compares the comparison data with the data in the repetition database to determine whether the comparison data is repetitive data; and a writing module that, in response to a result that the comparison data is not repetitive data, stores the comparison data in the database. - View Dependent Claims (13, 14, 15, 16, 17, 18)
-
-
19. A system comprising:
-
a distribution device that sends comparison data to one or more apparatuses for processing repetitive data based on load-balance of the one or more apparatuses for processing repetitive data; and at least one of the one or more apparatuses for processing repetitive data including; a processing module that processes a data structure of comparison data to be same or substantially same as a data structure of data in a repetition database, the repetition database being formed by an internal memory mapping after data in a database is processed according to a preset data structure; a comparison module that compares the comparison data with the data in the repetition database to determine that the comparison data is repetitive data after determining that first information of the comparison data is same or substantially same as first information of the data in the repetition database and a similarity degree between second information of the comparison data and second information of the data in the repetition data is higher than a threshold, the first information including at least a combination formed by one or more items in the comparison data that requires complete matching and a value of the combination after the combination is processed by a hashing algorithm or an encryption algorithm, the second information including at least a value of a portion in the comparison data that requires similarity degree matching after the portion is processed by a compression algorithm; and a writing module that, in response to a result that the comparison data is not repetitive data, stores the comparison data in the database. - View Dependent Claims (20)
-
Specification