Systems and methods for data processing

• US 10,268,715 B2
• Filed: 06/04/2015
• Issued: 04/23/2019
• Est. Priority Date: 06/25/2013
• Status: Active Grant

• Current Assignee: Tencent Technology Shenzhen Company Limited (Tencent Holdings Limited)
• Original Assignee: Tencent Technology Shenzhen Company Limited (Tencent Holdings Limited)
• Inventors: Huang, Yaoxian
• Primary Examiner(s): Ruiz, Angelica
• Application Number:
  US14/730,326
• Publication Number:
  US 20150269206A1
• Time in Patent Office: 1,419 Days
• Field of Search: 707600-831, 707899, 707999001-999206
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/2237 Vectors, bitmaps or matrices

Abstract

Systems and methods are provided for data processing. For example, first digest values associated with first contents of a plurality of first data points are calculated, the plurality of first data points including a second data point and one or more third data points; a second digest value associated with a second content of the second data point is compared with one or more third digest values associated with third contents of the third data points, the third data points preceding the second data point; in response to the second digest value being the same as a fourth digest value associated with a fourth content of a fourth data point, the second content of the second data point is deleted, the fourth data point being within the one or more third data points; and a mapping between the second digest value and the fourth content is established.

Claims

1. A method for data processing comprising:

calculating first digest values associated with first contents of a plurality of first data points, the plurality of first data points including a second data point and one or more third data points;

comparing a second digest value associated with a second content of the second data point with one or more third digest values associated with third contents of the one or more third data points, the one or more third data points preceding the second data point;

in response to the second digest value being the same as a fourth digest value associated with a fourth content of a fourth data point, deleting the second content of the second data point, the fourth data point being within the one or more third data points; and

establishing a mapping between the second digest value and the fourth content,

wherein the method further comprises;

classifying data points into a plurality of data-point groups according to different content sizes of the data points, the classified data points for each data-point group having a same content size; and

assigning a plurality of fifth data points into a given data-point group, and

wherein the comparing is performed between digest values of contents of the plurality of fifth data points that belong to the given data-point group, and not performed between a digest value of a content of a fifth data point and a digest value of a content of a data point that belongs to another data-point group.

2. The method of claim 1, wherein the calculating and the comparing comprise:

calculating fifth digest values associated with fifth contents of the plurality of fifth data points, the plurality of fifth data points including a sixth data point and one or more seventh data points; and

comparing a sixth digest value associated with a sixth content of the sixth data point with one or more seventh digest values associated with seventh contents of the one or more seventh data points, the one or more seventh data points preceding the sixth data point.

3. The method of claim 1, wherein the calculating and the comparing comprise:

comparing the second digest value with the one or more third digest values when the first digest values associated with the first contents of the plurality of first data points are calculated.

4. The method of claim 1, further comprising:

loading a plurality of first files associated with the plurality of first data points;

detecting the plurality of first files; and

in response to a second file within the plurality of first files being empty, acquiring an eighth data point associated with a name of the second file.

5. The method of claim 4, further comprising:

merging one or more ninth data points in a third file, ninth contents of the one or more ninth data points not being mapped to any of the first digest values.

6. A data-processing device comprising:

a calculation unit configured to calculate first digest values associated with first contents of a plurality of first data points, the plurality of first data points including a second data point and one or more third data points;

a comparison unit configured to compare a second digest value associated with a second content of the second data point with one or more third digest values associated with third contents of the one or more third data points, the one or more third data points preceding the second data point;

a deletion unit configured to, in response to the second digest value being the same as a fourth digest value associated with a fourth content of a fourth data point, delete the second content of the second data point, the fourth data point being within the one or more third data points; and

a mapping unit configured to establish a mapping between the second digest value and the fourth content,

wherein the device further comprises a classification unit configured to classify data points into a plurality of data-point groups according to different content sizes of the data points, the classified data points for each data-point group having a same content size, and the classification unit further configured to assign a plurality of fifth data points into a given data-point group, and

wherein the comparison unit is configured to compare between digest values of contents of the plurality of fifth data points that belong to the given data-point group, and not compare between a digest value of a content of a fifth data point and a digest value of a content of a data point that belongs to another data-point group.

7. The device of claim 6, wherein:

the calculation unit is further configured to calculate fifth digest values associated with fifth contents of the plurality of fifth data points, the plurality of fifth data points including a sixth data point and one or more seventh data points; and

the comparison unit is configured to compare a sixth digest value associated with a sixth content of the sixth data point with one or more seventh digest values associated with seventh contents of the one or more seventh data points, the one or more seventh data points preceding the sixth data point.

8. The device of claim 6, wherein:

the comparison unit is configured to compare the second digest value with the one or more third digest values when the first digest values associated with the first contents of the plurality of first data points are calculated.

9. The device of claim 6, further comprising:

a loading unit configured to load a plurality of first files associated with the plurality of first data points;

a detection unit configured to detect the plurality of first files; and

an acquisition unit configured to, in response to a second file within the plurality of first files being empty, acquire an eighth data point associated with the second file.

10. The device of claim 9, further comprising:

a merging unit configured to merge one or more ninth data points in a third file, ninth contents of the one or more ninth data points not being mapped to any of the first digest values.

11. The device of claim 9, further comprising:

one or more data processors; and

a non-transitory computer-readable storage medium,

wherein one or more of the calculation unit, the comparison unit, the detection unit, and the mapping unit are stored in the non-transitory computer-readable storage medium and configured to be executed by the one or more data processors.

12. A non-transitory computer readable storage medium comprising programming instructions for data processing, the programming instructions configured to cause one or more data processors to execute operations comprising:

calculating first digest values associated with first contents of a plurality of first data points, the plurality of first data points including a second data point and one or more third data points;

comparing a second digest value associated with a second content of the second data point with one or more third digest values associated with third contents of the one or more third data points, the one or more third data points preceding the second data point;

in response to the second digest value being the same as a fourth digest value associated with a fourth content of a fourth data point, deleting the second content of the second data point, the fourth data point being within the one or more third data points; and

establishing a mapping between the second digest value and the fourth content,

wherein the programming instructions are further configured to cause the one or more data processors to;

classify data points into a plurality of data-point groups according to different content sizes of the data points, the classified data points for each data-point group having a same content size; and

assign a plurality of fifth data points into a given data-point group, and

wherein the comparing is performed between digest values of contents of the plurality of fifth data points that belong to the given data-point group, and not performed between a digest value of a content of a fifth data point and a digest value of a content of a data point that belongs to another data-point group.