Systems and Methods for Data Processing

• US 20150269206A1
• Filed: 06/04/2015
• Published: 09/24/2015
• Est. Priority Date: 06/25/2013
• Status: Active Application

• Current Assignee: Tencent Technology Company Limited (Tencent Holdings Limited)
• Original Assignee: Tencent Technology Company Limited (Tencent Holdings Limited)
• Inventors: Huang, Yaoxian
• Granted Patent: US 10,268,715 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/2237 Vectors, bitmaps or matrices

Abstract

Systems and methods are provided for data processing. For example, first digest values associated with first contents of a plurality of first data points are calculated, the plurality of first data points including a second data point and one or more third data points; a second digest value associated with a second content of the second data point is compared with one or more third digest values associated with third contents of the third data points, the third data points preceding the second data point; in response to the second digest value being the same as a fourth digest value associated with a fourth content of a fourth data point, the second content of the second data point is deleted, the fourth data point being within the one or more third data points; and a mapping between the second digest value and the fourth content is established.

Claims

1. A method for data processing comprising:

calculating first digest values associated with first contents of a plurality of first data points, the plurality of first data points including a second data point and one or more third data points;

comparing a second digest value associated with a second content of the second data point with one or more third digest values associated with third contents of the third data points, the third data points preceding the second data point;

in response to the second digest value being the same as a fourth digest value associated with a fourth content of a fourth data point, deleting the second content of the second data point, the fourth data point being within the one or more third data points; and

establishing a mapping between the second digest value and the fourth content.

2. The method of claim 1, further comprising:

assigning a plurality of fifth data points into a data-point group based on at least information associated with content sizes of the plurality of fifth data points, the fifth data points having a same content size;

wherein the calculating first digest values associated with first contents of a plurality of first data points and the comparing a second digest value associated with a second content of the second data point with one or more third digest values associated with third contents of the third data points include;

calculating fifth digest values associated with fifth contents of the fifth data points, the fifth data points including a sixth data point and one or more seventh data points; and

comparing a sixth digest value associated with a sixth content of the sixth data point with one or more seventh digest values associated with seventh contents of the seventh data points, the seventh data points preceding the sixth data point.

3. The method of claim 1, wherein the calculating first digest values associated with first contents of a plurality of first data points and the comparing a second digest value associated with a second content of the second data point with one or more third digest values associated with third contents of the third data points include:

comparing the second digest value with the third digest values when the first digest values associated with the first contents of the plurality of first data points are calculated.

4. The method of claim 1, further comprising:

loading a plurality of first files associated with the first data points;

detecting the first files; and

in response to a second file within the first files being empty, acquiring an eighth data point associated with a name of the second file.

5. The method of claim 4, further comprising:

merging one or more ninth data points in a third file, ninth contents of the ninth data points not being mapped to any of the first digest values.

6. A data-processing device comprising:

a calculation unit configured to calculate first digest values associated with first contents of a plurality of first data points, the plurality of first data points including a second data point and one or more third data points;

a comparison unit configured to compare a second digest value associated with a second content of the second data point with one or more third digest values associated with third contents of the third data points, the third data points preceding the second data point;

a deletion unit configured to, in response to the second digest value being the same as a fourth digest value associated with a fourth content of a fourth data point, delete the second content of the second data point, the fourth data point being within the one or more third data points; and

a mapping unit configured to establish a mapping between the second digest value and the fourth content.

7. The device of claim 6, further comprising:

a classification unit configured to assign a plurality of fifth data points into a data-point group based on at least information associated with content sizes of the plurality of fifth data points, the fifth data points having a same content size;

wherein;

the calculation unit is further configured to calculate fifth digest values associated with fifth contents of the fifth data points, the fifth data points including a sixth data point and one or more seventh data points; and

the comparison unit is configured to compare a sixth digest value associated with a sixth content of the sixth data point with one or more seventh digest values associated with seventh contents of the seventh data points, the seventh data points preceding the sixth data point.

8. The device of claim 6, wherein:

the comparison unit is configured to compare the second digest value with the third digest values when the first digest values associated with the first contents of the plurality of first data points are calculated.

9. The device of claim 6, further comprising:

a loading unit configured to load a plurality of first files associated with the first data points;

a detection unit configured to detect the first files; and

an acquisition unit configured to, in response to a second file within the first files being empty, acquire an eighth data point associated with the second file.

10. The device of claim 9, further comprising:

a merging unit configured to merge one or more ninth data points in a third file, ninth contents of the ninth data points not being mapped to any of the first digest values.

11. The device of claim 6, further comprising:

one or more data processors; and

a computer-readable storage medium;

wherein one or more of the calculation unit, the comparison unit, the detection unit, and the mapping unit are stored in the storage medium and configured to be executed by the one or more data processors.

12. A non-transitory computer readable storage medium comprising programming instructions for data processing, the programming instructions configured to cause one or more data processors to execute operations comprising:

calculating first digest values associated with first contents of a plurality of first data points, the plurality of first data points including a second data point and one or more third data points;

comparing a second digest value associated with a second content of the second data point with one or more third digest values associated with third contents of the third data points, the third data points preceding the second data point;

in response to the second digest value being the same as a fourth digest value associated with a fourth content of a fourth data point, deleting the second content of the second data point, the fourth data point being within the one or more third data points; and

establishing a mapping between the second digest value and the fourth content.