Repetitive data block deleting system and method

• US 9,552,161 B2
• Filed: 10/29/2013
• Issued: 01/24/2017
• Est. Priority Date: 12/12/2012
• Status: Expired due to Fees

• Current Assignee: Shenzhen Airdrawing Technology Service Co., Ltd.
• Original Assignee: Shenzhen Airdrawing Technology Service Co., Ltd.
• Inventors: Chai, Zhi-Quan, Li, Da-Peng, Lin, Hai-Hong, Lee, Chung-I
• Primary Examiner(s): ASPINWALL, EVAN S
• Application Number:
  US14/065,490
• Publication Number:
  US 20140164339A1
• Time in Patent Office: 1,183 Days
• Field of Search: 707/692
• US Class Current: 1/1
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

An analysis device obtains hash lists from databases of a server cluster. The analysis device determines repetitive hash values and repetitive data blocks. The analysis device deletes the repetitive data blocks from servers of the server cluster.

Claims

1. An analysis device in electronic communication with a plurality of servers in a server cluster, each server comprising data blocks of files, comprising:

at least one processor; and

a storage system that stores one or more programs, when executed by the at least one processor, cause the at least one processor to perform a repetitive data block deleting method, the method comprising;

monitoring an available storage capacity of each storage space in each server of the server cluster;

obtaining all hash lists from all databases of the server cluster when the available storage capacity of one storage space does not exceed a predetermined storage capacity;

searching for each repetitive hash value from the obtained hash lists, and repetitive data blocks corresponding to the repetitive hash value;

obtaining a maximum storage space according to a pointer corresponding to each repetitive data block, and sending the pointer corresponding to the repetitive data block in the maximum storage space to other servers, wherein the maximum storage space is defined as the storage space that already stores one repetitive data block and remains a maximum available storage capacity for storing data; and

deleting repetitive data blocks from the other servers.

2. The analysis device of claim 1, wherein each data block comprises a name, and the name of each data block is generated in an alphabetical order or in a numerical order.

3. The analysis device of claim 1, wherein the hash value is determined as a repetitive hash value upon the condition that the hash value is the same as at least one other hash values.

4. The analysis device of claim 3, wherein the data block is determined as the repetitive data block upon the condition the data block corresponds to the repetitive hash value.

5. The analysis device of claim 2, wherein a method of downloading the file from the server comprises:

the client obtains the hash value of each data block of the file from the hash list stored in the database;

the client downloads each data block of the file according to the pointer of each data block from the server;

the client calculates a hash value of each downloaded data block, and determines if the hash value of each downloaded data block exists in the hash list stored in the database;

the client combines all downloaded data blocks to generate the file in the client according to the name of each downloaded data block, when the hash value of each downloaded data block exists in the hash list stored in the database;

the client calculates the hash value of the generated file and determines if the calculated hash value of the generated file exists in the hash list stored in the database; and

the client displays the generated file when the calculated hash value of the generated file exists in the hash list stored in the database.

6. A repetitive data block deleting method implemented by an analysis device, the analysis device in electronic communication with a plurality of servers in a server cluster, each server comprising data blocks of files, the method comprising:

monitoring an available storage capacity of each storage space in each server of the server cluster;

obtaining all hash lists from all databases of the server cluster when the available storage capacity of one storage space does not exceed a predetermined storage capacity;

searching for each repetitive hash value from the obtained hash lists, and repetitive data blocks corresponding to the repetitive hash value;

obtaining a maximum storage space according to a pointer corresponding to each repetitive data block, and sending the pointer corresponding to the repetitive data block in the maximum storage space to other servers, wherein the maximum storage space is defined as the storage space that already stores one repetitive data block and remains a maximum available storage capacity for storing data; and

deleting repetitive data blocks from the other servers.

7. The method of claim 6, wherein each data block comprises a name, and the name of each data block is generated in an alphabetical order or in a numerical order.

8. The method of claim 6, wherein the hash value is determined as a repetitive hash value upon the condition that the hash value is the same as at least one other hash values.

9. The method of claim 8, wherein the data block is determined as the repetitive data block upon the condition the data block corresponds to the repetitive hash value.

10. The method of claim 7, wherein a method of downloading the file from the server comprises:

the client obtains the hash value of each data block of the file from the hash list stored in the database;

the client downloads each data block of the file according to the pointer of each data block from the server;

the client calculates a hash value of each downloaded data block, and determines if the hash value of each downloaded data block exists in the hash list stored in the database;

the client combines all downloaded data blocks to generate the file in the client according to the name of each downloaded data block, when the hash value of each downloaded data block exists in the hash list stored in the database;

the client calculates the hash value of the generated file and determines if the calculated hash value of the generated file exists in the hash list stored in the database; and

the client displays the generated file when the calculated hash value of the generated file exists in the hash list stored in the database.

11. A repetitive data block deleting method implemented by an analysis device, the analysis device in electronic communication with a plurality of servers in a server cluster, each server comprising data blocks of files, the method comprising:

setting a trigger event in each database of the server cluster;

triggering each database by the trigger event to send all hash lists to the analysis device when the number of the hash lists stored in the database exceeds a predetermined number;

searching for each repetitive hash value from the obtained hash lists, and repetitive data blocks corresponding to the repetitive hash value;

obtaining a maximum storage space according to a pointer corresponding to each repetitive data block, and sending the pointer corresponding to the repetitive data block in the maximum storage space to other servers, wherein the maximum storage space is defined as the storage space that already stores one repetitive data block and remains a maximum available storage capacity for storing data; and

deleting repetitive data blocks from the other servers.

12. The method of claim 11, wherein each data block comprises a name, and the name of each data block is generated in an alphabetical order or in a numerical order.

13. The method of claim 11, wherein the hash value is determined as a repetitive hash value upon the condition that the hash value is the same as at least one other hash values.

14. The method of claim 13, wherein the data block is determined as the repetitive data block upon the condition the data block corresponds to the repetitive hash value.

15. The method of claim 12, wherein a method of downloading the file from the server cluster comprises:

the client obtains the hash value of each data block of the file from the hash list stored in the database;

the client downloads each data block of the file according to the pointer of each data block from the server;

the client calculates a hash value of each downloaded data block, and determines if the hash value of each downloaded data block exists in the hash list stored in the database;

the client combines all downloaded data blocks to generate the file in the client according to the name of each downloaded data block, when the hash value of each downloaded data block exists in the hash list stored in the database;

the client calculates the hash value of the generated file and determines if the calculated hash value of the generated file exists in the hash list stored in the database; and

the client displays the generated file when the calculated hash value of the generated file exists in the hash list stored in the database.