ON-DEMAND DATA DEDUPLICATION

• US 20120109907A1
• Filed: 10/30/2010
• Published: 05/03/2012
• Est. Priority Date: 10/30/2010
• Status: Abandoned Application

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Mandagere, Nagapramod S., Pease, David A., Uttamchandani, Sandeep M., Zhou, Pin
• Application Number:
  US12/916,524
• Publication Number:
  US 20120109907A1
• Time in Patent Office: Days
• US Class Current: 707/692
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors

Abstract

Embodiments of the invention relate to performing on-demand data deduplication for managing data and storage space. Redundant data in a system is detected. Availability of data storage space in the system is periodically evaluated. Performance parameters of the system are evaluated. Detected redundant data is selected based on the data storage availability and performance parameters of the system. If at least a portion of the selected redundant data is to be deduplicated is determined.

Claims

1. A method comprising:

detecting redundant data in a system;

periodically evaluating availability of data storage space in the system;

evaluating performance parameters of the system;

selecting detected redundant data based on the availability of data storage space of the system; and

determining if at least a portion of the selected redundant data is to be deduplicated.

2. The method of claim 1 further comprising:

wherein determining if at least a portion of the selected redundant data is to be deduplicated comprises;

determining the availability of data storage in the system; and

determining the performance of the system; and

if availability of data storage in the system is less than a predetermined value and if the performance of the system is greater than a predetermined value, then deduplicating at least a portion of the selected redundant data.

3. The method of claim 1 further comprising:

wherein redundant data in the system comprises;

detecting data as data is ingested by the system for determining if the ingested data is redundant data; and

detecting data stored in the system for determining if the data stored in the system is redundant data.

4. The method of claim 2 further comprising:

wherein deduplicating at least a portion of the selected redundant data comprises;

deduplicating redundant data in the system by logically chunking redundant data into data extents.

5. The method of claim 4 further comprising:

assigning a hash value and an extent identification to each data extent; and

recording the hash value, the extent identification, and an extent boundary for each data extent into a hash map, wherein at least one recorded hash value in the hash map corresponds to more than one data extent identification.

6. The method of claim 5 further comprising:

wherein a recorded hash value in the hash map corresponding to more than one data extent identification corresponds to redundant data extents, the redundant data extents corresponding to redundant data.

7. The method of claim 1 further comprising:

removing a data object from redundant data, the removed data object comprising a suppressed data object; and

recording suppressed data objects in a probabilistic data structure, the data structure configured for determining if data objects are suppressed data objects.

8. The method of claim 7 further comprising:

selecting a data object to be accessed;

searching the probabilistic data structure for determining if the selected data object is a suppressed data object; and

if the selected data object is a suppressed data object, then retrieving data extents comprising the selected data object and reconstructing the selected data object.

9. A computer program product comprising:

a computer readable storage medium having computer readable program code embodied therewith, the computer readable program code comprising;

computer readable program code configured to detect redundant data;

computer readable program code configured to periodically evaluate availability of data storage space in the system;

computer readable program code configured to periodically evaluate performance parameters of the system;

computer readable program code configured to select redundant data based on the evaluated availability of data storage space and performance parameters of the system; and

computer readable program code configured to determine if at least a portion of the selected redundant data is to be deduplicated.

10. The computer program product of claim 9 further comprising:

computer readable program code configured to determine if availability of data storage space in the system is determined to be less than a predetermined value and if the performance of the system is determined to be greater than a predetermined value, then the computer readable program code deduplicates at least a portion of the selected redundant data.

11. The computer program product of claim 9 further comprising:

computer readable program code configured to detect data as data is ingested by the system for determining if the data being ingested is redundant data; and

computer readable program code configured to detect data stored in the system for determining if the data stored in the system is redundant data.

12. The computer program product of claim 9 further comprising:

computer readable program code configured to deduplicate redundant data in the system by logically chunking redundant data into data extents.

13. The computer program product of claim 10 further comprising:

computer readable program code configured to assign a hash value and an extent identification to each data extent; and

computer readable program code configured to record the hash value, the extent identification, and an extent boundary for each data extent into a hash map, wherein at least one recorded hash value in the hash map corresponds to more than one data extent identification, wherein a recorded hash value in the hash map corresponding to more than one data extent identification corresponds to redundant data extents, the redundant data extents corresponding to redundant data.

14. The computer program product of claim 9 further comprising:

computer readable program code configured to remove a data object from redundant data, the removed data object comprising a suppressed data object; and

computer readable program code configured to record suppressed data objects in a probabilistic data structure, the data structure configured to determine if data objects are suppressed data objects.

15. The computer program product of claim 14 further comprising:

computer readable program code configured to select a data object to be accessed;

computer readable program code configured to search the probabilistic data structure to determine if the selected data object is a suppressed data object; and

computer readable program code configured to determine if the selected data object is a suppressed data object, the computer readable program code configured to retrieve data extents comprising the selected data object and reconstructing the selected data object.

17. A system comprising:

a processor operative to execute computer usable program code;

a memory for storing instructions operable with the processor;

at least one of a network interface and a peripheral device interface for receiving user input and for sending and receiving data;

a data storage for storing data coupled to the processor; and

a computer usable medium having computer usable program code embodied therewith, the computer usable program code comprising;

computer usable program code configured to detect redundant data;

computer readable program code configured to periodically evaluate availability data storage space in the system;

computer readable program code configured to evaluate performance parameters of the system;

computer readable program code configured to select redundant data based on the evaluated availability of data storage space and performance parameters of the system; and

computer readable program code configured to determine if at least a portion of the selected redundant data is to be deduplicated.

18. The system of claim 17 further comprising:

computer readable program code configured to determine the availability of data storage space in the system and to determine the performance of the system; and

computer readable program code configured to determine if availability of data storage space in the system is determined to be less than a predetermined value and if the performance of the system is determined to be greater than a predetermined value, then the computer readable program code deduplicates at least a portion of the selected redundant data.

19. The system of claim 17 further comprising:

computer readable program code configured to deduplicate redundant data in the system by logically chunking redundant data into data extents;

computer readable program code configured to assign a hash value and an extent identification to each data extent; and

computer readable program code configured to record the hash value, the extent identification, and an extent boundary for each data extent into a hash map, wherein at least one recorded hash value in the hash map corresponds to more than one data extent identification, wherein a recorded hash value in the hash map corresponding to more than one data extent identification corresponds to redundant data extents, the redundant data extents corresponding to redundant data in the system.

20. The system of claim 17 further comprising:

computer readable program code configured to remove a data object from redundant data, the removed data object comprising a suppressed data object;

computer readable program code configured to record suppressed data objects in a probabilistic data structure, the data structure configured to determine if data objects are suppressed data objects.

computer readable program code configured to select a data object to be accessed;

computer readable program code configured to search the probabilistic data structure to determine if the selected data object is a suppressed data object; and

computer readable program code configured to determine if the selected data object is a suppressed data object, the computer readable program code configured to retrieve data extents comprising the selected data object and reconstructing the selected data object.