METHOD AND SYSTEM FOR PROCESSING DATA

• US 20140101114A1
• Filed: 10/09/2012
• Published: 04/10/2014
• Est. Priority Date: 12/16/2010
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: AKIRAV, Shay H., EDELSTEIN, Ron, HIRSCH, Michael, ISH-SHALOM, Ariel J., LOYA, Liran, TZUR, Itai
• Granted Patent: US 9,933,978 B2
• Time in Patent Office: Days
• US Class Current: 707/692
• CPC Class Codes: G06F 16/1752 based on file chunks
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/0656 Data buffering arrangements
• CPC Class Codes: G06F 3/0661 Format or protocol conversion arrangements
• CPC Class Codes: G06F 3/0682 Tape device

Abstract

Methods, computer systems, and computer program products for processing data a computing environment are provided. The computer environment for data deduplication storage receives a plurality of write operations for deduplication storage of the data. The data is buffered in a plurality of buffers with overflow temporarily stored to a memory hierarchy when the data received for deduplication storage is sequential or non sequential. The data is accumulated and updated in the plurality of buffers per a data structure, the data structure serving as a fragment map between the plurality of buffers and a plurality of user file locations. The data is restructured in the plurality of buffers to form a complete sequence of a required sequence size. The data is provided as at least one stream to a stream-based deduplication algorithm for processing and storage.

Claims

1. A method for processing data by a processor device, comprising:

receiving a plurality of non-sequential write operations for deduplication storage of the data;

storing the data in a plurality of user file locations;

accumulating the data in a plurality of buffers;

restructuring the data in the plurality of buffers to form sequential data;

providing the sequential data as a plurality of streams to a stream-based deduplication algorithm for processing and storage; and

mapping, via a disk map, the data stored in the plurality of user file locations to the plurality of streams, wherein at least a portion of the data stored in the plurality of user file locations is mapped to at least two streams in the plurality of streams.

2. The method of claim 1, further comprising configuring a relationship between the plurality of user files and the plurality of buffers by performing one or more of:

associating at least one of the plurality of user files with at least one of the plurality of buffers;

opening at least one stream for the plurality of user files;

maintaining an offset for each stream, the offset representing a last offset in the plurality of user files written into the at least one stream; and

allocating the at least one of the plurality of buffers for each of the plurality of write operations, wherein the fragment map is updated reflecting the association of the plurality of buffers for each of the plurality of write operations.

3. The method of claim 2, wherein upon the data being adjacent to the offset, writing the data into plurality of streams and advancing the offset for each of the plurality of streams and freeing the plurality of buffers of the data.

4. The method of claim 1, wherein the upon the detection of an overwrite in the plurality of buffers by at least one of the plurality of write operations, creating an additional stream and layering the additional stream over an existing stream.

5. The method of claim 1, further comprising closing at least one of the plurality of user files by merging any one of adjacent and overlapping streams into at least one of the plurality of streams.

6. The method of claim 1, further comprising performing a defragmentation process after one of closing a plurality of user files, an arbitrary time period, and when the plurality of user files are determined to be quiescent, the defragmentation operation rearranging streams in each of the plurality of user files into a single stream.

7. The method of claim 1, further comprising temporarily storing an overflow of the data buffered in a plurality of buffers to a memory hierarchy.

8. A system for processing data in a computing environment, comprising:

a processor for;

receiving a plurality of non-sequential write operations for deduplication storage of the data;

storing the data in a plurality of user file locations;

accumulating the data in a plurality of buffers;

restructuring the data in the plurality of buffers to form sequential data;

providing the sequential data as a plurality of streams to a stream-based deduplication algorithm for processing and storage; and

mapping, via a disk map, the data stored in the plurality of user file locations to the plurality of streams, wherein at least a portion of the data stored in the plurality of user file locations is mapped to at least two streams in the plurality of streams.

9. The system of claim 8, wherein the processor is further for configuring a relationship between the plurality of user files and the plurality of buffers by performing one or more of:

associating at least one of the plurality of user files with at least one of the plurality of buffers;

opening at least one stream for the plurality of user files;

maintaining an offset for each stream, the offset representing a last offset in the plurality of user files written into the at least one stream; and

allocating the at least one of the plurality of buffers for each of the plurality of write operations, wherein the fragment map is updated reflecting the association of the plurality of buffers for each of the plurality of write operations.

10. The system of claim 9, wherein the processor is further for, wherein upon the data being adjacent to the offset, writing the data into the plurality of streams and advancing the offset for each of the plurality of streams and freeing the plurality of buffers of the data.

11. The system of claim 8, wherein the processor is further for, upon the detection of an overwrite in the plurality of buffers by at least one of the plurality of write operations, creating an additional stream and layering the additional stream over an existing stream.

12. The system of claim 8, wherein the processor is further for closing at least one of the plurality of user files by merging any one of adjacent and overlapping streams into at least one of the plurality of streams.

13. The system of claim 8, wherein the processor is further for running a defragmentation process after one of closing a plurality of user files, an arbitrary time period, and when the plurality of user files are determined to be quiescent, the defragmentation operation rearranging streams of each of the plurality of user files into a single stream.

14. The system of claim 8, wherein the processor is further for temporarily storing an overflow of the data buffered in a plurality of buffers to a memory hierarchy.

15. A computer program product for processing data in a computing environment by a processor device, the computer program product comprising a non-transitory computer-readable storage medium comprising:

computer code for receiving a plurality of non-sequential write operations for deduplication storage of the data;

computer code for storing the data in a plurality of user file locations;

computer code for accumulating the data in a plurality of buffers;

computer code for restructuring the data in the plurality of buffers to form sequential data;

computer code for providing the sequential data as a plurality of streams to a stream-based deduplication algorithm for processing and storage; and

computer code for mapping, via a disk map, the data stored in the plurality of user file locations to the plurality of streams, wherein at least a portion of the data stored in the plurality of user file locations is mapped to at least two streams in the plurality of streams.

16. The computer program product of claim 15, wherein the computer-readable storage medium further comprises computer code for configuring a relationship between the plurality of user files and the plurality of buffers by performing one or more of:

associating at least one of the plurality of user files with at least one of the plurality of buffers;

opening at least one stream for the plurality of user files;

maintaining an offset for each stream, the offset representing a last offset in the plurality of user files written into the at least one stream; and

allocating the at least one of the plurality of buffers for each of the plurality of write operations, wherein the fragment map is updated reflecting the association of the plurality of buffers for each of the plurality of write operations.

17. The computer program product of claim 15, wherein the computer-readable storage medium further comprises computer code for, upon the data being adjacent to the offset, writing the data into plurality of streams and advancing the offset for each of the plurality of streams and freeing the plurality of buffers of the data.

18. The computer program product of claim 15, wherein the computer-readable storage medium further comprises computer code for, upon the detection of an overwrite in the plurality of buffers by at least one of the plurality of write operations, creating an additional stream and layering the additional stream over an existing stream.

19. The computer program product of claim 15, wherein the computer-readable storage medium further comprises computer code for closing at least one of the plurality of user files by merging any one of adjacent and overlapping streams into at least one of the plurality of streams.

20. The computer program product of claim 15, wherein the computer-readable storage medium further comprises computer code for performing a defragmentation process after one of closing a plurality of user files, an arbitrary time period, and when the plurality of user files are determined to be quiescent, the defragmentation operation rearranging streams in each of the plurality of user files into a single stream.