Secure relational file system with version control, deduplication, and error correction

• US 9,697,228 B2
• Filed: 04/14/2015
• Issued: 07/04/2017
• Est. Priority Date: 04/14/2014
• Status: Active Grant

• Current Assignee: Vembu Technologies Private Limited
• Original Assignee: Vembu Technologies Private Limited
• Inventors: Subramanian, Jayashree, Vembu, Sekar
• Primary Examiner(s): Stevens, Robert
• Application Number:
  US14/685,604
• Publication Number:
  US 20150293817A1
• Time in Patent Office: 812 Days
• Field of Search: 707645
• CPC Class Codes: G06F 11/1435 using file system or storage system metadata
• CPC Class Codes: G06F 11/1448 Management of the data involved in backup or backup restore
• CPC Class Codes: G06F 16/184 implemented as replicated file system

Abstract

A computer implemented method and a secure relational file system (SRFS) for storing and managing data for backup and restore are provided. The SRFS receives data, generates first metadata including file-to-sector mapping information, splits the data into fixed sized data chunks (FSDCs), generates second metadata including logical boundaries used for splitting, creates fixed sized data blocks by prepending the second metadata to the FSDCs, splits each FSDC into variable sized data chunks (VSDCs), generates third metadata including unique identifiers (UIDs) for the VSDCs, creates variable sized data blocks (VSDBs) by prepending the third metadata and the second metadata to each VSDC, identifies unique variable sized data chunks (UVSDCs) of the VSDBs using the UIDs, stores the UVSDCs in chunk files, and stores the first metadata, the second metadata extracted from the VSDBs, and storage locations of the UVSDCs with the third metadata of the UVSDCs and duplicate VSDCs in databases.

Claims

1. A computer implemented method for storing data and managing changes to said data in a storage device for backup and restore, said method employing a secure relational file system comprising at least one processor configured to execute computer program instructions for performing said method, said method comprising:

receiving said data for said backup and said restore by said secure relational file system;

generating first metadata associated with said received data for said backup by said secure relational file system, said first metadata comprising file-to-sector mapping information associated with disk storage of said received data in a plurality of sectors in said storage device, said file-to-sector mapping information configured to identify a position of said received data in said storage device and one or more of unused sectors among said sectors and unused data blocks in said storage device for reducing size of storage required for said backup of said received data;

splitting said received data into a plurality of fixed sized data chunks by said secure relational file system using said generated first metadata;

generating second metadata for each of said fixed sized data chunks by said secure relational file system for said restore of said received data, said second metadata comprising logical boundaries used for said splitting of said received data;

creating a plurality of fixed sized data blocks by prepending said generated second metadata to said fixed sized data chunks by said secure relational file system;

splitting said fixed sized data chunks of each of said created fixed sized data blocks into a plurality of variable sized data chunks by said secure relational file system using a plurality of context aware breakpoints in said fixed sized data chunks;

generating a unique identifier for each of said variable sized data chunks by said secure relational file system;

generating third metadata for said each of said variable sized data chunks by said secure relational file system, said third metadata comprising said generated unique identifier;

creating a plurality of variable sized data blocks by said secure relational file system by prepending said generated third metadata of said each of said variable sized data chunks and said generated second metadata from said each of said created fixed sized data blocks to said each of said variable sized data chunks;

identifying unique variable sized data chunks of said created variable sized data blocks using each said generated unique identifier from said generated third metadata by said secure relational file system, in communication with one or more databases of said secure relational file system for deduplication of said variable sized data chunks;

storing said identified unique variable sized data chunks in one or more chunk files of said secure relational file system by said secure relational file system for said backup and said restore of said received data;

adding a location of storage of each of said identified unique variable sized data chunks in said one or more chunk files to said generated third metadata by said secure relational file system; and

storing said generated first metadata, said generated second metadata extracted from each of said created variable sized data blocks, and said generated third metadata of said identified unique variable sized data chunks and duplicate variable sized data chunks extracted from said each of said created variable sized data blocks in said one or more databases with minimum additional storage consumption by said secure relational file system for said restore of said received data.

2. The computer implemented method of claim 1, wherein said first metadata further comprises metadata associated with archived data and archived changes to said data for availability during said backup, metadata of a file system used in creation of said disk storage, metadata stored by a file system used in creation of a disk image of said disk storage, metadata of a virtual machine comprising said archived data, and one or more of attributes, permissions, locations, content specifics, offsets, a size of said disk storage, and directory information of said received data.

3. The computer implemented method of claim 1, wherein said second metadata further comprises a unique identifier for each of said fixed sized data chunks, wherein said unique identifier is a checksum of content in said each of said fixed sized data chunks.

4. The computer implemented method of claim 1, wherein said third metadata further comprises backup data associated with said backup, wherein said backup data comprises data of a file system used for storing said backup data and an index of contents of said variable sized data chunks.

5. The computer implemented method of claim 1, further comprising performing an integrity check configured to rectify errors in said identified unique variable sized data chunks by said secure relational file system by adding a parity file to each of said one or more chunk files where said identified unique variable sized data chunks are stored.

6. The computer implemented method of claim 5, further comprising maintaining information of said parity file in a backup file stored at one or more of a plurality of data restore levels by said secure relational file system for facilitating said integrity check at said one or more of said data restore levels, wherein said data restore levels comprise a disk image level, a chunk level, a repository level, a client level, and a backup level.

7. The computer implemented method of claim 1, further comprising restoring backup data by said secure relational file system, said restoration of said backup data comprising:

receiving said stored third metadata comprising said location of said storage of said each of said identified unique variable sized data chunks in said one or more chunk files and a pointer for each of said duplicate variable sized data chunks from said one or more databases by said secure relational file system, wherein said pointer points to a unique instance of said each of said duplicate variable sized data chunks;

retrieving said identified unique variable sized data chunks from said one or more chunk files by said secure relational file system using said location of said storage of said each of said identified unique variable sized data chunks in said one or more chunk files from said received third metadata;

retrieving, from said one or more chunk files, said unique instance of said each of said duplicate variable sized data chunks, to which said pointer from said received third metadata points, by said secure relational file system for retrieving said each of said duplicate variable sized data chunks;

receiving said stored second metadata comprising said logical boundaries used for said splitting of said received data into said fixed sized data chunks and said stored first metadata comprising said file-to-sector mapping information from said one or more databases; and

assembling said retrieved unique variable sized data chunks and said retrieved unique instance of said each of said duplicate variable sized data chunks in a predefined order by said secure relational file system for regenerating and restoring said each of said fixed sized data chunks using said received second metadata at a target location identified using said received first metadata.

8. The computer implemented method of claim 1, further comprising performing an incremental backup for maintaining data changed after each said backup of said received data, wherein said incremental backup comprises identifying and storing changed unique variable sized data chunks and additional unique variable sized data chunks in said one or more chunk files by said secure relational file system based on said logical boundaries obtained from said stored second metadata.

9. The computer implemented method of claim 8, further comprising storing metadata of each of said changed unique variable sized data chunks and each of said additional unique variable sized data chunks in said one or more databases by said secure relational file system for allowing each said incremental backup to be used instantly as a full backup, without merging with previous increments.

10. A secure relational file system for storing data and managing changes to said data in a storage device for backup and restore, said secure relational file system comprising:

a non-transitory computer readable storage medium configured to store computer program instructions defined by modules of said secure relational file system;

at least one processor communicatively coupled to said non-transitory computer readable storage medium, said at least one processor configured to execute said defined computer program instructions; and

said modules of said secure relational file system comprising;

a data reader configured to receive and read said data for said backup and said restore;

a first metadata generation module configured to generate first metadata associated with said received data for said backup, said first metadata comprising file-to-sector mapping information associated with disk storage of said received data in a plurality of sectors in said storage device, said file-to-sector mapping information configured to identify a position of said received data in said storage device and one or more of unused sectors among said sectors and unused data blocks in said storage device for reducing size of storage required for said backup of said received data;

a fixed sized data splitter configured to split said received data into a plurality of fixed sized data chunks using said generated first metadata;

a second metadata generation module configured to generate second metadata for each of said fixed sized data chunks for said restore of said received data, said second metadata comprising logical boundaries used by said fixed sized data splitter for said splitting of said received data;

said fixed sized data splitter further configured to create a plurality of fixed sized data blocks by prepending said generated second metadata to said fixed sized data chunks;

a variable sized data splitter configured to split said fixed sized data chunks of each of said created fixed sized data blocks into a plurality of variable sized data chunks using a plurality of context aware breakpoints in said fixed sized data chunks;

a unique identifier generator configured to generate a unique identifier for each of said variable sized data chunks;

a third metadata generation module configured to generate third metadata for said each of said variable sized data chunks, said third metadata comprising said generated unique identifier;

said variable sized data splitter configured to create a plurality of variable sized data blocks by prepending said generated third metadata of said each of said variable sized data chunks and said generated second metadata from said each of said created fixed sized data blocks to said each of said variable sized data chunks;

a deduplication engine configured to identify unique variable sized data chunks of said created variable sized data blocks using each said generated unique identifier from said generated third metadata, in communication with one or more databases of said secure relational file system for deduplication of said variable sized data chunks;

a data writer configured to store said identified unique variable sized data chunks in one or more chunk files for said backup and said restore of said received data;

said data writer further configured to add a location of storage of each of said identified unique variable sized data chunks in said one or more chunk files to said generated third metadata; and

said data writer further configured to store said generated first metadata, said generated second metadata extracted from each of said created variable sized data blocks, and said generated third metadata of said identified unique variable sized data chunks and duplicate variable sized data chunks extracted from said each of said created variable sized data blocks in said one or more databases with minimum additional storage consumption for said restore of said received data.

11. The secure relational file system of claim 10, wherein said first metadata further comprises metadata associated with archived data and archived changes to said data for availability during said backup, metadata of a file system used in creation of said disk storage, metadata stored by a file system used in creation of a disk image of said disk storage, metadata of a virtual machine comprising said archived data, and one or more of attributes, permissions, locations, content specifics, offsets, a size of said disk storage, and directory information of said received data.

12. The secure relational file system of claim 10, wherein said second metadata further comprises a unique identifier for each of said fixed sized data chunks, wherein said unique identifier is a checksum of content in said each of said fixed sized data chunks.

13. The secure relational file system of claim 10, wherein said third metadata further comprises backup data associated with said backup, wherein said backup data comprises data of a file system used for storing said backup data and an index of contents of said variable sized data chunks.

14. The secure relational file system of claim 10, further comprising an integrity check performance module configured to perform an integrity check for rectifying errors in said identified unique variable sized data chunks by adding a parity file to each of said one or more chunk files where said identified unique variable sized data chunks are stored.

15. The secure relational file system of claim 14, wherein said integrity check performance module is further configured to maintain information of said parity file in a backup file stored at one or more of a plurality of data restore levels for facilitating said integrity check at said one or more of said data restore levels, wherein said data restore levels comprise a disk image level, a chunk level, a repository level, a client level, and a backup level.

16. The secure relational file system of claim 10, further comprising:

said data reader further configured to receive said stored third metadata comprising said location of said storage of said each of said identified unique variable sized data chunks in said one or more chunk files and a pointer for each of said duplicate variable sized data chunks from said one or more databases, wherein said pointer points to a unique instance of said each of said duplicate variable sized data chunks;

said data reader further configured to retrieve said identified unique variable sized data chunks from said one or more chunk files using said location of said storage of said each of said identified unique variable sized data chunks in said one or more chunk files from said received third metadata;

said data reader further configured to retrieve, from said one or more chunk files, said unique instance of said each of said duplicate variable sized data chunks, to which said pointer from said received third metadata points, for retrieving said each of said duplicate variable sized data chunks;

said data reader configured to receive said stored second metadata comprising said logical boundaries used for said splitting of said received data into said fixed sized data chunks and said stored first metadata comprising said file-to-sector mapping information from said one or more databases; and

a data assembler configured to assemble said retrieved unique variable sized data chunks and said retrieved unique instance of said each of said duplicate variable sized data chunks in a predefined order for regenerating and restoring said each of said fixed sized data chunks using said received second metadata at a target location identified using said received first metadata.

17. The secure relational file system of claim 10, further comprising a version control module configured to perform an incremental backup for maintaining data changed after each said backup of said received data by identifying changed unique variable sized data chunks and additional unique variable sized data chunks based on said logical boundaries obtained from said stored second metadata, wherein said data writer, in communication with said version control module, is further configured to store said changed unique variable sized data chunks and said additional unique variable sized data chunks in said one or more chunk files.

18. The secure relational file system of claim 17, wherein said data writer is further configured to store metadata of each of said changed unique variable sized data chunks and each of said additional unique variable sized data chunks in said one or more databases for allowing each said incremental backup to be used instantly as a full backup, without merging with previous increments.

19. A computer program product comprising a non-transitory computer readable storage medium, said non-transitory computer readable storage medium storing computer program codes that comprise instructions executable by at least one processor, said computer program codes comprising:

a first computer program code for receiving data for backup and restore;

a second computer program code for generating first metadata associated with said received data for said backup, said first metadata comprising file-to-sector mapping information associated with disk storage of said received data in a plurality of sectors in a storage device, said file-to-sector mapping information configured to identify a position of said received data in said storage device and one or more of unused sectors among said sectors and unused data blocks in said storage device for reducing size of storage required for said backup of said received data;

a third computer program code for splitting said received data into a plurality of fixed sized data chunks using said generated first metadata;

a fourth computer program code for generating second metadata for each of said fixed sized data chunks for said restore of said received data, said second metadata comprising logical boundaries used for said splitting of said received data;

a fifth computer program code for creating a plurality of fixed sized data blocks by prepending said generated second metadata to said fixed sized data chunks;

a sixth computer program code for splitting said fixed sized data chunks of each of said created fixed sized data blocks into a plurality of variable sized data chunks using a plurality of context aware breakpoints in said fixed sized data chunks;

a seventh computer program code for generating a unique identifier for each of said variable sized data chunks;

an eighth computer program code for generating third metadata for each of said variable sized data chunks, said third metadata comprising said generated unique identifier;

a ninth computer program code for creating a plurality of variable sized data blocks by prepending said generated third metadata of said each of said variable sized data chunks and said generated second metadata from said each of said created fixed sized data blocks to said each of said variable sized data chunks;

a tenth computer program code for identifying unique variable sized data chunks of said created variable sized data blocks using each said generated unique identifier from said generated third metadata, in communication with one or more databases for deduplication of said variable sized data chunks;

an eleventh computer program code for storing said identified unique variable sized data chunks in one or more chunk files for said backup and said restore of said received data;

a twelfth computer program code for adding a location of storage of each of said identified unique variable sized data chunks in said one or more chunk files to said generated third metadata; and

a thirteenth computer program code for storing said generated first metadata, said generated second metadata extracted from each of said created variable sized data blocks, and said generated third metadata of said identified unique variable sized data chunks and duplicate variable sized data chunks extracted from said each of said created variable sized data blocks in said one or more databases with minimum additional storage consumption for said restore of said received data.

20. The computer program product of claim 19, further comprising a fourteenth computer program code for performing an integrity check configured to rectify errors in said identified unique variable sized data chunks by adding a parity file to each of said one or more chunk files where said identified unique variable sized data chunks are stored.

21. The computer program product of claim 20, further comprising a fifteenth computer program code for maintaining information of said parity file in a backup file stored at one or more of a plurality of data restore levels for facilitating said integrity check at said one or more of said data restore levels, wherein said data restore levels comprise a disk image level, a chunk level, a repository level, a client level, and a backup level.

22. The computer program product of claim 19, further comprising:

a sixteenth computer program code for receiving said stored third metadata comprising said location of said storage of said each of said identified unique variable sized data chunks in said one or more chunk files and a pointer for each of said duplicate variable sized data chunks from said one or more databases, wherein said pointer points to a unique instance of said each of said duplicate variable sized data chunks;

a seventeenth computer program code for retrieving said identified unique variable sized data chunks from said one or more chunk files using said location of said storage of said each of said identified unique variable sized data chunks in said one or more chunk files from said received third metadata;

an eighteenth computer program code for retrieving, from said one or more chunk files, said unique instance of said each of said duplicate variable sized data chunks, to which said pointer from said received third metadata points for retrieving said each of said duplicate variable sized data chunks;

a nineteenth computer program code for receiving said stored second metadata comprising said logical boundaries used for said splitting of said received data into said fixed sized data chunks and said stored first metadata comprising said file-to-sector mapping information from said one or more databases; and

a twentieth computer program code for assembling said retrieved unique variable sized data chunks and said retrieved unique instance of said each of said duplicate variable sized data chunks in a predefined order for regenerating and restoring said each of said fixed sized data chunks using said received second metadata at a target location identified using said received first metadata.

23. The computer program product of claim 19, further comprising a twenty first computer program code for performing an incremental backup for maintaining data changed after each said backup of said received data, wherein said incremental backup comprises identifying and storing changed unique variable sized data chunks and additional unique variable sized data chunks in said one or more chunk files based on said logical boundaries obtained from said stored second metadata.

24. The computer program product of claim 23, wherein said twenty first computer program code comprises a twenty second computer program code for storing metadata of each of said changed unique variable sized data chunks and each of said additional unique variable sized data chunks in said one or more databases for allowing each said incremental backup to be used instantly as a full backup, without merging with previous increments.