De-duplication systems and methods for application-specific data
First Claim
1. A system for creating a backup copy of application-generated data objects, the system comprising:
- computer readable memory comprising at least a first de-duplication database, the first de-duplication database associated with first application-specific data, the first application-specific data comprising a first plurality of data objects generated by at least first and second clients;
at least first and second data agents, the first data agent in communication with at least the first client, and the second data agent in communication with at least the second client, the first and second data agents comprising one or more computer processors comprising computer hardware, the first and second data agents monitor the first application-specific data created by the at least first and second clients and insert headers comprising de-duplication indicators in the first application-specific data to identify portions of the first plurality of data objects to be processed for duplicate data, the first application-specific data further comprising metadata associated with one or more of the first plurality of data objects, wherein the metadata further identifies at least the first and second clients,wherein one or more of the de-duplication indicators comprises an offset value identifying a beginning of at least the portions of the first plurality of data objects within the first application-specific data;
the at least one media agent executing on one or more computer processors, the at least one media agent manages a backup of the first application-specific data, the headers comprising the de-duplication indicators, and the metadata received from the first and second data agents;
a de-duplication module that communicates with the media agent, the de-duplication module executing on one or more computer processors comprising computer hardware, the de-duplication module receives the application-specific data from the media agent and performs block-level de-duplication on the first application-specific data as part of the backup of the first application-specific data, the de-duplication module further configured to;
use the inserted headers comprising the de-duplication indicators to identify the beginning of at least one of a first portion of the first plurality of data objects from the first client based on the offset value; and
determine if a duplicate copy of blocks associated with the first portion of the first plurality of data objects from the first client exist in the first de-duplication database; and
if a duplicate copy does not exist in the first de-duplication database, storing the first portion of the blocks in the first de-duplication database and storing first metadata that identifies the first client in associated with the duplicate copy;
determine if a duplicate copy of any of blocks associated with a second portion of the first plurality of data objects from the second client exist in the first de-duplication database;
if a duplicate copy exists in the first de-duplication database, removing the duplicate data in the second portion the first application-specific data associated with the second client;
determining whether second metadata in the second portion of the first application-specific data that identifies the second client is unique; and
if the second metadata is unique, creating a backup copy of the second metadata in association with the duplicate copy in the first de-duplication database, wherein the backup copy stores multiple instances of differing metadata associated with the duplicate copy wherein the first metadata that identifies the first client and the second metadata that identifies the second client are stored in association with the duplicate copy.
4 Assignments
0 Petitions
Accused Products
Abstract
Content-aware systems and methods for improving de-duplication, or single instancing, in storage operations. In certain examples, backup agents on client devices parse application-specific data to identify data objects that are candidates for de-duplication. The backup agents can then insert markers or other indictors in the data that identify the location(s) of the particular data objects. Such markers can, in turn, assist a de-duplication manager to perform object-based de-duplication and increase the likelihood that like blocks within the data are identified and single instanced. In other examples, the agents can further determine if a data object of one file type can or should be single-instanced with a data object of a different file type. Such processing of data on the client side can provide for more efficient storage and back-end processing.
421 Citations
15 Claims
-
1. A system for creating a backup copy of application-generated data objects, the system comprising:
-
computer readable memory comprising at least a first de-duplication database, the first de-duplication database associated with first application-specific data, the first application-specific data comprising a first plurality of data objects generated by at least first and second clients; at least first and second data agents, the first data agent in communication with at least the first client, and the second data agent in communication with at least the second client, the first and second data agents comprising one or more computer processors comprising computer hardware, the first and second data agents monitor the first application-specific data created by the at least first and second clients and insert headers comprising de-duplication indicators in the first application-specific data to identify portions of the first plurality of data objects to be processed for duplicate data, the first application-specific data further comprising metadata associated with one or more of the first plurality of data objects, wherein the metadata further identifies at least the first and second clients, wherein one or more of the de-duplication indicators comprises an offset value identifying a beginning of at least the portions of the first plurality of data objects within the first application-specific data; the at least one media agent executing on one or more computer processors, the at least one media agent manages a backup of the first application-specific data, the headers comprising the de-duplication indicators, and the metadata received from the first and second data agents; a de-duplication module that communicates with the media agent, the de-duplication module executing on one or more computer processors comprising computer hardware, the de-duplication module receives the application-specific data from the media agent and performs block-level de-duplication on the first application-specific data as part of the backup of the first application-specific data, the de-duplication module further configured to; use the inserted headers comprising the de-duplication indicators to identify the beginning of at least one of a first portion of the first plurality of data objects from the first client based on the offset value; and determine if a duplicate copy of blocks associated with the first portion of the first plurality of data objects from the first client exist in the first de-duplication database; and if a duplicate copy does not exist in the first de-duplication database, storing the first portion of the blocks in the first de-duplication database and storing first metadata that identifies the first client in associated with the duplicate copy; determine if a duplicate copy of any of blocks associated with a second portion of the first plurality of data objects from the second client exist in the first de-duplication database; if a duplicate copy exists in the first de-duplication database, removing the duplicate data in the second portion the first application-specific data associated with the second client; determining whether second metadata in the second portion of the first application-specific data that identifies the second client is unique; and if the second metadata is unique, creating a backup copy of the second metadata in association with the duplicate copy in the first de-duplication database, wherein the backup copy stores multiple instances of differing metadata associated with the duplicate copy wherein the first metadata that identifies the first client and the second metadata that identifies the second client are stored in association with the duplicate copy. - View Dependent Claims (2, 3, 4, 5, 6, 7)
-
-
8. A method for creating a backup copy of application-generated data objects, the method comprising:
-
storing a first de-duplication database associated with first application-specific data, the first application-specific data comprising a first plurality of data objects; inserting with at least first and second data agents de-duplication indicators in the first application-specific data generated by first and second clients, wherein the at least first and second data agents monitor the first application-specific data generated by the first and second clients and insert headers comprising one or more de-duplication indicators, wherein the one or more de-duplication indicators comprises an offset value identifying a beginning of at least portions of the first plurality of data objects within the first application-specific data to be processed for duplicate data, the first application-specific data further comprising metadata associated with one or more of the first plurality of data objects, and wherein the metadata further identifies at least the first and second clients; sending to at least one media agent, the first application-specific data, the headers comprising the de-duplication indicators, and the metadata, wherein the media agent manages a backup of the first application-specific data received from the first and second data agents; using the inserted headers comprising de-duplication indicators as part of the backup of the first application-specific data to identify the beginning of at least one of a first portion of the first plurality of data objects from the first client based on the offset value; determining if a duplicate copy of blocks associated with the first portion of the first plurality of data objects from the first client exist in the first de-duplication database; if a duplicate copy does not exist in the first de-duplication database, storing the first portion of the first plurality of data objects in the first de-duplication database; determining if a duplicate copy of blocks associated with a second portion of the first plurality of data objects from the second client exist in the first de-duplication database; if a duplicate copy exists in the first de-duplication database, removing the duplicate data in the second portion the first application-specific data associated with the second client; determining whether the metadata in the second portion of the first application-specific data that identifies the second client is unique; and if the metadata is unique, creating a backup copy of the metadata in association with the duplicate copy in the first de-duplication database, wherein the backup copy stores multiple instances of differing metadata associated with the duplicate copy wherein a first instance of the duplicate copy is stored in association with the first client and a second instance of the duplicate copy is stored in association with the second client. - View Dependent Claims (9, 10, 11, 12, 13, 14, 15)
-
Specification