Replication of data objects from a source server to a target server

• US 9,910,904 B2
• Filed: 08/30/2011
• Issued: 03/06/2018
• Est. Priority Date: 08/30/2011
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Anglin, Matthew J., Cannon, David M., Dawson, Colin S., Fruchtman, Barry, Haye, Mark A., Martin, Howard N.
• Primary Examiner(s): Mackes, Kris
• Assistant Examiner(s): NGUYEN, MERILYN P
• Application Number:
  US13/221,670
• Publication Number:
  US 20130054523A1
• Time in Patent Office: 2,380 Days
• Field of Search: 707624
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation

Abstract

Data objects are replicated from a source storage managed by a source server to a target storage managed by a target server. A source list is built of objects at the source server to replicate to the target server. The target server is queried to obtain a target list of objects at the target server. A replication list is built indicating objects on the source list not included on the target list to transfer to the target server. For each object in the replication list, data for the object not already at the target storage is sent to the target server and metadata on the object is sent to the target server to cause the target server to include the metadata in an entry for the object in a target server replication database. An entry for the object is added to a source server replication database.

Claims

1. A computer program product for replicating objects from a source storage managed by a source server to a target storage managed by a target server, the computer program product comprising a non-transitory computer readable storage medium having computer readable program code embodied therein that executes to cause the source server to perform operations, the operations comprising:

building a source list of objects at the source server to replicate to the target server as part of a replication request, wherein each of the objects is comprised of chunks;

querying the target server to obtain a target list of objects at the target server satisfying a replication criteria of the replication request, wherein the replication criteria allows for selection and filtering of objects to replicate at an object level;

building a replication list indicating objects on the source list not included on the target list to transfer to the target server;

for each object in the replication list, performing;

determining chunks in the object;

determining a first set of the chunks in the object that are not stored at the target storage;

sending the first set of the chunks in the object not already at the target storage to the target server;

sending metadata on the object, in the replication list, to the target server to cause the target server to include the metadata in an entry for the object, in the replication list, in a target server replication database;

adding an entry having information for the object being replicated to a source server replication database;

determining a second set of chunks in the object that are currently stored at the target storage; and

sending chunk identifiers of chunks in the second set of chunks to the target server; and

sending metadata to the target server for objects at the source server matching objects at the target server in the target list and that are not in the replication list.

2. The computer program product of claim 1, wherein the operations further comprise:

for each object in the replication list, including in the entry for the object in the source server replication database an identifier of the object at the source server and an identifier of the object at the target server.

3. The computer program product of claim 1, wherein the operations further comprise:

for each object on both the source list and the target list, performing;

sending metadata for the object to the target server to cause the target server to include the metadata in an entry for the object in the target server replication database;

and

adding an entry having information for the object being replicated to the source server replication database.

4. The computer program product of claim 1, wherein the replication criteria comprises a second criteria, wherein the operations further comprise:

querying the target server for an inventory of objects at the target server that satisfy a first criteria;

receiving from the target server the inventory;

determining objects in the source server matching those listed in the inventory from the target server;

wherein the sent metadata is for the determined objects matching those listed in the inventory to the target server to cause the target server to include the metadata in an entry for the object in the target server replication database; and

adding an entry having information for the object being replicated to the source server replication database, wherein the source list includes objects to replicate at the source server that satisfy the second criteria.

5. The computer program product of claim 4, wherein the first criteria comprises a client node originating the objects to replicate and wherein the second criteria comprises at least one of data type and filespace of objects at the client node.

6. The computer program product of claim 1, wherein the operations further comprise:

determining objects to delete that are on the target list but not on the source list; and

communicating with the target server to cause the target server to delete the determined objects to delete from the target server and target storage.

7. The computer program product of claim 1, wherein the source and target servers provide for at least one of:

separate and independently operating hardware and operating systems;

having the target server provide a hot standby mode for the source server; and

heterogeneous hardware and operating systems.

8. A computer program product for replicating objects from a source storage managed by a source server to a target storage managed by a target server, the computer program product comprising a non-transitory computer readable storage medium having computer readable program code embodied therein that executes to cause the target server to perform operations, the operations comprising:

receiving a query from the source server for a target list of objects at the target server satisfying a replication criteria of a replication request, wherein the replication criteria allows for selection and filtering of objects to replicate at an object level, wherein each of the objects is comprised of chunks;

sending the source server the target list of the objects at the target server satisfying the replication criteria;

receiving, from the source server, only those chunks for at least one object from the source server that are not already stored in the target storage to store in the target storage;

receiving, from the source server, metadata for the objects to replicate not included in the target list;

receiving, from the source server, a list of chunks in at least one object currently stored in the target storage;

for each object for which less than all chunks in the object were received from the target server, indicating, in an entry for the object in a target server replication database, links to the chunks in the list of chunks already stored in the target storage;

adding an entry to the target server replication database for each object for which data is received including the metadata received for the object; and

receiving, from the source server, metadata for objects at the source server matching objects at the target server in the target list.

9. The computer program product of claim 8, wherein the operations further comprise:

for the entry for each object in the target server replication database, including an identifier of the object at the source server and an identifier of the object at the target server.

10. The computer program product of claim 8, wherein the operations further comprise:

for each object to replicate already stored in the target storage including the received metadata for the object, adding an entry having information for the object to the target server replication database.

11. The computer program product of claim 8, wherein the operations further comprise:

receiving from the source server a query for an inventory of objects at the target server that satisfy a first criteria;

sending to the source server the inventory;

wherein the received metadata is for objects at the source server matching those listed in the inventory; and

adding an entry to the target server replication database for each object for which metadata is received for the objects matching those listed in the inventory, wherein the entries include the metadata for the object, wherein the target list includes objects at the target server that satisfy a second criteria.

12. The computer program product of claim 8, wherein multiple source servers are replicated to the target server.

13. The computer program product of claim 8, wherein the operations further comprise:

maintaining a group of objects stored in the target storage; and

indicating the group of objects as incomplete in response to determining that not all the objects in the group have been received at the target server, wherein the group of objects indicated as incomplete are not made available to a client node during a restore operation.

14. The computer program product of claim 13, wherein the operations further comprise:

receiving objects for the group of objects indicated as incomplete; and

indicating the group of objects as complete in response to having received all the objects in the group, wherein objects in the group indicated as complete are available to the client node during a restore operation.

16. The system of claim 14, wherein the operations further comprise:

for each object on both the source list and the target list, performing;

sending metadata for the object to the target server to cause the target

server to include the metadata in an entry for the object in the target server

replication database; and

adding an entry having information for the object being replicated to the source server replication database.

15. A system for replicating objects from a source storage to a target storage managed by a target server, comprising:

a processor; and

a computer readable storage medium having code executed by the processor to perform operations, the operations comprising;

building a source list of objects at the source storage to replicate to the target server as part of a replication request, wherein each of the objects is comprised of chunks;

querying the target server to obtain a target list of objects at the target server satisfying a replication criteria of the replication request, wherein the replication criteria allows for selection and filtering of objects to replicate at an object level;

building a replication list indicating objects on the source list not included on the target list to transfer to the target server;

for each object in the replication list, performing;

determining chunks in the object;

determining a first set of the chunks in the object that are not stored at the target storage;

sending the first set of the chunks in for the object not already at the target storage to the target server;

sending metadata on the object, in the replication list, to the target server to cause the target server to include the metadata in an entry for the object, in the replication list, in a target server replication database;

adding an entry having information for the object being replicated to a source server replication database;

determining a second set of chunks in the object that are currently stored at the target storage; and

sending chunk identifiers of chunks in the second set of chunks to the target server; and

sending metadata to the target server for objects at the source storage matching objects at the target server in the target list and that are not in the replication list.

17. The system of claim 15, wherein the replication criteria comprises a second criteria, wherein the operations further comprise:

querying the target server for an inventory of objects at the target server that satisfy a first criteria;

receiving from the target server the inventory;

determining objects in the source storage matching those listed in the inventory from the target server;

wherein the sent metadata is for the determined objects matching those listed in the inventory to the target server to cause the target server to include the metadata in an entry for the object in the target server replication database; and

adding an entry having information for the object being replicated to the source server replication database, wherein the source list includes objects to replicate at the source storage that satisfy the second criteria.

18. The system of claim 17, wherein the first criteria comprises a client node originating the objects to replicate and wherein the second criteria comprises at least one of data type and filespace of objects at the client node.

19. The system of claim 15, wherein the operations further comprise:

determining objects to delete that are on the target list but not on the source list; and

communicating with the target server to cause the target server to delete the determined objects to delete from the target server and target storage.

20. The system of claim 15, wherein the operations further comprise:

for each object in the replication list, including in the entry for the object in the source server replication database an identifier of the object at the source storage and an identifier of the object at the target server.

21. The system of claim 15, wherein the system and target server provide for at least one of:

separate and independently operating hardware and operating systems;

having the target server provide a hot standby mode for the system; and

heterogeneous hardware and operating systems.

22. A system for replicating objects from a source storage managed by a source server to a target storage, comprising:

a processor; and

a computer readable storage medium having code executed by the processor to perform operations, the operations comprising;

receiving a query from the source server for a target list of objects at the target storage satisfying a replication criteria of a replication request, wherein the replication criteria allows for selection and filtering of objects to replicate at an object level, wherein each of the objects is comprised of chunks;

sending the source server the target list of the objects at the target storage satisfying the replication criteria;

receiving, from the source server, only those chunks for at least one object from the source server that are not already stored in the target storage to store in the target storage;

receiving, from the source server, metadata for the objects to replicate not included in the target list;

receiving, from the source server, a list of chunks in at least one object currently stored in the target storage;

for each object for which less than all chunks in the object were received from the target storage, indicating, in an entry for the object in a target server replication database, links to the chunks in the list of chunks already stored in the target storage;

adding an entry to the target server replication database for each object for which data is received including the metadata received for the object; and

receiving, from the source server, metadata for objects at the source server matching objects at the target storage in the target list.

23. The system of claim 22, wherein the operations further comprise:

for each object to replicate already stored in the target storage including the received metadata for the object, adding an entry having information for the object to the target server replication database.

24. The system of claim 22, wherein the operations further comprise:

receiving from the source server a query for an inventory of objects at the target storage that satisfy a first criteria;

sending to the source server the inventory wherein the received metadata is for objects at the source server matching those listed in the inventory; and

adding an entry to the target server replication database for each object for which metadata is received for the objects matching those listed in the inventory, wherein the entries include the metadata for the object, wherein the target list includes objects at the target storage that satisfy a second criteria.

25. The system of claim 22, wherein the operations further comprise:

maintaining a group of objects stored in the target storage; and

indicating the group of objects as incomplete in response to determining that not all the objects in the group have been received at the target storage, wherein the group of objects indicated as incomplete are not made available to a client node during a restore operation.

26. The system of claim 25, further comprising:

receiving objects for the group of objects indicated as incomplete; and

indicating the group of objects as complete in response to having received all the objects in the group, wherein objects in the group indicated as complete are available to the client node during a restore operation.

27. The system of claim 22, wherein the operations further comprise:

for the entry for each object in the target server replication database, including an identifier of the object at the source server and an identifier of the object at the target storage.

28. The system of claim 22, wherein multiple source servers are replicated to the target storage.