System of and method for transparent management of data objects in containers across distributed heterogenous resources

• US 7,269,604 B2
• Filed: 10/24/2003
• Issued: 09/11/2007
• Est. Priority Date: 04/27/2000
• Status: Expired due to Fees

• Current Assignee: General Atomics, Inc.
• Original Assignee: General Atomics, Inc.
• Inventors: Rajasekar, Arcot, Moore, Reagan W., Wan, Michael Y.
• Primary Examiner(s): Alam; Shahid Al
• Application Number:
  US10/693,599
• Publication Number:
  US 20040133609A1
• Time in Patent Office: 1,418 Days
• Field of Search: 707/2, 707/8, 707/10, 707/103.R, 707/200, 707/104.1, 707/201, 707/202, 707/204, 709201-203, 709/213, 709/216, 709/220, 709/224, 709/227, 709/239, 710/100, 714/1, 714/6, 718/102
• US Class Current: 1/1
• CPC Class Codes: G06F 16/188 Virtual file systems
• CPC Class Codes: G06F 16/256 in federated or virtual databases
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: Y10S 707/99944 Object-oriented database structure

Abstract

A system of and method for maintaining data objects in containers across a network of distributed heterogeneous resources in a manner which is transparent to a client. A client request pertaining to containers is resolved by querying meta data for the container, processing the request through one or more copies of the container maintained on the system, updating the meta data for the container to reflect any changes made to the container as a result processing the re quest, and, if a copy of the container has changed, changing the status of the copy to indicate dirty status or synchronizing the copy to one or more other copies that may be present on the system.

Claims

1. A method for transparently managing containers in a network of distributed physical resources, a container comprising a plurality of data objects and an association of the plurality of data objects with a plurality of physical resources in the network assigned to the container at which the container may be stored or copied, comprising:

issuing a request to write one or more data objects in or to a container in response to a user command;

accessing a meta-data catalog in response to the request, the meta-data catalog associating the data objects in the container with the plurality of physical resources assigned to the container, including querying meta-data about the container;

locating on or staging to a selected physical resource a copy of the container, including copying a container from one physical resource to another assigned to the container transparently to the user should a staging condition arise;

writing the data object to the copy of the container;

synchronizing the copy with any other copies that may be present in the system, or making the changed copy for synchronization, transparently to the user should an out-of-synchronization condition arise; and

updating the meta-data catalog to reflect changes incidental to the request, including updating the meta-data for the container to reflect the write operation.

2. The method of claim 1 wherein the meta-data for a container associates at least one cache resource and at least one archival resource with the container.

3. The method of claim 2 further comprising

generating, responsive to a request to create a container, meta-data for the container;

associating one or more physical resources with the container;

storing the meta-data in a meta-data catalog; and

allocating one or more of the physical resources to the container.

4. The method of claim 2 wherein:

the step of querying meta-data about the container includes obtaining a next available offset for storing in or to the container; and

the step of writing one or more data objects to the copy of the container comprises writing the one or more data objects beginning at the next available offset.

5. The method of claim 2 further comprising

querying, responsive to a request to synchronize copies of a container, meta-data for the container to locate a copy of the container mark by the synchronization;

overwriting one or more of the other copies of the container with the dirty copy; and

resetting the dirty status of one or more of the copies of the container.

6. The method of claim 2 further comprising

querying, responsive to a request to read a data object or part of an object as specified by a method or parameter, from a container, mete-data for the container, including an offset for the object within the container;

locating on or staging to a selected physical resource a copy of the container; and

reading the required data from the copy at the specified offset.

7. The method of either of claims 1 or 6 wherein the selected physical resource comprises a primary cache resource.

8. The method of claim 1 wherein the plurality of physical resources is selected front the group comprising a file system, an archival storage system, a cache system, and a database system.

9. The method of claim 1 wherein the plurality of physical resources is selected from the group comprising object-oriented, relational, structured, and semi-structured systems.

10. The method of claim 1 further comprising receiving a request front a client and responsive thereto, spawning a broker process, executable on a server, for responding to the request.

11. The method of claim 1 further comprising the meta-data associating at least one archival resource with the container.

12. The method of claim 1 further comprising staging a copy of the container to a cache assigned to the container in response to a container read or write request upon determining that the data objects in the container are not stored in the cache.

13. The method of claim 1 further comprising synchronizing multiple copies of the container in response to a container write operation performed on a copy of the container stored in a cache assigned to the container.

14. The method of claim 1 further comprising marking for synchronization at least one copy of the container as stored ma cache assigned to the container in response to a container write operation performed on a copy of the container stored in the cache.

15. The method of claim 1 further comprising purging at least one copy of the container in response to a purge condition.

16. The method of claim 15 further comprising purging any cache copies of the container after synchronizing multiple copies of the container including at least one archival copy.

17. The method of claim 1 wherein the data objects in the container are unstructured.