Relational objects for the optimized management of fixed-content storage systems

• US 8,171,065 B2
• Filed: 01/26/2011
• Issued: 05/01/2012
• Est. Priority Date: 02/22/2008
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: Bycast Inc. (NetApp, Inc.)
• Inventors: Slik, David, Seiler, Oliver, Montour, Mike
• Primary Examiner(s): Mahmoudi, Tony
• Assistant Examiner(s): Khoshnoodi, Fariborz
• Application Number:
  US13/014,659
• Publication Number:
  US 20110125814A1
• Time in Patent Office: 461 Days
• Field of Search: 707/822, 707/823, 707/511, 707/530, 707/100, 707/101
• US Class Current: 707/822
• CPC Class Codes: G06F 16/11 File system administration, e.g. details of archiving or snapshots file system backup G06F11/14
• CPC Class Codes: G06F 16/164 File meta data generation

Abstract

A system and method is described for managing data objects in a fixed-content storage system. Metadata is provided for each variable size packet and may include offset information, packet size data, reference content blocks, and the like. Using this information, intelligently decomposed objects, consolidated objects, differenced objects, and composite objects may be stored in the storage system. The data structure provided by these objects allows for the reduction of necessary storage resources and the total number of stored objects.

Claims

1. A method of storing multiple immutable objects on a computer readable medium, comprising:

receiving a first immutable data object;

receiving a second immutable data object;

generating first metadata for the first immutable data object and second metadata for the second immutable data object;

constructing, using one or more computer processors, a consolidated data object comprising the first immutable data object and the second immutable data object, the consolidated data object further comprising consolidated object metadata;

configuring the consolidated object metadata to indicate the relative locations of the first immutable data object and the second immutable data object within the consolidated data object, the relative locations indicating one or more storage positions of the first immutable data object and the second immutable data object; and

storing in a fixed content storage system the consolidated object as a single data object, whereby the first immutable data object and the immutable second data object need not be stored separately in the fixed content storage system.

2. The method of claim 1, wherein constructing the consolidated object comprises concatenating the first metadata, the first immutable data object, the second metadata, and the second immutable data object in logically adjacent locations on the computer readable medium.

3. The method of claim 1, wherein the consolidated object metadata comprises an offset and a size for each of the first and second immutable data objects.

4. The method of claim 1, further comprising:

receiving a third immutable data object;

generating third metadata for the third immutable data object;

constructing, using the one or more computer processors, a second consolidated data object comprising the first, the second, and the third data objects, wherein the consolidated data object further comprises second consolidated object metadata;

configuring the second consolidated object metadata to indicate the relative locations of the first, the second, and the third immutable data objects within the consolidated data object, the relative locations indicating one or more storage positions of the first, the second, and the third immutable data objects; and

storing in the fixed content storage system the second consolidated object.

5. The method of claim 1, wherein the consolidated data object further comprises the first metadata and the second metadata.

6. The method of claim 1, wherein the consolidated object metadata further comprises rules relating to data retention and replication, the fixed content storage system being configured to retain and replicate the first immutable data object and the second data object in accordance with the rules, without requiring configuration of rules for the first immutable data object or the second immutable data object.

7. A computer system, comprising:

a fixed content storage system configured to store immutable data objects; and

one or more computer processors configured to perform operations comprising;

receiving a first data object;

receiving a second data object;

generating first metadata for the first data object and second metadata for the second data object;

constructing a consolidated data object comprising the first data object and the second data object, the consolidated data object further comprising consolidated object metadata;

configuring the consolidated object metadata to indicate the relative locations of the first data object and the second data object within the consolidated data object, the relative locations indicating one or more storage positions of the first data object and the second data object; and

storing, in the fixed content storage system, the consolidated object as a single data object, whereby the first data object and the second data object need not be stored separately in the fixed content storage system.

8. The system of claim 7, wherein constructing the consolidated object comprises concatenating the first metadata, the first data object, the second metadata, and the second data object in logically adjacent locations on the fixed content storage system.

9. The system of claim 7, wherein the consolidated object metadata comprises an offset and a size for each of the first and second data objects.

10. The system of claim 7, wherein the one or more computer processors are further configured to perform operations comprising:

receiving a third data object;

generating third metadata for the third data object;

constructing, using the one or more computer processors, a second consolidated data object comprising the first data object, the second data object, and the third data object, wherein the consolidated data object further comprises consolidated object metadata;

configuring the consolidated object metadata to indicate the relative locations of the first data object, the second data object, and the third data object within the consolidated data object, the relative locations indicating one or more storage positions of the first data object, the second data object, and the third data object; and

storing in the fixed content storage system the second consolidated object.

11. The system of claim 7, wherein the consolidated data object further comprises the first metadata and the second metadata.

12. The system of claim 7, wherein the consolidated object metadata further comprises rules relating to data retention and replication, the fixed content storage system being configured to retain and replicate the first data object and the second data object in accordance with the rules, without requiring configuration of rules for the first data object or the second data object.

13. The system of claim 7, wherein the system comprises a plurality of distributed nodes, each distributed node comprising at least one processor and at least one storage device, and wherein storing the consolidated data object comprises storing the consolidated data object on more than one distributed node, wherein the plurality of distributed nodes spans multiple geographically separated sites, and wherein at least a portion of the plurality of distributed nodes are configured to communicate on a network.