Metadata subsystem for a distributed object store in a network storage system

• US 9,043,372 B2
• Filed: 06/12/2013
• Issued: 05/26/2015
• Est. Priority Date: 12/08/2009
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Makkar, Gaurav, Srinivasan, Sudhir, Kavuri, Ravi
• Primary Examiner(s): HASAN, SYED HAROON
• Application Number:
  US13/916,389
• Publication Number:
  US 20130346444A1
• Time in Patent Office: 713 Days
• Field of Search: None
• US Class Current: 707/827
• CPC Class Codes: G06F 16/90335 Query processing
• CPC Class Codes: G06F 16/907 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually

Abstract

A network storage server system includes a distributed object store and a metadata subsystem. The metadata subsystem stores metadata relating to the stored data objects and allows data objects to be located and retrieved easily via user-specified search queries. It manages and allows searches on at least three categories of metadata via the same user interface and technique. These categories include user-specified metadata, inferred metadata and system-defined metadata. Search queries for the metadata can include multi-predicate queries.

Claims

1. A method, comprising:

maintaining a distributed object store to store a plurality of data objects in a network storage system, wherein a first data object of the plurality of data objects is accessed using a first global object identification (ID) that uniquely identifies the first data object in the distributed object store and a first location ID associated with the first global object ID that identifies a storage location of the first data object in the distributed object store; and

operating a metadata subsystem in the network storage system to store and retrieve metadata of a plurality of types, relating to the plurality of data objects, the plurality of types of metadata including system defined metadata, inferred metadata and user-defined metadata, and further operating the metadata subsystem to search the metadata of the plurality of types to identify a set of the data objects that satisfy a user-specified search query, the operating the metadata subsystem to identify the set of the data objects including;

retrieving a first metadata from the search query,

retrieving a metadata subsystem-specific object ID of a target data object of the set of the data objects from a first mapping based on a value of the first metadata, the first mapping containing a mapping of a plurality of metadata subsystem-specific object IDs and values of metadata of the set of the data objects,

retrieving, based on the metadata subsystem-specific object ID and from a second mapping, a second global object ID of the target data object, the second mapping containing a mapping of the plurality of metadata subsystem-specific object IDs and a plurality of global object IDs of the set of the data objects, and

obtaining the target data object from the distributed object store using the second global object ID and an associated second location ID.

2. A method as recited in claim 1, wherein the inferred metadata comprises latent metadata and discovered metadata.

3. A method as recited in claim 1, wherein operating the metadata subsystem comprises implementing a plurality of mutually isolated query domains, wherein said metadata includes a plurality of metadata attribute-value pairs, and wherein each of the metadata attribute-value pairs is assigned to one of the plurality of query domains.

4. A method as recited in claim 3, wherein operating the metadata subsystem further comprises operating a plurality of instances of the metadata subsystem, each instance residing on a different one of a plurality of storage nodes, to collectively implement a particular query domain of the plurality of query domains.

5. A method as recited in claim 1, wherein operating the metadata subsystem comprises allowing at least some of the plurality of data objects to have different numbers of attributes than others of the plurality of data objects.

6. A method as recited in claim 1, wherein operating the metadata subsystem comprises allowing fixed-schema storage and allowing flexible-schema storage to be used to store metadata associated with the plurality of data objects.

7. A method as recited in claim 6, wherein operating the metadata subsystem comprises using both fixed-schema storage and flexible-schema storage concurrently to store metadata associated with the plurality of data objects.

8. A method as recited in claim 1, wherein operating the metadata subsystem comprises storing all of the metadata for each particular data object of the plurality of data objects in the same network storage node.

9. A method as recited in claim 8, wherein operating the metadata subsystem comprises storing all of the metadata for any particular data object in the network storage node in which the particular data object is stored.

10. A method as recited in claim 8, wherein the metadata for a particular data object comprises a plurality of attribute values, and wherein operating the metadata subsystem comprises placing each of the attribute values of the particular data object in a separate one of a plurality of indexes.

11. A method as recited in claim 1, wherein operating the metadata subsystem comprises maintaining a plurality of indexes, each corresponding to a different metadata attribute.

12. A method as recited in claim 1, wherein operating the metadata subsystem comprises receiving and servicing a multi-predicate user metadata query.

13. A method as recited in claim 1, wherein operating the metadata subsystem comprises:

using a log structure to store in a chronological order a plurality of metadata operations relating to different data objects; and

updating a plurality of metadata structures based on contents of the log structure, wherein said updating is not directly triggered by any of said plurality of metadata operations.

14. A network storage server node comprising:

a network module using which the node can communicate with a network storage client;

a data module using which the node can manage a persistent storage subsystem; and

a management host configured to provide management services of the network storage server node and to interface with an external administrative user, the management host including a metadata subsystem to store and retrieve metadata of a plurality of types, relating to a plurality of data objects stored in a distributed object store implemented at least partially in the persistent storage subsystem, the plurality of types of metadata including system defined metadata, inferred metadata and user-defined metadata, the metadata subsystem further being capable of searching the metadata of the plurality of types to identify a set of the data objects that satisfy a user-specified search query, the searching to identify the set of the data objects including;

retrieving a first metadata from the search query,

retrieving a metadata subsystem-specific object ID of a target data object of the set of the data objects from a first mapping based on a value of the first metadata, the first mapping containing a mapping of a plurality of metadata subsystem-specific object IDs and values of metadata of the set of the data objects,

retrieving, based on the metadata subsystem-specific object ID and from a second mapping, a global object ID of the target data object, the second mapping containing a mapping of the plurality of metadata subsystem-specific object IDs and a plurality of global object IDs of the set of the data objects, and

obtaining the target data object from the distributed object store using the global object ID and an associated location ID.

15. A network storage server node as recited in claim 14, wherein the network storage server node is one of a plurality of network storage server nodes in a clustered network storage system.

16. A network storage server node as recited in claim 14, wherein the inferred metadata comprises latent metadata and discovered metadata.

17. A network storage server node as recited in claim 14, wherein the metadata subsystem is configured to implement a plurality of mutually isolated query domains, wherein said metadata includes a plurality of metadata attribute-value pairs, and wherein each of the metadata attribute-value pairs is assigned to one of the plurality of query domains.

18. A network storage server node as recited in claim 17, wherein the metadata subsystem is one of a plurality of similar metadata subsystem instances, each instance residing on a different one of a plurality of storage nodes, to collectively implement a particular query domain of the plurality of query domains.

19. A network storage server node as recited in claim 14, wherein the metadata subsystem is configured to allow at least some of the plurality of data objects to have different numbers of attributes than others of the plurality of data objects.

20. A network storage server node as recited in claim 14, wherein the metadata subsystem is configured to allow both fixed-schema storage and flexible-schema storage to be used to store metadata associated with the plurality of data objects.

21. A network storage server node as recited in claim 20, wherein the metadata subsystem is configured to use both fixed-schema storage and flexible-schema storage concurrently to store metadata associated with the plurality of data objects.

22. A network storage server node as recited in claim 14, wherein the metadata subsystem is configured to store all of the metadata for each particular data object of the plurality of data objects in the same network storage node.

23. A network storage server node as recited in claim 22, wherein the metadata subsystem is configured to store all of the metadata for any particular data object in a network storage node in which the particular data object is stored.

24. A network storage server node as recited in claim 22, wherein the metadata for a particular data object comprises a plurality of attribute values, and wherein the metadata subsystem is configured to place each of the attribute values of the particular data object in a separate one of a plurality of indexes.

25. A network storage server node as recited in claim 14, wherein the metadata subsystem is configured to maintain a plurality of indexes, each corresponding to a different metadata attribute.

26. A network storage server node as recited in claim 14, wherein the metadata subsystem is configured to receive and service a multi-predicate user metadata query.

27. A network storage server node as recited in claim 14, wherein the metadata subsystem is configured to:

use a log structure to store in a chronological order a plurality of metadata operations relating to different data objects; and

update a plurality of metadata structures based on contents of the log structure, wherein said updating is not directly triggered by any of said plurality of metadata operations.

28. A network storage server node comprising:

a network adapter through which to receive a data access request from a storage client via a network;

a storage adapter through which to access a persistent storage subsystem;

a memory; and

a processor coupled to the memory and implementing a metadata subsystem, to store and retrieve metadata of a plurality of types, relating to a plurality of data objects stored in a distributed object store implemented at least partially in said persistent storage subsystem, the plurality of types of metadata including system defined metadata, inferred metadata and user-defined metadata, the metadata subsystem being configured to search metadata of the plurality of types to identify a set of the data objects that satisfy a user-specified search query, the metadata subsystem configured to perform the search to identify the set of the data objects including;

retrieving a first metadata from the search query,

retrieving a metadata subsystem-specific object ID of a target data object of the set of the data objects from a first mapping based on a value of the first metadata, the first mapping containing a mapping of a plurality of metadata subsystem-specific object IDs and values of metadata of the set of the data objects,

retrieving, based on the metadata subsystem-specific object ID and from a second mapping, a global object ID of the target data object, the second mapping containing a mapping of the plurality of metadata subsystem-specific object IDs and a plurality of global object IDs of the set of the data objects, and

obtaining the target data object from the distributed object store using the global object ID and an associated location ID.

29. A network storage server node as recited in claim 28, wherein the inferred metadata comprises latent metadata and discovered metadata.

30. A network storage server node as recited in claim 28, wherein the metadata subsystem is configured to implement a plurality of mutually isolated query domains, wherein said metadata includes a plurality of metadata attribute-value pairs, and wherein each of the metadata attribute-value pairs is assigned to one of the plurality of query domains.

31. A network storage server node as recited in claim 28, wherein the metadata subsystem is configured to use both fixed-schema storage and flexible-schema storage concurrently to store metadata associated with the plurality of data objects.

32. A network storage system comprising:

a distributed object store to store a plurality of data objects, the distributed object store implemented in a plurality of storage nodes of a network storage system;

means for storing and retrieving metadata of a plurality of types, relating to the data objects, the plurality of types of metadata including system defined metadata, inferred metadata and user-defined metadata; and

means for identifying a set of the data objects that satisfy a user-specified search query by searching the metadata of the plurality of types, the identifying the set of the data objects including;

retrieving a first metadata from the search query,

retrieving a metadata subsystem-specific object ID of a target data object of the set of the data objects from a first mapping based on a value of the first metadata, the first mapping containing a mapping of a plurality of metadata subsystem-specific object IDs and values of metadata of the set of the data objects,

retrieving, based on the metadata subsystem-specific object ID and from a second mapping, a global object ID of the target data object, the second mapping containing a mapping of the plurality of metadata subsystem-specific object IDs and a plurality of global object IDs of the set of the data objects, and

obtaining the target data object from the distributed object store using the global object ID and an associated location ID.