System and methods for metadata management in content addressable storage

• US 8,788,519 B2
• Filed: 10/23/2009
• Issued: 07/22/2014
• Est. Priority Date: 10/24/2008
• Status: Active Grant

• Current Assignee: Giancarlo Canessa, Gino G. Canessa, John C. Canessa
• Original Assignee: Giancarlo Canessa, Gino G. Canessa, John C. Canessa
• Inventors: Canessa, John C., Canessa, Giancarlo, Canessa, Gino G.
• Primary Examiner(s): Nguyen, Cindy
• Application Number:
  US12/605,036
• Publication Number:
  US 20100138446A1
• Time in Patent Office: 1,733 Days
• Field of Search: 707/770, 707/769, 707/722
• US Class Current: 707/769
• CPC Class Codes: G06F 15/16 Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs coordinating program control therefor G06F9/52; in regulating and control system G05B
• CPC Class Codes: G06F 16/137 Hash-based content-based indexing of textual data G06F16/31
• CPC Class Codes: G06F 16/152 using file content signatures, e.g. hash values
• CPC Class Codes: G06F 16/156 Query results presentation
• CPC Class Codes: G06F 16/1827 Management specifically adapted to NAS management of storage area networks [SAN] G06F3/067
• CPC Class Codes: G06F 16/185 Hierarchical storage management [HSM] systems, e.g. file migration or policies thereof details of archiving G06F16/11
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/90339 by using parallel associative memories or content-addressable memories

Abstract

Provided is a content addressable storage (CAS) system that allows a user to request, either through an application server or directly to one or more CAS servers, files and content related to a query. In some embodiments, the content can be discovered by searching previously-stored metadata related to each file at the content addressable storage server. The search can also be replicated across multiple content addressable storage servers in order to obtain varied results and redundant results. Duplicate results may be flagged or omitted, and the results are returned to the requester.

Claims

1. A computer-implemented method for managing metadata in a content addressable storage system, the method comprising:

receiving, using one or more computer processors, a file for storage at a first content addressable storage (CAS) server, the file comprising a header and data, and wherein the first CAS server stores data that can be retrieved based on content of the data rather than its storage location or with a hierarchical file system;

receiving, using one or more computer processors, the same one or more files for storage at a second CAS server;

automatically obtaining, with the one or more computer processors, from the header of the file, metadata associated with the data;

storing the metadata in a first metadata storage device, wherein the metadata is stored in association with the data stored in the CAS server;

replicating the stored metadata and storing the replicated metadata in a second metadata storage device;

receiving, using the one or more computer processors, a query from a requester for content at the CAS server;

performing a local search within locally-stored content related to the received query;

sending the query to one or more CAS servers;

searching beyond a temporary data cache in a local storage device for local content not stored in the CAS server and related to the received query, wherein the local storage device and the CAS server are distinct;

sending results of the local search to the requestor;

searching the metadata storage device for content related to the received query; and

when the metadata associated with the file is indicated by the query;

retrieving the file stored in the content addressable storage; and

sending the retrieved file to the requester;

wherein sending the results of the local search and the retrieved file to the requester further comprises excluding or flagging any duplicate files.

2. The method of claim 1, wherein the CAS servers store DICOM images and the metadata is related to the DICOM images.

3. The method of claim 1, wherein the method further comprises:

distributing the file and the metadata to a plurality of CAS servers in a hierarchical fashion.

4. The method of claim 3, wherein the method further comprises:

sending the query to the plurality of CAS servers, based on the metadata;

receiving one or more files related to the query from the plurality of CAS servers; and

sending to the requester the retrieved file and the one or more files related to the query from the plurality of CAS servers.

5. The method of claim 1, wherein the metadata storage device comprises a database, XML file, flat file, or other searchable storage mechanism that associates the file stored in the CAS server with corresponding metadata to allow more efficient searching of the contents of the CAS server.

6. The method of claim 5, wherein the CAS server comprises the metadata storage device.

7. The method of claim 1, wherein the CAS server hashes the file and stores the data in the file according to the resulting hash rather than based on a physical storage location.

8. The method of claim 7, wherein the CAS system comprises fixed content storage (FCS).

9. A computer-implemented method for managing metadata in a hashed storage system, comprising:

receiving, using one or more computer processors, one or more files for storage at a first hashed storage server, wherein each of the one or more files comprises a header and data, and wherein the hashed storage server stores and retrieves the data with a hash function that generates unique identifiers linked to content of the data rather than with a location-based, hierarchical file system;

receiving, using one or more computer processors, the same one or more files for storage at a second hashed storage server;

automatically obtaining with the one or more computer processors metadata associated with the data from the header of each of the one or more files;

storing the metadata in a first metadata storage device, wherein the metadata is stored in association with the data stored in the hashed storage server;

replicating the stored metadata and storing the replicated metadata in a second metadata storage devices;

receiving, using the one or more computer processors, a first query from a requester for content at an application server, wherein the application server comprises a local storage device, the local storage device and the content addressable storage server are distinct;

after or simultaneously with the local search according to the first query, sending a second query, related to the first query, to the hashed storage server;

receiving one or more files related to the second query from the hashed storage server;

excluding or flagging any duplicate files resulting from the first query, the second query, or both, the duplicate files comprising the same one or more files; and

sending to the requester a result set comprising the one or more files found at the local storage device based on the first query and the one or more files received from the hashed storage server based on the second query, wherein any duplicate files are excluded or flagged in the result set.

10. The method of claim 9, wherein the hashed storage servers store DICOM images and the metadata is related to the DICOM images.

11. A computer-implemented system for managing metadata in a content addressable storage system comprising:

a content addressable storage system comprising at least one computer processor configured to;

receive a file for storage and backup, said file to be stored using a first content addressable storage server and also backed up to a second content addressable storage server;

store metadata associated with the file in a first storage mechanism for storing metadata for content addressable storage;

replicate the stored metadata in a second storage mechanism;

receive a query from a requester for content;

search an application server for local content that is related to the received query but that is not stored in the content addressable storage system, wherein the application server and the content addressable storage system are distinct;

retrieve the local content stored on the application server;

send the local content to the requestor;

search the metadata storage mechanism for content related to the received query; and

when the metadata associated with the file is indicated by the query;

retrieve the associated file stored in the content addressable storage; and

send the retrieved file to the requester;

wherein sending the local content to the requester and sending the retrieved file to the requester comprise excluding or flagging any duplicate files.

12. The system of claim 11, wherein the computer-implemented system for managing metadata in a content addressable storage system is further configured to:

send the query to a second content addressable storage system;

receive one or more files related to the query from the second content addressable storage system; and

send to the requester the retrieved file and the one or more files related to the query from the second content addressable storage system.

13. The system of claim 11, wherein the content addressable storage system stores DICOM images having headers and the metadata is related to the headers of the DICOM images.

14. A computer-implemented system for managing metadata in a content addressable storage (CAS) system, comprising:

a CAS system comprising at least one computer processor configured to;

receive a file for storage and receive a second copy of the same file, said file and the second copy of the same file to be stored using content addressable storage; and

store metadata associated with the file and the second copy of the file in a searchable storage mechanism for storing metadata for CAS; and

an application server comprising at least one computer processor, the application server configured to;

receive a first query from a requester for content at an application server, wherein the application server comprises a local storage device and wherein the local storage device and the CAS system are distinct;

send a second query, related to the first query, to the CAS system;

receive one or more files related to the second query from the CAS system; and

send a result set to the requester, the result set comprising one or more files found locally based on the first query and the one or more files received from the CAS system based on the second query, wherein any duplicate files are excluded or flagged in the result set.

15. The system of claim 14, wherein the application server is further configured to:

send the second query to a second CAS system;

receive one or more files related to the second query from the second CAS system; and

send to the requester the one or more files received from the second CAS system based on the second query.

16. The system of claim 14, wherein the CAS system stores DICOM images and the metadata is related to the DICOM images.

17. A computer-implemented method comprising:

receiving, using one or more computer processors, a file for storage at a first fixed content storage CAS server, the file comprising a header and data;

receiving, using one or more computer processors, the same file for storage at a second CAS server;

automatically determining that the file meets the criteria for storing the file at a CAS server and storing the file at one or more CAS servers, wherein the criteria does not prevent or exclude duplicate or backup files;

automatically obtaining, with the one or more computer processors, from the header of the file, metadata associated with the file;

storing the metadata in at least one searchable metadata storage device at one or more CAS servers;

receiving, at a CAS server, using the one or more computer processors, a query from a requester for content at a CAS server;

forwarding the query, using the one or more computer processors, to at least one additional CAS server;

simultaneously with or after forwarding the query to the at least one additional CAS server, searching a local storage device for local content that is related to the query;

sending the local content to the requestor, wherein sending the local content comprises excluding or flagging any duplicate content;

searching the metadata storage device for content related to the received query; and

when the metadata associated with the file is indicated by the query;

retrieving the associated file stored in the CAS server; and

sending the retrieved file to the requester, wherein sending the retrieved file comprises excluding or flagging any duplicates.

18. The method of claim 17, wherein receiving a file for storage at a CAS server comprises storing the file using a hash function rather than a location-based directory table, the hash function configured to create an identifier uniquely and permanently linked to the content of the file itself, such that the content of the file may not change without changing the identifier.

19. The method of claim 17, additionally comprising waiting to send the retrieved file to the requester until after receiving results from forwarding the query to at least one additional CAS server, thereby allowing the results from searching the metadata storage device and searching the additional CAS server to be delivered to the requester together, and thereby allowing duplicates to be excluded or flagged in the combined results.

20. A computer-implemented system for managing fixed content storage, the system comprising:

a CAS server having at least one computer processor, the CAS server configured to;

receive files for storage and automatically recognize files as eligible for fixed content storage, wherein duplicate files are eligible for storage; and

store and retrieve files, including duplicate files, using a function that is independent of physical storage location and that maintains an identifier that is consistent for each file as long as the data comprising that file does not change;

an application server having at least one computer processor, the application server configured to;

receive files, including duplicate files, for local storage outside of the local cache, in non-temporary storage;

receive a query from a requester and convey the query to the fixed content storage server;

after or at the same time as conveying the inquiry, perform a search, based on the query, within its own local, non-temporary storage; and

retrieve any relevant local content or files and convey them to the requestor, while at the same time flagging or excluding duplicates; and

a metadata storage device associated with either the FCS server or the application server, the metadata storage device configured to;

store metadata associated with the file, and any duplicate files, in a searchable metadata database;

associate metadata with files stored on the CAS server or the application server; and

using these associations, allow the CAS server or the application server to search for content or files related to any received query, and when metadata associated with a file is indicated by the query, retrieve the relevant file or files, whether stored in the CAS server or in local storage and send any retrieved files to the requester, excluding or flagging duplicates.

21. The system of claim 20, wherein the metadata storage device comprises an XML file, flat file, or other searchable storage mechanism that associates the file stored in the CAS with corresponding metadata to allow more efficient searching of the contents of the CAS server.

22. The system of claim 20, wherein the CAS server is further configured to store and retrieve files using a hash function rather than a location-based directory table, the hash function configured to create an identifier uniquely and permanently linked to the content of the file itself, such that the content of the file may not change without changing the identifier.

23. The system of claim 20, wherein the application server is configured to store and retrieve data in its local storage based on a physical storage location or hierarchical file system rather than using CAS or a hashing function.