Limited deduplication scope for distributed file systems

• US 10,430,385 B1
• Filed: 06/30/2016
• Issued: 10/01/2019
• Est. Priority Date: 12/05/2014
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Desai, Rajiv, Tolia, Niraj
• Primary Examiner(s): Le, Uyen T
• Application Number:
  US15/199,599
• Time in Patent Office: 1,188 Days
• Field of Search: None
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/2379 Updates performed during online database operations; commit processing

Abstract

A method, article of manufacture, and apparatus for limited deduplication scope on a distributed file system is discussed. A write request is received from a client at the metadata server (“MDS”), where the write request comprises a data object identifier and a preferred object store identifier. The MDS determines whether a preferred object store associated with the preferred object store identifier contains a copy of a data object associated with the data object identifier. A write URL comprising the data object identifier and a object store location associated with the preferred object store is transmitted to the client when the preferred object store does not contain the copy of the data object.

Claims

1. A method for limited deduplication scope on a distributed file system (“

DFS”

) in a system that includes a plurality of object stores, the method comprising;

receiving a write request from a client at the metadata server (“

MDS”

), wherein the write request comprises a data object identifier and a preferred object store identifier;

determining at the MDS whether a preferred object store associated with the preferred object store identifier contains a copy of a data object associated with the data object identifier, wherein the MDS is separate from the preferred object store; and

transmitting a write URL comprising the data object identifier and an object store location associated with the preferred object store to the client when the preferred object store does not contain the copy of the data object, wherein the client writes the data object to the preferred object store using the write URL, wherein communications between the client and the MDS are independent and separate from communications between the client and the preferred object store.

2. The method of claim 1, further comprising receiving a data object commit from the client.

3. The method of claim 2, further comprising updating an object store mask on the MDS in response to the data object commit.

4. The method of claim 1, wherein the data object is stored on a second object store within the DFS, wherein the second object store is not the preferred object store.

5. A computer program product for limited deduplication scope on a distributed file system (“

DFS”

) in a system that includes a plurality of object stores, the computer program product comprising a non-transitory computer readable medium encoded with computer executable program, the code enabling;

receiving a write request from a client at the metadata server (“

MDS”

), wherein the write request comprises a data object identifier and a preferred object store identifier;

determining at the MDS whether a preferred object store associated with the preferred object store identifier contains a copy of a data object associated with the data object identifier, wherein the MDS is separate from the preferred object store; and

transmitting a write URL comprising the data object identifier and an object store location associated with the preferred object store to the client when the preferred object store does not contain the copy of the data object, wherein the client writes the data object to the preferred object store using the write URL, wherein communications between the client and the MDS are independent and separate from communications between the client and the preferred object store.

6. The computer program product of claim 5, further comprising receiving a data object commit from the client.

7. The computer program product of claim 6, further comprising updating an object store mask on the MDS in response to the data object commit.

8. The computer program product of claim 5, wherein the data object is stored on a second object store within the DFS, wherein the second object store is not the preferred object store.

9. A system for limited deduplication scope on a distributed file system (“

DFS”

) in a system that includes a plurality of object stores, the system comprising a computer processor configured to execute instructions comprising;

receiving a write request from a client at the metadata server (“

MDS”

), wherein the write request comprises a data object identifier and a preferred object store identifier;

determining at the MDS whether a preferred object store associated with the preferred object store identifier contains a copy of a data object associated with the data object identifier, wherein the MDS is separate from the preferred object store; and

transmitting a write URL comprising the data object identifier and an object store location associated with the preferred object store to the client when the preferred object store does not contain the copy of the data object, wherein the client writes the data object to the preferred object store using the write URL, wherein communications between the client and the MDS are independent and separate from communications between the client and the preferred object store.

10. The computer program product of claim 9, further comprising receiving a data object commit from the client.

11. The computer program product of claim 10, further comprising updating an object store mask on the MDS in response to the data object commit.

12. The computer program product of claim 9, wherein the data object is stored on a second object store within the DFS, wherein the second object store is not the preferred object store.