Scalable file replication and web-based access

• US 7,743,023 B2
• Filed: 02/01/2006
• Issued: 06/22/2010
• Est. Priority Date: 02/01/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Bozeman, Patrick E, Bjorner, Nikolaj S, Annamalai, Muthukaruppan, Bohlander, Michael J, Teodosiu, Dan, Steere, David C., Liu, Huisheng
• Primary Examiner(s): Vital; Pierre M
• Assistant Examiner(s): OBISESAN, AUGUSTINE KUNLE
• Application Number:
  US11/275,870
• Publication Number:
  US 20070185932A1
• Time in Patent Office: 1,602 Days
• Field of Search: None
• US Class Current: 707/612
• CPC Class Codes: G06F 16/184 implemented as replicated file system

Abstract

Various embodiments introduce the notion of a replication entity which implements a highly scalable file replication system. In one embodiment, the replication entity resides in the form of a scalable replication service. In at least some embodiments, the replication service provides a “drive in the sky” facility that can be used by individuals, such as subscribers, to synchronize their individual machines, such that their files are automatically replicated to a safe and always-on location. Alternatively or additionally, individuals such as subscribers can also access their files via a web-based interface when they are away from their machines.

Claims

1. A system comprising:

a processor; and

a memory coupled to the processor, the memory comprising computer-program instructions executable by the processor to perform operations comprising;

replicating a file with a replication entity to produce a staged file by determining whether the staged file is acceptable to a file system via determinations comprising;

in an event a directory corresponding to a unique global identifier (UID) of the parent resource is identified, contents of the staged file are accepted in the file system;

in an event meta-data corresponding to the UID of the parent indicates that the directory corresponding to the UID of the parent resource has been deleted, contents of the staged file are accepted in the file system responsive to re-animating the deleted directory corresponding to the UID of the parent resource by downloading the directory corresponding to the UID of the parent resource from a peer;

in an event meta-data corresponding to the UID of the parent indicates that the directory corresponding to the UID of the parent resource has been deleted and re-animating the deleted directory corresponding to the UID of the parent resource by downloading the directory corresponding to the UID of the parent resource from a peer is not successful, contents of the staged file are not accepted in the file system;

in an event no directory corresponding to the UID of the parent resource is identified, contents of the staged file are not accepted in the file system;

storing the staged file in a scalable binary large object (blob) store located on a first node, wherein the scalable blob store includes put, get, and delete operations in order to reduce storage semantics as compared to the file system, thereby minimizing file system constraints;

receiving from the scalable blob store, a metadata write request to write;

replication metadata,

file system metadata for the staged file, and

a locator token;

performing the metadata write to obtain;

the replication metadata,

the file system metadata for the staged file, and

the locator token;

storing, in a scalable metadata store located on a second node, the replication metadata, file system metadata for the staged file, and the locator token for the staged file from the scalable blob store;

using the locator token, in response to a web client request, to locate and retrieve a datastream associated with the staged file;

merging the replication metadata and blob store metadata to facilitate combined transactions.

2. The system of claim 1, wherein the computer-executable instructions further comprise instructions for merging replication metadata with file metadata in the scalable metadata store.

3. The system of claim 2, wherein the replication entity is configured to, responsive to updates to the scalable blob store, transact both file metadata and replication metadata together.

4. The system of claim 3, wherein the replication entity is configured to receive at least some updates via a web interface.

5. The system of claim 1, wherein the replication entity comprises a service.

6. The system of claim 1, wherein the replication entity comprises one or more hubs.

7. The system of claim 1, wherein the replication entity is configured as a state-based replication entity.

8. A system comprising:

one or more replication front ends individual ones of which being associated with a file replication engine that is executable on one or more processors to provide state-based file replication via exchange of version vectors;

at least one scalable binary large object (blob) store for storing file data in a manner that does not require full file system semantics;

at least one scalable metadata store for storing associated file metadata and a locator token for the file data in the at least one scalable blob store, wherein the system is embodied as a service, and wherein a replication client can participate in bi-directional replication with the replication service using the same replication protocols that would be utilized when replicating with another replication client running on a file system; and

one or more scalable file service access nodes at least some of which being configured to;

marshal files into a staged file format, the staged file format comprising;

a compressed marshaled representation of a file,

a signature of the file for differential transfer of the file, and

a jump table for navigating in the compressed marshaled representation of the file, and

unmarshal staged files.

9. The system of claim 8, wherein said at least one scalable metadata store stores replication metadata that is associated with said file metadata.

10. The system of claim 9 further comprising one or more access nodes individual ones of which being configured to transact both the replication metadata and associated file metadata together.

11. The system of claim 8 further comprising one or more replication group redirection services to direct replication clients to a particular instance of a replication front end.

12. The system of claim 8 further comprising one or more servers individual ones of which hosting a presentation layer configured to communicate with one or more web clients and to present HTML file system views for the one or more web clients.

13. A computer-implemented method comprising:

receiving, with a replication entity via the computer, a staged file;

sending, via the computer, to a storage access node, a write request for the staged file;

sending, via the computer, the staged file to a scalable binary large object (blob) store, wherein the scalable blob store;

stores the staged file via a marshaled replicated blob; and

provides reduced storage semantics comprising object put, get, and delete operations;

receiving, via the computer, from the scalable blob store, a locator token for the staged file;

sending, via the computer, to the storage access node, a metadata write request to write replication metadata, file system metadata for the staged file, and the locator token;

forwarding, via the computer, the metadata write request to a scalable metadata store; and

committing, via the computer, said metadata write request on said scalable metadata store, wherein the committing of the metadata write request comprises storing the replication metadata, the locator token and the file system metadata in the scalable metadata store.

14. The method of claim 13 further comprising, prior to said act of receiving a staged file, receiving from a replication client, a communication and responsive to said communication, directing said replication client to a particular instance of a replication front end comprising a replication engine for replicating files.

15. The method of claim 13 further comprising uploading a file from one or more web clients to the replication entity.

16. The method of claim 13 further comprising downloading a file from the replication entity to one or more web clients.

17. The method of claim 13 further comprising:

utilizing the locator token, in response to a web client request, to locate and retrieve a datastream associated with the staged file.

18. The method of claim 17, wherein:

the scalable blob store and the scalable metadata store are located on separate nodes; and

the staged file is stored in the scalable blob store in a staged format as a result of marshalling.

19. The method of claim 18, wherein the reduced file system semantics do not support a hierarchical file namespace.

20. The method of claim 13, wherein the staged file includes a compressed marshaled representation of the file.