Distributed file system

• US 10,853,329 B2
• Filed: 06/25/2017
• Issued: 12/01/2020
• Est. Priority Date: 03/31/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Thereska, Eno, Gunawardena, Dinan Srilal, Scott, James W., Harper, Richard
• Primary Examiner(s): Jacob, Ajith
• Application Number:
  US15/632,389
• Publication Number:
  US 20170293633A1
• Time in Patent Office: 1,255 Days
• CPC Class Codes: G06F 1/3203 Power management, i.e. event-based initiation of a power-saving mode
• CPC Class Codes: G06F 1/3209 Monitoring remote activity, e.g. over telephone lines or network connections
• CPC Class Codes: G06F 1/3296 by lowering the supply or operating voltage
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/1824 implemented using Network-attached Storage [NAS] architecture distributed or networked storage systems G06F3/067; protocols for distributed storage of data in a network H04L67/1097
• CPC Class Codes: G06F 16/183 Provision of network file services by network file servers, e.g. by using NFS, CIFS network file access protocols H04L67/1097
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/487 using geographical or spatial information, e.g. location
• CPC Class Codes: G06F 9/4418 Suspend and resume; Hibernate and awake
• CPC Class Codes: H04L 63/205 involving negotiation or determination of the one or more network security mechanisms to be used, e.g. by negotiation between the client and the server or between peers or by selection according to the capabilities of the entities involved negotiation of communication capabilities H04L69/24

Abstract

A distributed file system for devices is described. In an embodiment, each data element stored on one of the devices has an associated location and availability attribute. The location attribute is stored co-located with the data element. The availability attribute and a copy of the location attribute are stored by a metadata service. When a client on a device needs to access a data element, it sends a request to the metadata service to find the location of the data element. If the data element is available, this information is provided to the client and this may involve waking a dormant device which holds the data element. Where the data element is not available, read only access may be granted to a cached copy of the data element. Where replication is used and one of the devices holding a replica is unavailable, the system may use write off-loading.

Claims

1. A method of operating a distributed data storage system, the method comprising:

storing, in a data store, a plurality of data elements, each data element having an associated location attribute identifying a location of the data element and availability attribute identifying availability of the data element;

sending, using a main processor, a request to send one or more read/write requests to a remote device in the distributed storage system based on location information associated with one of the plurality of data elements, the request being sent via a communication interface; and

maintaining, using a processing element, network connectivity of a device associated with the main processor when the main processor is off; and

waking the main processor on receipt of a read/write request from a remote device in the distributed storage system.

2. A method according to claim 1, further comprising providing read only access to the plurality of data elements when communication with a metadata service is not possible.

3. A method according to claim 1, wherein said one of the plurality of data elements comprises one of a set of replicas of a data element stored on a different device in the distributed data storage system, wherein a replica stored in the data store being designated a primary replica, and further comprising sending a message to a metadata service prior to the device entering a reduced power mode to trigger migration of the primary designation to another of the replicas of the data element.

4. A method according to claim 1, further comprising:

writing data to a data element in the data store in response to receipt of a write request relating to the data element from a remote device;

determining remote devices storing replicas of the data element are not available; and

sending the write request to a short-term store in the distributed storage system incident to the remote devices storing the replicas not being available.

5. A method according to claim 1, further comprising:

receiving a read/write request relating to a data element in the data store from a remote device

determining a user location associated with the write request;

determining if an access control policy associated with the data element is satisfied, the access control policy being defined in terms of at least one of the user location and the location attribute associated with the data element; and

servicing the read/write request if the access control policy is satisfied.

6. A method according to claim 1, wherein the processing element is mounted within a module, the module further comprising a network interface and wherein the processing element is arranged to maintain network connectivity when the main processor is off via the network interface and to receive read/write requests via the network interface.

7. A method according to claim 1, wherein the data store is further arranged to store a cached copy of metadata received from a metadata service in the distributed data storage system, the metadata comprising location attributes associated with the plurality of data elements, and wherein the main processor is arranged to send a request for updated metadata in response to a failure of a read/write request sent to a remote device based on location information from the cached copy of metadata.

8. A method according to claim 1, wherein the device comprises a consumer device and wherein the distributed data storage system comprises a plurality of different consumer devices.

9. A device for use in a distributed data storage system, the device comprising:

a data store arranged to store a plurality of data elements, each data element having an associated location attribute identifying a location of the data element and availability attribute identifying availability of the data element;

a communication interface;

one or more processors arranged to send one or more requests for read/write requests to a remote device in the distributed storage system based on location information received associated with one of the plurality of data elements; and

a processing element arranged to maintain network connectivity when the device is in a reduced power mode and the main processor is off and to wake the main processor on receipt of a read/write request from a remote device in the distributed storage system.

10. A device according to claim 9, wherein the main processor is arranged to provide read-only access to the plurality of data elements when communication with a metadata service is not possible.

11. A device according to claim 9, wherein one of the plurality of data elements comprises one of a set of replicas of a data element, each replica being stored on a different device in the distributed data storage system and the replica stored in the data store being designated a primary replica, and wherein the main processor is further arranged to send a message to a metadata service prior to entering the reduced power mode to trigger migration of the primary designation to another of the replicas of the data element.

12. A device according to claim 9, wherein the main processor is further arranged to:

write data to a data element in the data store in response to receipt of a write request relating to one of the data element from a remote device;

determine if all remote devices storing replicas of the data element are available; and

send the write request to a short-term store in the distributed storage system if all the remote devices are not available.

13. A device according to claim 9, wherein the main processor is further arranged to:

receive a read/write request relating to a data element in the data store from a remote device

determine a user location associated with the write request;

determine if an access control policy associated with the data element is satisfied, the access control policy being defined in terms of at least one of the user location and the location attribute associated with the data element; and

service the read/write request if the access control policy is satisfied.

14. A device according to claim 9, wherein the processing element is mounted within a module, the module further comprising a network interface and wherein the processing element is arranged to maintain network connectivity when the main processor is off via the network interface and to receive read/write requests via the network interface.

15. A device according to claim 9, wherein the data store is further arranged to store a cached copy of metadata received from a metadata service, the metadata comprising the location and availability attributes associated with each data element stored in the distributed data storage system and wherein the main processor is arranged to send a request for updated metadata in response to a failure of a read/write request sent to a remote device based on location information from the cached copy of metadata.

16. A device according to claim 9, which comprises a consumer device and wherein the distributed data storage system comprises a plurality of different consumer devices.

17. A system for managing a distributed data storage system, the system comprising:

memory arranged to store data elements and a location attribute associated with each data element, the location attribute indicating a geographical location of the data element; and

one or more processors;

a processing element arranged to maintain network connectivity when a in the distributed data storage system is in a reduced power mode and the main processor is off and to wake the main processor on receipt of a read/write request from another device in the distributed data storage system; and

metadata stored with the data element arranged to store both a copy of the location attribute and an availability attribute associated of the data element, the availability attribute indicating an availability status of the associated data element and one or more service access requests for data elements received from devices based on the location and availability attributes.

18. A system according to claim 17, further comprising a short-term store and wherein the one or more processors are arranged to:

write data to the data element in the memory in response to receipt of a write request relating to the data element from a remote device;

determine if all remote devices storing replicas of the data element are available; and

incident to all the remote devices not being available, send the write request to the short-term store.

19. A system according to claim 17, wherein the main processor is further arranged to:

receive a read/write request relating to the data element;

determine a user location associated with the write request;

determine an access control policy associated with the data element is satisfied, the access control policy being defined in terms of at least one of the user location and the location attribute associated with the data element; and

service the read/write request incident to the access control policy being satisfied.

20. A system according to claim 17, wherein the memory is further arranged to store a cached copy of metadata received from a metadata service, the metadata comprising the location and availability attributes associated with each data element stored in the distributed data storage system and wherein the main processor is arranged to send a request for updated metadata in response to a failure of a read/write request sent to a remote device based on location information from the cached copy of metadata.