MANAGING DATA TRANSFER BETWEEN ENDPOINTS IN A DISTRIBUTED COMPUTING ENVIRONMENT

• US 20090248693A1
• Filed: 03/27/2008
• Published: 10/01/2009
• Est. Priority Date: 03/27/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Fedorov, Vladimir D., Sagar, Akash Jeevan, Chung, Richard Y., Annamalai, Muthukaruppan
• Granted Patent: US 8,620,889 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor

Abstract

A file fetcher manager provides commonly-utilized management and resource allocation for multiple file fetchers that each implement a different type of mechanism or protocol for transferring data files between peer endpoints in meshes associated with a cloud-computing service. Each file fetcher is configured with both client-side and server-side components to retrieve and serve out data files. The file fetcher manager encapsulates the file fetchers to provide an abstract interface to callers while hiding the underlying details of the file fetchers. The file fetcher manager is arranged for managing simultaneous operations of the multiple file fetchers to route requests from the callers to the appropriate file fetchers, and for scheduling work items for the file fetchers so that data transfers efficiently utilize available resources while keeping the entire transfer process coherent and organized.

Claims

1. A method for providing a file fetcher arranged for serving and retrieving files to and from endpoints on a network supporting peer-to-peer file transfer, the method comprising the steps of:

associating work items in a work list with pending file requests from a caller at an endpoint, the work list being arranged to support prioritization of the work items;

obtaining a list of endpoints that contain at least a portion of files associated with the pending file requests;

maintaining a state for a file, the state identifying the file as being partially or completely downloaded;

retrieving at least a portion of a file in accordance with the work list, the list of endpoints, and the file state; and

serving at least a portion of a file responsively to a request from a caller at a remote endpoint.

2. The method of claim 1 including a further step of authorizing the remote endpoint to a core object from which the file portion will be served out to the remote caller.

3. The method of claim 1 in which the list of endpoints is provided by an enclosure locator service that is operable over the network.

4. The method of claim 1 in which the retrieving comprises downloading the file using chunked file fetching and including the further steps of receiving a list of available file chunks at an endpoint and receiving, from the caller, a number of file chunks and associated size of each chunk for creating the file state for a file.

5. The method of claim 4 in which the work list contains a plurality of work items for a file so that the file is retrieved in chunks in parallel from a plurality of endpoints.

6. The method of claim 5 including the further steps of locking a chunk to prevent duplicate requests from the caller, and requesting additional file chunks.

7. The method of claim 6 including a further step of implementing RDC-based chunked file fetching.

8. The method of claim 1 including a further step of implementing an interface between the file fetcher and a file fetcher manager that is commonly utilized by a plurality of file fetchers that are operable at an endpoint, the file fetcher manager routing requests among ones of the plurality of file fetchers.

9. A method performed by a local file fetcher manager for managing a plurality of local file fetchers, the method comprising the steps of:

exposing file retrieval and server functionality provided by the file fetchers to callers comprising a local caller at a local endpoint and to a remote caller at a remote peer endpoint, the local endpoint and remote peer endpoint being coupled to a cloud service supporting peer-to-peer file sharing;

querying one or more remote file fetcher managers to determine capabilities of remote file fetchers at the remote peer endpoints;

maintaining a list of the capabilities in a cache; and

routing requests from the callers to a file fetcher selected among the plurality of local file fetchers according to the capabilities.

10. The method of claim 9 including a further step of transferring work for retrieving a file among the local file fetchers.

11. The method of claim 9 including a further step of scheduling work items associated with the plurality of file fetchers with a global thread pool.

12. The method of claim 9 including the further steps of rank-ordering the local file fetchers by preference and routing file fetch requests by preference.

13. The method of claim 9 in which the capabilities include at least one of chunking capability or RDC capability.

14. The method of claim 9 including a further step of passing a pointer to a retrieved file to the local caller.

15. A method for scheduling work in a file fetcher, the method comprising the steps of:

maintaining state information pertaining to a plurality of remote peer endpoints on a cloud service, the state information including at least one of active communication channels to a remote peer endpoint, online and offline status of the remote peer endpoint, or request service history for the remote peer endpoint, the request service history identifying a quality of service provided by the remote peer endpoint; and

implementing a scheduler arranged for i) consuming policies governing the serving and retrieval of files to and from the plurality of remote peer endpoints, and receiving the state information and ii) scheduling the work in the file fetcher responsively to the state information and policies, the file fetcher being arranged with client-side and server-side components to respectively retrieve and serve files over communication channels supported by the cloud service.

16. The method of claim 15 in which the policies include client-side policies applicable to the client-side component, the client-side policies being selected from one of picking a set of active file retrieval requests from queued pending file retrieval requests, establishing or terminating connections to the remote peer endpoints, or queuing file retrieval requests on an open connection.

17. The method of claim 15 in which the policies include server-side policies applicable to the server-side component, the server-side policies being selected from one of picking a set of active incoming connections for servicing, setting allowable usage of an active incoming connection, or selecting queued requests to be de-queued for servicing.

18. The method of claim 15 in which the quality of service includes at least one of latency when serving a request, service success rate, or bandwidth provided by the remote peer endpoint.

19. The method of claim 15 including the further steps of queuing incoming requests to the server-side component and calling the scheduler when new requests have been queued.

20. The method of claim 15 including the further steps of queuing fetch requests for the client-side component and calling the scheduler when new requests have been queued or when a file download is completed or cancelled.