Accelerated data transfer using common prior data segments

• US 9,317,506 B2
• Filed: 09/22/2006
• Issued: 04/19/2016
• Est. Priority Date: 09/22/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Keith, Robert O. Jr.
• Primary Examiner(s): NGUYEN, KIM T
• Application Number:
  US11/525,729
• Publication Number:
  US 20080077630A1
• Time in Patent Office: 3,497 Days
• Field of Search: 707/699, 707/747
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: G06F 16/2358 Change logging, detection, and notification replication G06F16/27
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/24562 Pointer or reference processing operations

Abstract

Accelerating data transfers is described herein. When a second computing system is requested to transfer a file to a first computing system, a data segment is sent to the first computing system instead of the entire file. The data segment is then compared to data stored within a data store on the first computing system. If the data segment and data within the data store match, then the file does not need to be transferred, and a pointer points to the file already located on the first computing system. If the data segment does not match any data stored in the data store, then the file is transferred from the second computing system to the first computing system. By comparing only the data segment instead of sending an entire file, data transfer is able to be greatly expedited in situations where the data is common between systems.

Claims

1. A method of accelerating data transfer comprising:

a. storing data in a data store on a first computing system wherein the data corresponds to one or more files stored on the first computing system;

b. transferring a data segment from a source file from a second computing system to the first computing system over a network;

c. scanning the data store on the first computing system for the data segment received from the second computing system;

d. generating one or more pointers on the first computing system to the one or more corresponding files of one or more matching data segments, if the one or more matching data segments are identified in the data store on the first computing system; and

e. transferring a copy of the source file from the second computing system to the first computing system, if the one or more matching data segments are not identified in the data store on the first computing system.

2. The method as claimed in claim 1 wherein the data store is a database.

3. The method as claimed in claim 1 wherein the first computing system is a target system and the second computing system is a source system.

4. The method as claimed in claim 1 wherein the first computing system is a server and the second computing system is a client system.

5. The method as claimed in claim 4 wherein the client system is selected from the group consisting of a personal computer, a PDA, a cell phone, a laptop, a thin client, a Mac computer, an mp3 player and a gaming console.

6. The method as claimed in claim 1 wherein the first computing system is a first client system and the second computing system is a second client system.

7. The method as claimed in claim 1 wherein the data segment is one or more cyclic redundancy checks and the data in the data store includes cyclic redundancy checks and the data segment and the data are compared.

8. The method as claimed in claim 1 wherein the data segment is a unique database key and the data in the data store includes database keys and the data segment and the data are compared.

9. The method as claimed in claim 1 wherein the data segment is a hash and the data in the data store includes hashes and the data segment and the data are compared.

10. The method as claimed in claim 1 wherein the data store grows as more files are stored on the first computing system.

11. The method as claimed in claim 1 wherein the files stored on the first computing system are minimized by implementing the data store.

12. The method as claimed in claim 1 wherein one or more additional computing systems are coupled to the first computing system.

13. The method as claimed in claim 1 further comprising transferring only a first section of the source file when only a second section of the source file is found within the data store.

14. The method as claimed in claim 1 wherein a standard operating system and file system are utilized on the first computing system and the second computing system.

15. The method of claim 1 wherein if at least one of the corresponding files of the matching data segments is deleted from a first location on the first computer system where the file was stored, the file is automatically transferred to and stored at a second location on the first computer system where at least one of the one or more pointers were stored as a replacement for the at least one of the one or more pointers.

16. The method of claim 15 wherein the remaining one or more pointers that were not replaced by the file are automatically reconfigured such that the remaining one or more pointers point to the file stored at the second location instead of the first location.

17. A system for accelerating data transfer comprising:

a. a first computing system having a first memory device for storing one or more files and a data store for storing data corresponding to the one or more files; and

b. a second computing system having a second memory device, wherein the second computing system is coupled to the first computing system,

wherein a data segment stored in the second memory device is transferred to the first computing system, and is compared to the data within the data store on the first memory device after being received from the second memory device, further wherein a pointer to the one or more files on the first memory device is added on the first memory device if the data segment is found within the data store on the first memory device, but a copy of a source file corresponding to the data segment is transferred from the second memory device to the first memory device if the data segment is not found in the data store.

18. The system as claimed in claim 17 wherein the data store is a database.

19. The system as claimed in claim 17 wherein the first computing system is a target system and the second computing system is a source system.

20. The system as claimed in claim 17 wherein the first computing system is a server and the second computing system is a client system.

21. The system as claimed in claim 20 wherein the client system is selected from the group consisting of a personal computer, a PDA, a cell phone, a laptop, a thin client, a Mac computer, an mp3 player and a gaming console.

22. The system as claimed in claim 17 wherein the first computing system is a first client system and the second computing system is a second client system.

23. The system as claimed in claim 17 wherein the data segment is one or more cyclic redundancy checks and the data in the data store includes cyclic redundancy checks and the data segment and the data are compared.

24. The system as claimed in claim 17 wherein the data segment is a unique database key and the data in the data store includes database keys and the data segment and the data are compared.

25. The system as claimed in claim 17 wherein the data segment is a hash and the data in the data store includes hashes and the data segment and the data are compared.

26. The system as claimed in claim 17 wherein the data store grows as more files are stored on the first memory device.

27. The system as claimed in claim 17 wherein the files stored on the first memory device are minimized by implementing the data store.

28. The system as claimed in claim 17 further comprising one or more additional computing systems coupled to the first computing system.

29. The system as claimed in claim 17 wherein only a first section of the source file is transferred when only a second section of the source file is found within the data store.

30. The system as claimed in claim 17 wherein a standard operating system and file system are utilized on the first computing system and the second computing system.

31. The system as claimed in claim 17 further comprising a network coupling the first computing system and the second computing system.

32. A network of systems for accelerating data transfers comprising:

a. one or more source systems having one or more source memory devices for transferring a data segment corresponding to a source file stored on the one or more source memory devices; and

b. one or more target systems having one or more target memory devices for storing data in a data store corresponding to one or more files on the one or more target memory devices and for comparing the data segment received from the one or more source memory devices with the data in the data store where if the data segment is found, a pointer is generated on the one or more target memory devices to point to a corresponding file in the one or more files on the target memory device instead of transferring the source file over a network, wherein the one or more target systems are coupled to the one or more source systems.

33. The network of systems as claimed in claim 32 wherein the data store is a database.

34. The network of systems as claimed in claim 32 wherein the one or more target systems are one or more servers and the one or more source systems are one or more client systems.

35. The network of systems as claimed in claim 32 wherein the one or more client systems are selected from the group consisting of personal computers, PDAs, cell phones, laptops, thin clients, Mac computers, mp3 players and gaming consoles.

36. The network of systems as claimed in claim 32 wherein the data segment is one or more cyclic redundancy checks and the data in the data store includes cyclic redundancy checks and the data segment and the data are compared.

37. The network of systems as claimed in claim 32 wherein the data segment is a unique database key and the data in the data store includes database keys and the data segment and the data are compared.

38. The network of systems as claimed in claim 32 wherein the data segment is a hash and the data in the data store includes hashes and the data segment and the data are compared.

39. The network of systems as claimed in claim 32 wherein the data store grows as more files are stored on the one or more target memory devices.

40. The network of systems as claimed in claim 32 wherein the files stored on the one or more target memory devices are minimized by implementing the data store.

41. The network of systems as claimed in claim 32 wherein only a first section of the source file is transferred when only a second section of the source file is found within the data store.

42. The network of systems as claimed in claim 32 wherein a standard operating system and file system are utilized on the one or more target systems and the one or more source systems.

43. A storage system configured to receive data from a plurality of source computing systems comprising:

a memory device comprising;

a. one or more files;

b. a set of information corresponding to the one or more files; and

c. a data store for storing the set of information,

wherein a data segment received from a source computing system in the plurality of source computing systems is compared with the set of information stored within the data store and a pointer is generated on the storage system to point to a corresponding file in the one or more files if the data segment is found but if the data segment is not found within the data store, a copy of a source file on the source computing system corresponding to the data segment is transferred to the storage system.

44. The storage system as claimed in claim 43 wherein the data store is a database.

45. The storage system as claimed in claim 43 wherein the data segment is one or more cyclic redundancy checks and the set of information in the data store includes cyclic redundancy checks and the data segment and the set of information are compared.

46. The storage system as claimed in claim 43 wherein the data segment is a unique database key and the set of information in the data store includes database keys and the data segment and the set of information are compared.

47. The storage system as claimed in claim 43 wherein the data segment is a hash and the set of information in the data store includes hashes and the data segment and the set of information are compared.

48. The storage system as claimed in claim 43 wherein the data store grows as more files are stored on the storage system.

49. The storage system as claimed in claim 43 wherein the files stored on the storage system are minimized by implementing the data store.

50. The storage system as claimed in claim 43 wherein only a first section of the source file is transferred when only a second section of the source file is found within the data store.

51. The storage system as claimed in claim 43 wherein a standard operating system and file system are utilized on the storage system.

52. A method of accelerating data transfer comprising:

a. storing data in a data store on a target computing system wherein the data corresponds to one or more files stored on the target computing system;

b. transferring a data segment of a source file from a source computing system to the target computing system over a network in response to a request for the source file by the first computing system;

c. scanning the data store on the target computing system for the data segment received from the source computing system;

d. generating a pointer on the target computing system that points to at least one of the one or more files, if the data that corresponds to the at least one of the one or more files matches the data segment; and

e. transferring a copy of the source file from the source computing system to the target computing system, if the data that corresponds to the one or more files does not match the data segment.