Data storage and retriveval systems and related methods of storing and retrieving data

• US 20050044170A1
• Filed: 08/19/2003
• Published: 02/24/2005
• Est. Priority Date: 08/19/2003
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Postini, Inc. (Alphabet Inc.)
• Inventors: Prall, Joh H., Petry, Scott M., Oswall, Michael J., Lund, Peter Kevin, Carroll, Dorion A., Cox, Fredric L.
• Granted Patent: US 7,584,264 B2
• Time in Patent Office: Days
• US Class Current: 709/217
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: H04L 51/00 User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
• CPC Class Codes: H04L 51/222 using geographical location information, e.g. messages transmitted or received in proximity of a certain spot or area
• CPC Class Codes: H04L 61/4511 using domain name system [DNS]
• CPC Class Codes: H04L 61/4555 Directories for electronic mail or instant messaging
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 69/329 in the application layer [OSI layer 7]
• CPC Class Codes: H04L 9/40 Network security protocols

Abstract

Data storage and retrieval systems and related methods are disclosed herein. In one aspect, a system includes data processing servers configured to receive incoming data and transmit the data for storage. The system also includes data storage servers each coupled to one or more data storage units and configured to receive transmitted data for writing to the one or more data storage units, and to read data from the one or more data storage units. Furthermore, the systems may include data retrieval servers coupled to one or more of the data storage servers and configured to retrieve data from the one or more data storage units. The system further includes process modules each associated with one of the data storage servers, where at least two of the process modules are configured to write a portion of the data to corresponding data storage units. In addition, each of the process modules are configured to transmit an acknowledgment associated with each of the corresponding at least two data storage units upon the writing of the data portion in the at least two data storage units. Furthermore, some embodiments may include a key manager coupled to the data processing servers and configured to receive keys that indicate the location of the data portions in the one or more data storage units. Additionally, each data retrieval server may be coupled to the key manager and employ the keys to retrieve data from the one or more data storage units.

Claims

20. A method for storing and retrieving data, comprising:

receiving incoming data and transmitting the data for storage;

writing a portion of the data in at least two data storage units;

transmitting an acknowledgment associated with each of the at least two data storage units upon the writing of the data portion in the at least two data storage units; and

retrieving the data portion from one or more of the at least two data storage units.

21. A method according to claim 20, wherein the transmitting the data comprises transmitting the data for storage across a TCP/IP connection.

22. A method according to claim 20, wherein the writing the portion of data and the transmitting an acknowledgment comprise writing the data portion in a first of the at least two data storage units and transmitting an acknowledgment upon the writing of the data portion in the first data storage unit using a first process module, and comprise writing the data portion in a second of the at least two data storage units and transmitting an acknowledgment upon the writing of the data portion in the second data storage unit using a second process module.

23. A method according to claim 20, wherein the writing the portion of data and the transmitting an acknowledgment comprise writing the data in the at least two data storage units and transmitting an acknowledgement using a process module in response to the writing of the data portion in the at least two data storage units.

24. A method according to claim 20, further comprising transmitting data storage information keys in response to the writing of the data portion in the at least two data storage units, the data storage information keys corresponding to the at least two data storage units.

25. A method according to claim 24, wherein the data storage information keys comprise the location of the data portion on the corresponding data storage units.

26. A method according to claim 25, wherein retrieving the data portion comprises locating the data portion on the one or more data storage units using the data storage information keys and retrieving the data portion therefrom.

27. A method according to claim 20, wherein transmitting an acknowledgment associated with each of the at least two data storage units further comprises transmitting the acknowledgements from a first process module in response to:

receiving an acknowledgment from a second process module based on the writing of the data portion in a second of the at least two data storage units, and

writing the data portion in a first of the at least two data storage units.

28. A method according to claim 20, wherein receiving incoming data and transmitting the data for storage comprises receiving incoming data and transmitting the data for storage using an electronic mail data processing server.

29. A method according to claim 28, wherein the data comprises electronic mail message data.

30. A method according to claim 20, wherein retrieving the data portion from the one or more data storage units comprises retrieving the data portion from the one or more data storage units across a computer network using a web server.

31. A method according to claim 20, wherein transmitting an acknowledgment comprises transmitting an acknowledgement associated with each of the at least two data storage units upon the writing of the data portion in the at least two data storage units when the data portion is not written to at least two other data storage units.

32. A method according to claim 20, wherein writing the portion of data further comprises writing the portion of data in at least two data storage units selected at random from plurality of data storage units.

33. A method according to claim 32, wherein the at least two data storage units are selected at random using records stored within a domain name system server associated with data storage servers coupled to the data storage units.

34. A method according to claim 20, wherein the at least two data storage units are selected using records stored within a domain name system server associated with data storage servers coupled to the data storage units based on the loading of the data storage servers.

35. A method according to claim 20, wherein writing the portion of data in at least two data storage units further comprises buffering the data portion in a memory module, and writing the data portion in the at least two data storage units when the memory module is full.

36. A method according to claim 35, wherein writing the portion of data in at least two data storage units further comprises buffering the data portion in a memory module in consecutive data blocks based on the storage date of the data portion, and writing the data portion in the at least two data storage units when the memory module is full.

37. A data storage and retrieval system, comprising:

a data processing server configured to receive incoming data and to transmit the data for storage;

a plurality of data storage servers each coupled to one or more of a plurality of data storage units and configured to receive a portion of the data for writing to at least two of the plurality of data storage units;

storage server records comprising configuration information corresponding to connection path and availability of each of the plurality of data storage servers; and

a domain name system server coupled to the data processing server and configured to store the storage server records and to supply the storage server records to the data processing server for use in identifying at least two of the plurality of data storage servers having an available connection, the data processing server further configured to establish connections with the at least two data storage servers based on the identification of the at least two data storage servers using the storage server records.

38. A data storage and retrieval system according to claim 37, wherein the configuration information comprises TCP/IP connection information for each of the plurality of data storage servers.

39. A data storage and retrieval system according to claim 37, further comprising a data retrieval server coupled to one or more of the at least two data storage servers and configured to retrieve the data portion read by the one or more data storage servers from one or more of the at least two data storage units.

40. A data storage and retrieval system according to claim 39, wherein the data retrieval server comprises a web server configured to retrieve the data portion from the one or more data storage units across a computer network.

41. A data storage and retrieval system according to claim 37, wherein the storage server records are configured to be updated with configuration information corresponding to data storage servers added to the plurality of data storage servers.

42. A data storage and retrieval system according to claim 37, wherein the storage server records are configured to be updated in response to the removal of data storage servers from the plurality of data storage servers.

43. A data storage and retrieval system according to claim 37, wherein the data processing server comprises an electronic mail data processing server.

44. A data storage and retrieval system according to claim 43, wherein the data comprises electronic mail message data.

45. A method for storing and retrieving data, comprising:

receiving incoming data and transmitting the data for storage;

creating storage server records comprising configuration information corresponding to connection path and availability of each of a plurality of data storage servers;

identifying at least two of the plurality of data storage servers having an available connection using the storage server records;

establishing a connection to the at least two data storage servers; and

writing a portion of the data in at least two data storage units corresponding to the at least two data storage servers and coupled thereto.

46. A method according to claim 45, wherein the configuration information comprises TCP/IP connection information for each of the plurality of data storage servers.

47. A method according to claim 45, further comprising retrieving the data portion from one or more of the at least two data storage units.

48. A method according to claim 47, wherein retrieving the data portion further comprises retrieving the data portion from one or more of the at least two data storage units across a computer network using a web server.

49. A method according to claim 45, wherein the storage server records are configured to be updated with configuration information corresponding to data storage servers added to the plurality of data storage servers.

50. A method according to claim 45, wherein the storage server records are configured to be updated in response to the removal of data storage servers from the plurality of data storage servers.

51. A method according to claim 45, wherein receiving incoming data and transmitting the data for storage comprises receiving incoming data and transmitting the data for storage using an electronic mail data processing server.

52. A method according to claim 51, wherein the data comprises electronic mail message data.

53. A data storage and retrieval system, comprising:

a data processing server configured to receive incoming data and to transmit the data for storage;

a plurality of data storage servers each coupled to one or more of a plurality of data storage units and configured to receive a portion of the data for writing to at least two of the data storage units;

a data retrieval server coupled to one or more of the plurality of data storage servers and configured to retrieve the data portion read by the one or more data storage servers and written to the at least two data storage units from one or more of the at least two data storage units;

data storage information keys corresponding to each of the data storage units and comprising offset information corresponding to the location of the data portion in the at least two data storage units; and

a key manager associated with the data retrieval server and configured to store the data storage information keys therein.

54. A data storage and retrieval system according to claim 53, wherein each of the plurality of data storage servers is configured to transmit a data storage information key in response to the writing of the data portion to the at least to data storage units.

55. A data storage and retrieval system according to claim 53, wherein each of the plurality of the data storage servers are further configured to generate the offset information in corresponding data storage information keys.

56. A data storage and retrieval system according to claim 53, wherein the data storage information keys further comprise deletion information corresponding to a date on which the data portion written to the at least two data storage units is deleted.

57. A data storage and retrieval system according to claim 56, wherein the data processing server is further configured to generate the deletion information.

58. A data storage and retrieval system according to claim 56, wherein the data storage units are configured to delete the data portion on a predetermined date corresponding to the deletion information.

59. A data storage and retrieval system according to claim 53, wherein the data retrieval server employs one of the data storage information keys to locate the data portion in the one or more data storage units for retrieval.

60. A data storage and retrieval system according to claim 53, wherein the data processing server comprises an electronic mail data processing server.

61. A data storage and retrieval system according to claim 60, wherein the data comprises electronic mail message data.

62. A data storage and retrieval system according to claim 53, wherein the data retrieval server comprises a web server configured to retrieve the data portion read by the one or more data storage servers from the one or more data storage units across a computer network.

63. A method for storing and retrieving data, comprising:

receiving incoming data and transmitting the data for storage;

writing a portion of the data in at least two data storage units;

creating data storage information keys corresponding to each of the at least two data storage units and comprising offset information corresponding to the location of the data portion in the at least two data storage units;

storing the data storage information keys in a key manager; and

retrieving the data portion from one or more of the at least two data storage units.

1. A data storage and retrieval system, comprising:

a data processing server configured to receive incoming data and to transmit the data for storage;

a plurality of data storage servers each coupled to one or more data storage units and configured to receive transmitted data for writing to the one or more data storage units, and configured to read data from the one or more data storage units;

a data retrieval server coupled to one or more of the plurality of data storage servers and configured to retrieve data read by the one or more data storage servers from the one or more data storage units; and

a plurality of process modules each associated with one of the plurality of data storage servers, at least two of the process modules configured to write a portion of the data to corresponding data storage units, each of the at least two process modules further configured to transmit an acknowledgment associated with each of the corresponding at least two data storage units upon the writing of the portion of data in the corresponding at least two data storage units.

2. A data storage and retrieval system according to claim 1, wherein the plurality of data storage servers is configured to receive the transmitted data across a TCP/IP connection with the data processing server.

3. A data storage and retrieval system according to claim 1, wherein the process module is further configured to generate the acknowledgements associated with the at least two data storage servers.

4. A data storage and retrieval system according to claim 1, wherein each of the at least two process module is configured to transmit the acknowledgments from each of the corresponding at least two data storage servers to the data processing server in response to the writing of the data portion in the at least two data storage units.

5. A data storage and retrieval system according to claim 1, further comprising a key manager associated with the data retrieval server, wherein the key manager is operative to receive a data storage information key transmitted by one of the plurality of process modules associated with one of the plurality of data storage servers in response to the writing of the data portion in one of the data storage units coupled to the one data storage server.

6. A data storage and retrieval system according to claim 1, further comprising a key manager associated with the data retrieval server, wherein the key manager is operative to receive at least two data storage information keys transmitted by the at least two process modules associated with at least two of the plurality of data storage servers in response to the writing of the data portion in the at least two data storage units coupled to the at least two data storage servers.

7. A data storage and retrieval system according to claim 6, wherein the data storage information keys comprise the location of the data portion on the corresponding data storage units.

8. A data storage and retrieval system according to claim 7, wherein the data retrieval server employs one of the data storage information keys to locate the data portion on the corresponding data storage unit and retrieve it therefrom.

9. A data storage and retrieval system according to claim 1, wherein a first of the at least two data storage servers associated with a first data storage unit is coupled to a second data storage server associated with a second data storage unit, and wherein the process module in the first data storage server is configured to transmit an acknowledgment associated with each of the first and second data storage servers in response to:

receiving an acknowledgment from the process module in the second data storage server based on the writing of the data portion in second data storage unit, and

the writing of the data portion in the first data storage unit.

10. A data storage and retrieval system according to claim 9, wherein the process module associated with the first data storage server is configured to transmit an acknowledgment from each of the at least two data storage servers to the data processing server.

11. A data storage and retrieval system according to claim 1, wherein the data processing server comprises an electronic mail data processing server.

12. A data storage and retrieval system according to claim 11, wherein the data comprises electronic mail message data.

13. A data storage and retrieval system according to claim 1, wherein the data retrieval server comprises a web server configured to retrieve the data portion read by the one or more data storage servers from the one or more data storage units across a computer network.

14. A data storage and retrieval system according to claim 1, wherein at least two of the plurality of data storage servers comprise a first group of data storage servers coupled to corresponding data storage units and at least two others of the plurality of data storage servers comprise a second group of data storage servers coupled to corresponding data storage units, and wherein each of the plurality of process modules is further configured to transmit an acknowledgment associated with corresponding data storage servers of the second group in response to the writing of the data portion to the data storage units associated with the second group of data storage servers when the data portion is not written to the data storage units associated with data storage servers of the first group.

15. A data storage and retrieval system according to claim 1, wherein at least two data storage servers having an available connection are selected from the plurality of data storage servers at random for storage of the data portion in corresponding data storage units.

16. A data storage and retrieval system according to claim 15, further comprising a domain name system server coupled to the data processing server, wherein the data processing server is configured to select the at least two data storage servers at random using records stored in the domain name system server.

17. A data storage and retrieval system according to claim 1, wherein at least two data storage servers having an available connection are selected from the plurality of data storage servers for storage of the data portion in corresponding data storage units based on the loading of each of the plurality of data storage servers.

18. A data storage and retrieval system according to claim 1, wherein the process module is further configured to buffer the data portion in a memory module, and to cause the data portion to be written to the one or more data storage units when the memory module is full.

19. A data storage and retrieval system according to claim 18, wherein the process module is further configured to buffer the data portion in consecutive data blocks based on the storage date of the data portion.

64. A method according to claim 63, further comprising transmitting the data storage information keys to the key manager in response to the writing of the data portion in the at least two data storage units.

65. A method according to claim 63, further comprising generating the offset information using corresponding data storage servers coupled to the at least two data storage units.

66. A method according to claim 63, wherein the data storage information keys further comprise deletion information corresponding to a date on which the data portion written to the at least two data storage units is deleted.

67. A method according to claim 66, further comprising deleting the stored data portion from the at least two data storage units on the date corresponding to the deletion information.

68. A method according to claim 63, wherein receiving incoming data and transmitting the data for storage comprises receiving incoming data and transmitting the data for storage using an electronic mail data processing server.

69. A method according to claim 68, wherein the data comprises electronic mail message data.

70. A method according to claim 63, wherein retrieving the data portion further comprises retrieving the data portion from one or more of the at least two data storage units across a computer network using a web server.

67-1. A method according to claim 63, wherein retrieving the data portion further comprises locating the data portion in one or more of the at least two data storage units using at least one of the data storage information keys, and retrieving the located data portion.