Dynamic bulk-to-brick transformation of data

• US 20070150499A1
• Filed: 10/02/2006
• Published: 06/28/2007
• Est. Priority Date: 02/07/2005
• Status: Active Grant

• Current Assignee: Hewlett Packard Enterprise Development LP (Hewlett-Packard Enterprise Company)
• Original Assignee: Hewlett-Packard Company (HP Inc.)
• Inventors: Ravi, T.M., D'Souza, Roy
• Granted Patent: US 8,799,206 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1458 Management of the backup or restore process
• CPC Class Codes: G06F 11/2028 eliminating a faulty processor or activating a spare
• CPC Class Codes: G06F 11/203 using migration
• CPC Class Codes: G06F 11/2041 with more than one idle spare processing component
• CPC Class Codes: G06F 11/2046 where the redundant components share persistent storage G06F11/2043 takes precedence
• CPC Class Codes: G06F 11/2056 by mirroring
• CPC Class Codes: G06F 11/2071 using a plurality of controllers
• CPC Class Codes: G06F 11/2082 Data synchronisation
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups

Abstract

Multi-dimensional surrogation systems and methods are provided that generate at least one up-to-date data surrogate using information of data and numerous data changes received from at least one data source, including a production system. Embodiments described further perform bulk-to-brick transformation of the up-to-date data surrogate. Brick-level data, or item data, is further indexed and stored in an application-aware manner. The stored item data is easily accessible by many data management applications for integrated item search and recovery functions, audit functions, monitoring and supervision functions, legal discovery functions, compliance functions, archival functions, backup functions, disaster recovery functions, and more. Bulk-to-brick transformation and access of the stored item data occur off of the production system, thus contributing no performance degradation to the production system.

Claims

1. A system comprising:

a near-line server coupled to a first server that includes original data; and

a transformation system coupled to the near-line server and configured to determine a first structure of data, wherein the first structure is a proprietary structure, the transformation system configured to parse items of the data using information of the first structure, the transformation system configured to dynamically transform the data by organizing the items into a second structure that preserves the hierarchy of the first structure.

2. The system of claim 1, wherein the near-line server is configured to provide access to at least a portion of the items in the second structure via a plurality of applications.

3. The system of claim 2, wherein providing access includes providing access using a second protocol that is different from a first protocol used to access the data of the first structure.

4. The system of claim 1, wherein the near-line server is configured to allow a second protocol to access the second structure, wherein a first protocol is used to access the first structure, wherein the first protocol is different than the second protocol.

5. The system of claim 4, wherein the first protocol includes one or more of Messaging Application Programming Interface (MAPI) protocol, Post Office Protocol (POP), and Internet Message Access Protocol (IMAP).

6. The system of claim 1, wherein the second structure is stored on the near-line server, wherein an original version of the data is stored on a first server.

7. The system of claim 1, wherein the transformation system is configured to parse at predetermined intervals.

8. The system of claim 1, wherein the transformation system is configured to parse according to changes in state of the data.

9. The system of claim 1, wherein the transformation system is configured to parse according to availability of the data.

10. The system of claim 1, wherein the transformation system is configured to simultaneously parse a plurality of the items.

11. The system of claim 1, wherein the transformation system is configured to parse by systematically reading information of the data, the information including one or more of tables and the items.

12. The system of claim 11, wherein the tables include one or more of a mailbox table, a folder table, a message table, and an attachment table.

13. The system of claim 11, wherein the items include one or more of a folder, a mailbox, an electronic mail body, an electronic mail header, and an electronic mail attachment.

14. The system of claim 1, wherein the near-line server is configured to generate a shadow copy of the data on the near-line server, wherein an original version of the data is stored on a first server.

15. The system of claim 14, wherein the parsing includes parsing items of the shadow copy.

16. The system of claim 1, wherein the transformation system is configured to decompress one or more of the items.

17. The system of claim 1, wherein the transformation system is configured to organize by transferring a first portion of information of the items into a first file and transferring a second portion of information of the items into a second file.

18. The system of claim 1, wherein the transformation system is configured to organize by indexing the items into the second structure.

19. The system of claim 18, wherein the transformation system is configured to organize by traversing the first structure and locating information of the items.

20. The system of claim 19, wherein the information of the items includes one or more of type and content of the items.

21. The system of claim 19, wherein the transformation system is configured to index by:

generating indices corresponding to the items; and

associating the items with the indices using the information of the items.

22. The system of claim 19, wherein the transformation system is configured to:

compare modification information of each item with modification information of a previous instance of the item; and

cease the traversing at a current node of the first structure when the modification information shows the item is unmodified since the previous instance.

23. The system of claim 22, wherein the modification information includes one or more of modification date and modification time.

24. The system of claim 19, wherein the transformation system is configured to:

determine whether an item is a previously-encountered item that has previously been stored in the second structure;

generate a pointer to the previously-encountered item.

25. The system of claim 19, wherein the transformation system is configured to:

compare modification information of each item with modification information of a previous instance of the item; and

index difference information of the item when the modification information shows the item is modified since the previous instance.

26. The system of claim 25, wherein the difference information includes information of changes between a current version and at least one previously-modified version of the item.

27. The system of claim 19, wherein the transformation system is configured to organize by hashing information of each instance of the item by generating a checksum.

28. The system of claim 27, wherein the transformation system is configured to determine if the item has been altered since the organizing by comparing checksums of different instances of the item in the second structure, the determining performed in response to a request to retrieve the item from the second structure.

29. The system of claim 27, wherein the transformation system is configured to transfer the checksum to a hash table.

30. The system of claim 19, wherein the transformation system is configured to decompose the item into at least one component.

31. The system of claim 30, wherein the item is an electronic mail message, wherein the at least one component includes one or more of a message header, a message body, and a message attachment.

32. The system of claim 18, wherein the transformation system is configured to index by transferring the items into an indexed object repository.

33. The system of claim 18, wherein the transformation system is configured to index by transferring metadata of the items into a metadata repository.

34. The system of claim 1, configured to:

receive a plurality of delta data at the near-line server, wherein the delta data describes incremental changes to the data of a first server;

generate an updated version of the data at the near-line server, wherein generating the updated version includes applying the delta data to a copy of the data on the near-line server; and

controlling the applying using modified information of a component of the near-line server.

35. The system of claim 34, wherein the component includes structural metadata of the copy of the data.

36. The system of claim 34, wherein the component includes a log file of the delta data.

37. The system of claim 34, configured to modify the component.

38. The system of claim 37, wherein the component is structural metadata of the copy.

39. The system of claim 37, configured to modify by detecting a first state of the copy, wherein the first state indicates the delta data has been applied to the copy.

40. The system of claim 39, configured to modify by changing the first state to a second state, wherein the second state is a state from which another updated version can be generated by applying additional delta data to the updated version.

41. The system of claim 40, wherein changing the first state to the second state includes modifying the structural metadata of the copy.

42. The system of claim 39, wherein the additional log file is received after generating the updated version.

43. The system of claim 34, configured to update structural metadata of the copy to reference a first unapplied delta data, wherein the first unapplied delta data is a first log file unapplied to the copy.

44. The system of claim 34, configured to apply by invoking an engine of the first server.

45. The system of claim 37, wherein the component is a log file of the delta data.

46. The system of claim 45, configured to apply by invoking an engine of the first server and the terminating includes stalling the engine.

47. The system of claim 34, configured to receive at the near-line server a copy of the data from the first server.

48. The system of claim 47, wherein the copy is one or more of a full copy and an incremental copy.

49. The system of claim 34, configured to transfer the updated version to an indexed object repository.

50. The system of claim 34, configured to generate in response to at least one of an automatic trigger, a timer notification, an event notification, a poll, and a request.

51. The system of claim 50, wherein the automatic trigger includes a trigger automatically initiated in response to at least one pre-specified parameter.

52. The system of claim 51, wherein the automatic trigger includes content of the updated version.

53. The system of claim 50, wherein the timer notification includes notifications corresponding to scheduled events including at least one of maintenance operations, user activities, server activities, and data population operations.

54. The system of claim 50, wherein the event notification includes notifications corresponding to changes to the items of the data.

55. The system of claim 50, wherein the request includes at least one of access attempts and configuration attempts to the data by one or more of the users, servers and applications.

56. The system of claim 34, configured to generate in near real-time and maintain complete integrity and consistency of the data.

57. The system of claim 34, configured to maintain the updated version by generating another updated version by applying at least one set of log files to the updated version, the at least one set of log files received later in time than the plurality of log files.

58. The system of claim 1, wherein the first server includes one or more of local servers, remote servers, data servers, messaging servers, electronic mail servers, instant messaging servers, voice-over Internet Protocol servers, collaboration servers, messaging and collaboration servers, Exchange servers, portals, customer relationship management (CRM) servers, enterprise resource planning (ERP) servers, business-to-business servers, and content management servers.

59. The system of claim 1, wherein the data is Exchange data and the first structure is a B+ tree structure.

60. The system of claim 1, wherein the second structure is a composite data container.

61. The system of claim 1, wherein the hierarchy is preserved using metadata.

62. The system of claim 1, wherein the hierarchy is preserved using a file structure.

63. A system comprising:

a near-line server coupled to a first server that includes original data; and

a transformation system coupled to the near-line server and configured to determine a first structure of data, the transformation system configured to dynamically transform the data into a second structure that preserves the hierarchy of the first structure, the transformation system configured to provide access to the data of the second structure by a plurality of applications via a second protocol different from a first protocol used to access the data of the first structure.

64. The system of claim 63, configured to dynamically transform by parsing items of the data and organizing the items into the second structure.

65. The system of claim 64, configured to parse according to one or more of predetermined intervals, changes in state of the data, and availability of the data.

66. The system of claim 64, configured to parse by systematically reading information of the data, the information including one or more of tables and the items.

67. The system of claim 66, wherein the tables include one or more of a mailbox table, a folder table, a message table, and an attachment table, wherein the items include one or more of a folder, a mailbox, an electronic mail body, an electronic mail header, and an electronic mail attachment.

68. The system of claim 63, wherein the second structure is stored on the near-line server, wherein an original version of the data is stored on the first server.

69. The system of claim 63, comprising generating a shadow copy of the data on the near-line server, wherein an original version of the data is stored on the first server.

70. The system of claim 63, configured to transform by organizing items of the data into the second structure.

71. The system of claim 70, configured to organize by transferring a first portion of information of the items into a first file and transferring a second portion of information of the items into a second file.

72. The system of claim 70, configured to organize by indexing the items into the second structure.

73. The system of claim 70, configured to organize by traversing the first structure and locating information of the items.

74. The system of claim 73, configured to decompose each item of the items into at least one component.

75. The system of claim 74, wherein the item is an electronic mail message, wherein the at least one component includes one or more of a message header, a message body, and a message attachment.

76. A device comprising a transformation system coupled to a near-line server, the transformation system configured to determine a first structure of data, wherein the data is stored on a first server coupled to the near-line server, the transformation system configured to dynamically transform the data into a second structure that preserves the hierarchy of the first structure, the transformation system configured to provide access to the data of the second structure by a plurality of applications via a second protocol different from a first protocol used to access the data of the first structure.