Dynamic bulk-to-brick transformation of data

• US 8,812,433 B2
• Filed: 10/02/2006
• Issued: 08/19/2014
• Est. Priority Date: 02/07/2005
• Status: Expired due to Fees

• Current Assignee: Hewlett Packard Enterprise Development LP (Hewlett-Packard Enterprise Company)
• Original Assignee: Mimosa Systems Incorporated (Iron Mountain Incorporated)
• Inventors: D'Souza, Roy P., Ravi, Thirumalai Muppur
• Primary Examiner(s): Bhatia, Ajay
• Assistant Examiner(s): MINA, FATIMA P
• Application Number:
  US11/541,891
• Publication Number:
  US 20070156792A1
• Time in Patent Office: 2,878 Days
• Field of Search: 707/673, 707/672, 707/656, 707/650, 707/651, 707/602, 707/611, 707/613, 707/615, 707/618, 707/655
• US Class Current: 707/602
• CPC Class Codes: G06F 11/1458 Management of the backup or restore process
• CPC Class Codes: G06F 11/2028 eliminating a faulty processor or activating a spare
• CPC Class Codes: G06F 11/203 using migration
• CPC Class Codes: G06F 11/2041 with more than one idle spare processing component
• CPC Class Codes: G06F 11/2046 where the redundant components share persistent storage G06F11/2043 takes precedence
• CPC Class Codes: G06F 11/2056 by mirroring
• CPC Class Codes: G06F 11/2071 using a plurality of controllers
• CPC Class Codes: G06F 11/2082 Data synchronisation
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups
• CPC Class Codes: G06F 16/21 Design, administration or maintenance of databases
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data

Abstract

Multi-dimensional surrogation systems and methods are provided that generate at least one up-to-date data surrogate using information of data and numerous data changes received from at least one data source, including a production system. Embodiments described further perform bulk-to-brick transformation of the up-to-date data surrogate. Brick-level data, or item data, is further indexed and stored in an application-aware manner. The stored item data is easily accessible by many data management applications for integrated item search and recovery functions, audit functions, monitoring and supervision functions, legal discovery functions, compliance functions, archival functions, backup functions, disaster recovery functions, and more. Bulk-to-brick transformation and access of the stored item data occur off of the production system, thus contributing no performance degradation to the production system.

Claims

1. A method comprising:

determining a first structure of a data surrogate of application data, wherein the first structure is a B+ tree structure, and the data surrogate is generated in part by shadowing the application data at servers providing services associated with the application data;

parsing items of the data surrogate using information of the first structure;

storing the data surrogate using at least one utility server including at least one processor and coupled to the servers providing the services;

dynamically transforming the data surrogate by systematically reading the data surrogate according to the first structure using a raw read method and using the first structure to organize the items into a second, different structure that preserves a hierarchy of the first structure, wherein the second structure is a directory structure; and

extracting metadata of at least one of the items, the first structure, the data surrogate and the second structure during the parsing and the transforming to generate a searchable component comprising information of the parsed and the transformed data surrogate, the searchable component providing a plurality of applications access to the information of the parsed and transformed data surrogate using the at least one utility server; and

providing access to at least a portion of the items in the second structure using a second protocol that is different from a first protocol used to access the data surrogate according to the first structure.

2. The method of claim 1, wherein providing the access to at least the portion of the items in the second structure is via the plurality of applications.

3. The method of claim 1, wherein the first protocol includes one or more of Messaging Application Programming Interface (MAPI) protocol, Post Office Protocol (POP), and Internet Message Access Protocol (IMAP).

4. The method of claim 1, wherein the second structure is stored on a second server, wherein an original version of the application data is stored on a first server.

5. The method of claim 1, wherein the parsing is performed at predetermined intervals.

6. The method of claim 1, wherein the parsing is performed according to changes in state of the data surrogate.

7. The method of claim 1, wherein the parsing is performed according to availability of the data surrogate.

8. The method of claim 1, wherein the parsing includes parallel parsing of a plurality of the items.

9. The method of claim 1, wherein the parsing includes systematically reading information of the data surrogate, the information including one or more of tables and the items.

10. The method of claim 9, wherein the tables include one or more of a mailbox table, a folder table, a message table, and an attachment table.

11. The method of claim 9, wherein the items include one or more of a folder, a mailbox, an electronic mail body, an electronic mail header, and an electronic mail attachment.

12. The method of claim 1, wherein the data surrogate is on a second server, and wherein an original version of the application data is stored on a first server.

13. The method of claim 1, further comprising decompressing one or more of the items.

14. The method of claim 1, wherein the organizing includes transferring a first portion of information of the items into a first file and transferring a second portion of the information of the items into a second file.

15. The method of claim 1, wherein the organizing includes indexing the items into the second structure.

16. The method of claim 15, wherein the organizing includes traversing the first structure and locating information of the items.

17. The method of claim 16, wherein the information of the items includes one or more of type and content of the items.

18. The method of claim 16, wherein the indexing comprises:

generating indices corresponding to the items; and

associating the items with the indices using the information of the items.

19. The method of claim 16, comprising:

comparing modification information of each item with modification information of a previous instance of the item; and

ceasing the traversing at a current node of the first structure when the modification information shows the item is unmodified since the previous instance.

20. The method of claim 19, wherein the modification information includes one or more of modification date and modification time.

21. The method of claim 16, comprising:

determining whether an item is a previously-encountered item that has previously been stored in the second structure;

generating a pointer to the previously-encountered item.

22. The method of claim 16, comprising:

comparing modification information of each item with modification information of a previous instance of the item; and

indexing difference information of the item when the modification information shows the item is modified since the previous instance.

23. The method of claim 22, wherein the difference information includes information of changes between a current version and at least one previously-modified version of the item.

24. The method of claim 16, wherein the organizing comprises hashing information of each instance of a given item by generating a checksum.

25. The method of claim 24, further comprising determining if the given item has been altered since the organizing by comparing checksums of different instances of the given item in the second structure, the determining performed in response to a request to retrieve the given item from the second structure.

26. The method of claim 24, further comprising transferring the checksum to a hash table.

27. The method of claim 16, further comprising decomposing a given item into at least one component.

28. The method of claim 27, wherein the given item is an electronic mail message, wherein the at least one component includes one or more of a message header, a message body, and a message attachment.

29. The method of claim 15, wherein the indexing includes transferring the items into an indexed object repository.

30. The method of claim 15, wherein the indexing includes transferring metadata of the items into a metadata repository.

31. The method of claim 1, further comprising:

receiving a plurality of delta data at a second server, wherein the delta data describes incremental changes to corresponding data of a first server;

generating an updated version of the corresponding data at the second server, wherein generating the updated version includes applying the delta data to a copy of the corresponding data on the second server; and

controlling the applying using modified information of a component of the second server.

32. The method of claim 31, wherein the component includes structural metadata of the copy of the corresponding data.

33. The method of claim 31, wherein the component includes a log file of the delta data.

34. The method of claim 31, comprising modifying the component.

35. The method of claim 34, wherein the component is structural metadata of the copy.

36. The method of claim 34, wherein the modifying comprises detecting a first state of the copy, wherein the first state indicates the delta data has been applied to the copy.

37. The method of claim 36, wherein the modifying comprises changing the first state to a second state, wherein the second state is a state from which another updated version can be generated by applying additional delta data to the updated version.

38. The method of claim 37, wherein changing the first state to the second state includes modifying structural metadata of the copy.

39. The method of claim 36, wherein a log file is received after generating the updated version.

40. The method of claim 31, further comprising updating structural metadata of the copy to reference a first unapplied delta data, wherein the first unapplied delta data is a first log file unapplied to the copy.

41. The method of claim 31, wherein the applying includes invoking an engine of the first server.

42. The method of claim 34, wherein the component is a log file of the delta data.

43. The method of claim 31, further comprising receiving at the second server a copy of the corresponding data from the first server.

44. The method of claim 43, wherein the copy is one or more of a full copy and an incremental copy.

45. The method of claim 31, further comprising transferring the updated version to an indexed object repository.

46. The method of claim 31, wherein the generating is in response to at least one of an automatic trigger, a timer notification, an event notification, a poll, and a request.

47. The method of claim 46, wherein the automatic trigger includes a trigger automatically initiated in response to at least one pre-specified parameter.

48. The method of claim 47, wherein the automatic trigger includes content of the updated version.

49. The method of claim 46, wherein the timer notification includes notifications corresponding to scheduled events including at least one of maintenance operations, user activities, server activities, and data population operations.

50. The method of claim 46, wherein the event notification includes notifications corresponding to changes to the items.

51. The method of claim 46, wherein the request includes at least one of access attempts and configuration attempts to the corresponding data by one or more of users, servers and applications.

52. The method of claim 31, wherein the second server includes a near-line server.

53. The method of claim 31, wherein the generating is in near real-time and maintains complete integrity and consistency of the corresponding data.

54. The method of claim 31, comprising maintaining the updated version, the maintaining including generating another updated version by applying at least one set of log files to the updated version.

55. The method of claim 1, wherein the first server includes one or more of local servers, remote servers, data servers, messaging servers, electronic mail servers, instant messaging servers, voice-over Internet Protocol servers, collaboration servers, messaging and collaboration servers, Exchange servers, portals, customer relationship management (CRM) servers, enterprise resource planning (ERP) servers, business-to-business servers, and content management servers.

56. The method of claim 1, wherein the hierarchy is preserved using metadata.

57. The method of claim 1, wherein the hierarchy is preserved using a file structure.

58. A method comprising:

determining a first structure of a data surrogate of application data, wherein the first structure is a B+ tree structure, and the data surrogate is provided in part by shadowing the application data at servers providing services associated with the application data;

storing the data surrogate using at least one utility server including at least one processor and coupled to the servers providing the services;

dynamically transforming the data surrogate by systematically reading the first structure of the data surrogate using a raw read method and using the first structure to organize the data surrogate into a second, different structure that preserves a hierarchy of the first structure, wherein the second structure is a directory structure;

extracting metadata of at least one of the first structure, the data surrogate and the second structure during the transforming to generate a searchable component comprising information of the transformed data surrogate, the searchable component providing a plurality of applications access to the information of the transformed data surrogate using the at least one utility server; and

providing access to at least a portion of data in the second structure using a second protocol that is different from a first protocol used to access the data surrogate according to the first structure.

59. The method of claim 58, wherein the dynamically transforming includes parsing items of the data surrogate and organizing the items into the second structure.

60. The method of claim 59, wherein the parsing is performed according to one or more of predetermined intervals, changes in state of the data surrogate, and availability of the data surrogate.

61. The method of claim 59, wherein the parsing includes systematically reading information of the data surrogate, the information including one or more of tables and the items.

62. The method of claim 61, wherein the tables include one or more of a mailbox table, a folder table, a message table, and an attachment table, wherein the items include one or more of a folder, a mailbox, an electronic mail body, an electronic mail header, and an electronic mail attachment.

63. The method of claim 58, wherein the second structure is stored on the at least one utility server, wherein an original version of the application data is stored on a first server.

64. The method of claim 58, wherein the transforming includes organizing items of the data surrogate into the second structure.

65. The method of claim 64, wherein the organizing includes transferring a first portion of information of the items into a first file and transferring a second portion of the information of the items into a second file.

66. The method of claim 64, wherein the organizing includes indexing the items into the second structure.

67. The method of claim 64, wherein the organizing includes traversing the first structure and locating information of the items.

68. The method of claim 67, comprising decomposing each item of the items into at least one component.

69. The method of claim 68, wherein the item is an electronic mail message, wherein the at least one component includes one or more of a message header, a message body, and a message attachment.

70. A non-transitory computer readable medium including executable instructions which, when executed in a processing system, support bulk-to-brick extraction of data by:

determining a first structure of a data surrogate of application data, wherein the first structure is a B+ structure, and the data surrogate is generated in part by shadowing the application data at servers providing services associated with the application data;

parsing items of the data surrogate using information of the first structure;

storing the data surrogate in memory using at least one utility server coupled to the servers providing the services;

dynamically transforming the data surrogate by systematically reading the data surrogate according to the first structure using a raw read method and using the first structure to organize the items into a second, different structure that preserves a hierarchy of the first structure, wherein the second structure is a directory structure;

extracting metadata of at least one of the items, the first structure, the data surrogate and the second structure during the parsing and the transforming to generate a searchable component comprising information of the parsed and the transformed data surrogate, the searchable component providing a plurality of applications access to the information of the parsed and transformed data surrogate using the at least one utility server; and

providing access to at least a portion of the items in the second structure using a second protocol that is different from a first protocol used to access the data surrogate according to the first structure.