System and method for transformation and analysis of messaging data

• US 7,231,403 B1
• Filed: 11/15/2002
• Issued: 06/12/2007
• Est. Priority Date: 11/15/2002
• Status: Expired due to Term

• Current Assignee: MessageOne, Inc. (Dell Technologies Inc.)
• Original Assignee: MessageOne, Inc. (Dell Technologies Inc.)
• Inventors: Nonemacher, Michael N., Howitt, Mathew A., Goldfein, Jocelyn E.
• Primary Examiner(s): LE, UYEN T
• Application Number:
  US10/295,427
• Time in Patent Office: 1,670 Days
• Field of Search: 707/104.1, 709/201, 709/206, 709/207, 709/223, 709/224, 714/47, 714/912, 379/1.03
• US Class Current: 1/1
• CPC Class Codes: H04L 41/00 Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
• CPC Class Codes: H04L 43/00 Arrangements for monitoring or testing data switching networks
• CPC Class Codes: H04L 43/12 Network monitoring probes
• CPC Class Codes: H04L 51/00 User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
• CPC Class Codes: H04L 51/222 using geographical location information, e.g. messages transmitted or received in proximity of a certain spot or area
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99948 Application of database or data structure, e.g. distributed, multimedia, or image

Abstract

Systems and methods which provide translation and/or transformation of messaging data, such as that collected by a distributed management method, for collection, consolidation, processing, and/or analysis are shown. Preferred embodiments of the present invention provide data translation schema to manipulate raw data stored in a first database or databases, such as a staging database, to store useful information in appropriate locations of a second database or databases, such as a data warehouse. Preferably, for each data source monitored in a messaging environment, collected raw messaging data is parsed into appropriate positions of a staging database. Thereafter, data transforms of the present invention preferably manipulate and/or process data of the staging database and provide useful information for storage in a data warehouse.

Claims

1. A method for processing messaging data, said method comprising:

collecting, by a data collector system, message server message environment metrics from a plurality of message servers;

processing, by said data collector system, said message environment metrics provided from different ones of said message servers to provide data useful in deriving desired messaging statistics; and

outputting, by said data collector system, said data useful in deriving desired messaging statistics to a data transform system for aggregation with data from a plurality of data collector systems to derive said desired messaging statistics.

2. The method of claim 1, further comprising:

deriving, by said data transform system, said desired messaging statistics using data from said plurality of data collector systems and predefined analysis rules.

3. The method of claim 2, further comprising:

generating a report including said desired messaging statistics.

4. The method of claim 2, further comprising:

automatically instigating said deriving according to a predetermined schedule.

5. The method of claim 2, further comprising:

instigating said deriving upon initiation by an operator.

6. The method of claim 2, wherein said deriving comprises:

enforcing counts with respect to operation of particular attributes of a messaging environment in accordance with said predefined analysis rules.

7. The method of claim 2, further comprising:

configuring, by an operator, said predefined analysis rules.

8. The method of claim 2, wherein said predefined analysis rules include a first rule for application with respect to data associated with a first entity and a second rule for application with respect to data associated with a second entity.

9. The method of claim 2, wherein said predefined analysis rules include a first rule for application with respect to data associated with a first organization of an entity and a second rule for application with respect to data associated with a second organization of said entity.

10. The method of claim 1, further comprising:

aggregating, by said data transform system, said desired messaging statistics.

11. The method of claim 10, wherein said aggregating includes rolling up information to provide statistical information having less detail.

12. The method of claim 1, further comprising:

storing, by said data collector system, at least a portion of said data useful in deriving desired messaging statistics in a data store for said deriving said desired messaging statistics.

13. The method of claim 1, wherein said message environment metrics include data components provided from different ones of said message servers having a same information content, and wherein said processing comprises identifying said data components having said same information content.

14. The method of claim 1, wherein said message environment metrics includes data components provided from a same one of said message servers having a same information content, and wherein said processing comprises identifying said data components having said same information content.

15. The method of claim 1, wherein said processing utilizes identifiers associated with data components of said message environment metrics selected from the group consisting of time information associated with said data components, a server identity associated with said data components, a mailbox identity associated with said data components, and a message ID associated with said data components.

16. The method of claim 15, wherein said processing utilizes a combination of two said identifiers to correlate different data components of said message environment metrics.

17. The method of claim 1, wherein message servers of said plurality of message servers are associated with different entities.

18. The method of claim 1, wherein message servers of said plurality of message servers provide different messaging services.

19. The method of claim 1, wherein said plurality of message servers include mail servers and wireless servers.

20. The method of claim 1, further comprising:

identifying late data for special processing.

21. A system for processing messaging data, said system comprising:

a processor based system operable, at least in part, to provide a data collector collecting raw data from a plurality of different messaging servers, correlating data components of said collected data provided from different ones of said messaging servers to parse said data, and storing at least a portion of said parsed data in a parsed data store; and

a processor based system operable, at least in part, to provide a data analyzer accessing said data store, utilizing predefined analysis rules to provide statistical information from data of said data store, and storing at least a portion of said statistical information in a data warehouse.

22. The system of claim 21, wherein said processor based data collector comprises:

a data collection system providing initial collection of said raw data, wherein said data collection system includes at least one probe for interfacing with at least one messaging server of said messaging servers for collection of said raw data; and

a data collection server in communication with said data collection system, wherein said data collection system is disposed remotely with respect to said data collection server.

23. The system of claim 22, wherein said correlating data components of said collected data provided from different ones of said messaging servers and said storing at least a portion of said parsed data in a parsed data store is provided by said data collection server.

24. The system of claim 23, wherein said data collection system operates to filter said raw data prior to passing data to said data collection server.

25. The system of claim 23, wherein said data collection system operates to pass said raw data to said data collection server.

26. The system of claim 21, wherein said raw data is data which has not been further processed after collection from said plurality of different messaging servers.

27. The system of claim 21, wherein said raw data includes data components provided from different ones of said data sources having a same information content.

28. The system of claim 27, wherein correlation by said processor based system data collector identifies said data components having said same information content and stores one said data component in said data store.

29. The system of claim 21, wherein said raw data includes data components provided from a same one of said data sources having a same information content.

30. The system of claim 29, wherein correlation by said processor based system data collector identifies said data components having said same information content and stores one said data component in said data store.

31. The system of claim 21, wherein correlation by said processor based system data collector utilizes identifiers associated with data components of said raw data.

32. The system of claim 31, wherein said identifiers include time information associated with said data components.

33. The system of claim 32, wherein said processor based system data collector utilizes a selected window of time for said correlation.

34. The system of claim 33, wherein said selected window of time is user selectable.

35. The system of claim 31, wherein said identifiers include a server identity associated with said data components.

36. The system of claim 31, wherein said identifiers include a mailbox identity associated with said data components.

37. The system of claim 36, wherein said processor based system data collector utilizes a canonical reference to a mailbox for said correlation.

38. The system of claim 31, wherein said identifiers include a message ID associated with said data components.

39. The system of claim 31, wherein correlation by said processor based system data collector utilizes a combination of two said identifiers associated with data components of said raw data.

40. The system of claim 21, wherein said messaging servers include messaging servers of multiple messaging environments.

41. The system of claim 40, wherein said multiple messaging environments are associated with different entities.

42. The system of claim 41, wherein said system is deployed in a service bureau configuration.

43. The system of claim 21, wherein said messaging servers include servers providing different messaging services.

44. The system of claim 21, wherein said messaging servers include mail servers and wireless servers.

45. The system of claim 21, wherein said processor based system data collector is further adapted to identify late data for special processing by said process based system data analyzer.

46. The system of claim 21, wherein operation of said processor based system data collector is provided continuously as data is accepted and operation of said processor based system data analyzer is provided periodically.

47. The system of claim 21, wherein said predefined analysis rules enforce counts with respect to operation of particular attributes of a messaging environment.

48. The system of claim 21, wherein said predefined analysis rules include different rules for enforcement with respect to particular data.

49. The system of claim 48, wherein said different rules include a first set of rules for application with respect to data associated with a first entity and a second set of rules for application with respect to data associated with a second entity.

50. The system of claim 48, wherein said different rules include a first set of rules for application with respect to data associated with a first organization of an entity and a second set of rules for application with respect to data associated with a second organization of said entity.

51. The system of claim 21, wherein said processor based system data analyzer is further adapted to aggregate statistical information stored in said data warehouse.

52. The system of claim 51, wherein said aggregation includes rolling up information to provide statistical information having less detail.

53. The system of claim 51, wherein said processor based system data analyzer is further adapted to identify late data and to adjust aggregated statistical information stored in said data warehouse accordingly.