Techniques for filtering data from a data stream of a web services application

• US 20050060372A1
• Filed: 08/27/2003
• Published: 03/17/2005
• Est. Priority Date: 08/27/2003
• Status: Abandoned Application

• Current Assignee: Service Integrity, Inc.
• Original Assignee: Service Integrity, Inc.
• Inventors: DeBettencourt, Jason, Lara, Marco, Yamane, Stanley
• Application Number:
  US10/649,160
• Publication Number:
  US 20050060372A1
• Time in Patent Office: Days
• US Class Current: 709/206
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 11/3495 for systems
• CPC Class Codes: G06F 2201/87 Monitoring of transactions
• CPC Class Codes: G06F 2201/875 Monitoring of systems including the internet
• CPC Class Codes: G06Q 20/202 Interconnection or interaction of plural electronic cash registers [ECR] or to host computer, e.g. network details, transfer of information from host to ECR or from ECR to ECR
• CPC Class Codes: H04L 43/028 by filtering

Abstract

Described are techniques used with application monitoring in connection with performing Web Services. Associated with each application performing a service are one or more application servers and associated stream sensors. The stream sensors filter received XML messages in accordance with a set of rules to gather data. The filtered data is gathered at each point and then may be aggregated locally for each transaction by an aggregation service. Data may be gathered from each aggregation service by a global aggregator. Rules are specified initially and may be revised during runtime while gathering data. Rules are evaluated at runtime in accordance with the data stream.

Claims

1. A method for filtering data from a data stream of a web services application comprising:

receiving a message;

evaluating, using data from the message, a current set of one or more rules producing a current set of evaluated rules; and

collecting data associated with the message in accordance with the current set of evaluated rules.

2. The method of claim 1, wherein said evaluating and said gathering are performed in real-time while said web services application is servicing requests, and the method further comprising:

performing, by said web services application, a service in accordance with said message.

3. The method of claim 1, further comprising:

specifying a first set of rules; and

specifying a second set of rules in real-time while said web services application services requests and filters data from said web services application'"'"'s data stream, said second set of rules replacing said first set of rules as said current set of rules.

4. The method of claim 3, wherein said message is a first message and the method further comprising:

associating a first session structure with said first message from said data stream, said first session structure including a representation of said first set of rules; and

associating a second session structure with a second message from said data stream, said second session structure including a representation of said second set of rules, a portion of a processing lifetime of said first message overlapping at least a portion of a processing lifetime of said second message, each of said processing lifetimes being a time period during which an associated message is processed to gather data.

5. The method of claim 4, wherein processing of said first message is complete and the method further comprising:

determining if said first set of rules included in said first session structure is a current set of rules; and

designating said first session structure for reuse in processing another incoming message if said first set of rules is said current set of rules.

6. The method of claim 5, further comprising:

receiving said second message;

determining whether there are any existing session structures designated for reuse; and

selecting one of said existing session structures if there are any available for reuse, and otherwise creating a new instance of a session structure for use in processing said second message.

7. The method of claim 6, further comprising:

using said current set of rules when creating a new instance of a session structure.

8. The method of claim 2, wherein each rule in said first set of rules has a condition portion and an associated action portion, and the method further comprising:

extracting one or more data items using said first message;

evaluating each condition portion in accordance with said one or more data items; and

for each condition portion evaluating to true, performing said associated action portion.

9. The method of claim 8, wherein said one or more data items include at least one of:

message origin information, message header information, web service methods, message parameter, message metadata, and authentication and authorization data.

10. The method of claim 8, wherein said associated action portion includes performing at least one of:

outputting data, executing an application, and sending a message.

11. The method of claim 10, wherein said associated action portion includes executing an application and said application is one of:

a customized application and an e-mail application.

12. The method of claim 10, wherein said associated action portion includes sending a message and said message sent is one of:

an e-mail message, and a message in accordance with a predefined protocol.

13. The method of claim 8, wherein said associated action portion includes performing at least one of:

rejecting a request, and transferring control to an application in accordance with at least one compliance condition.

14. The method of claim 13, wherein said at least one compliance condition includes a predetermined number of unauthorized accesses.

15. The method of claim 10, wherein said associated action portion includes outputting data to at least one of:

an event log, a system log, an error log, and a file included in a file system.

16. The method of claim 8, further comprising:

synchronizing access to a commonly accessed resource in accordance with said associated action portion.

17. The method of claim 8, further comprising:

concurrently accessing a commonly accessed resource in accordance with said associated action portion.

18. The method of claim 7, wherein processing a received message only synchronizes with processing of another received message when accessing a structure that includes one or more session objects for reuse and in connection with a globally accessed resource.

19. The method of claim 18, wherein said globally accessed resource is a resource requiring serial access.

20. The method of claim 3, further comprising:

receiving said first set of rules in one or more messages;

parsing said first set of rules producing a representation;

traversing said representation to construct one or more temporary definitions of objects in accordance with said first set of rules;

attempting to instantiate one or more objects in accordance with said one or more temporary definitions;

determining whether an error has occurred in connection with said attempting to instantiate; and

if no error has occurred in connection with said attempting to instantiate, defining said first set of rules as said current set of rules.

21. The method of claim 20, wherein, if no error has occurred in connection with said attempting to instantiate, discarding any existing session structures designated for reuse in connection with processing subsequently received messages.

22. The method of claim 20, wherein said specifying a second set of rules in real-time includes:

receiving said second set of rules in one or more messages;

parsing said second set of rules producing a representation;

traversing said representation to construct one or more temporary definitions of objects in accordance with said second set of rules;

determining whether there has been a change in one or more object definitions in accordance with said one or more temporary definitions and other current definitions associated with said current set of rules;

if there has been a change in one or more object definitions, performing;

attempting to instantiate one or more objects in accordance with said one or more temporary definitions;

determining whether an error has occurred in connection with said attempting to instantiate; and

if no error has occurred in connection with said attempting to instantiate, defining said second set of rules as said current set of rules.

23. The method of claim 22, wherein there has been a change in a first object definition and at least one other object definition has not changed, and the method further comprising:

attempting to instantiate only said first object definition that has changed; and

reusing an existing definition for said at least one other object definition that has not changed.

24. The method of claim 1, wherein said web services application is one of a plurality of applications providing web services.

25. The method of claim 1, wherein said web services application is associated with an application server and a stream sensor, said stream sensor performing said evaluating.

26. The method of claim 1, further comprising:

using a console to specify said current set of rules.

27. The method of claim 1, further comprising:

using a non-interactive input source to specify said current set of rules.

28. The method of claim 26, further comprising:

selecting at least one condition and at least one data item included in said current set of rules.

29. The method of claim 8, wherein said evaluating each condition portion further comprises:

performing expression processing in accordance with said one or more data items.

30. The method of claim 29, further comprising:

performing format processing in connection with outputting data for an action portion of a rule when a condition portion of said rule evaluates to true.

31. The method of claim 1, wherein each rule of said current set of one or more rules includes a condition portion and an action portion, said action portion being performed if said condition portion evaluates to true.

32. The method of claim 31, wherein said condition portion includes a reference to a user-defined routine.

33. The method of claim 31, wherein said action portion includes outputting data to an output destination, and the method further comprising:

outputting first data to a first output destination executing a first routine in accordance with a first condition; and

outputting second data to a second output destination executing a second routine in accordance with a second condition, said first output destination being different from said second output destination,.

34. The method of claim 33, wherein said first routine outputs first data to a third routine which aggregates said first data and outputs aggregated first data at predetermined time intervals to a third output destination different from said first and second output destinations.

35. The method of claim 1, wherein said message received is one of:

an input message and an output message of said data stream of said web services application.

36. The method of claim 1, wherein a plurality of messages are received and said plurality of messages include at least one input message and at least one output message of said data stream of said web services application.

37. The method of claim 15, wherein said event log is a Windows event log.

38. The method of claim 15, wherein said system log is a UNIX syslog.

39. A system for filtering data comprising:

at least one application providing a service;

at least one application server and stream sensor associated with each of said at least one application, said stream sensor filtering a data stream of said at least one application in accordance with a dynamically evaluated current set of rules producing filtered data, said current set of rules being dynamically evaluated with each received message; and

a context engine associated with each of said at least one application, said context engine including an aggregation service that aggregates said filtered data for said application in accordance with at least one local aggregation condition.

40. The system of claim 39, wherein said filtered data is stored in raw data files used as input to said aggregation service which output summary data.

41. The system of claim 40, wherein the system includes at least one application and a global aggregator gathering output summary data from each aggregation service associated with each of said at least one application.

42. The system of claim 39, wherein said local aggregation condition is aggregating data locally for each transaction.

43. A method for capturing data from a data stream of an application comprising:

receiving a first message from said data stream of said application;

storing first context information including a copy of a first set of one or more rules, said first context information describing a processing state of said first message used during a first processing lifetime of said first message, said first set of one or more rules used in obtaining first data related to said first message, said first set of one or more rules being said current set of rules;

defining a second set of one or more rules at runtime of said application while said application is servicing a request such that any subsequently received messages are processed in accordance with said second set of one or more rules, said second set of one or more rules replacing said first set of one or more rules as said current set;

receiving a second message from said data stream during said first processing lifetime of said first message; and

storing second context information including a copy of said second set of one or more rules, said second context information describing a processing state of said second message used during a second processing lifetime of said second message, said second set of rules used in obtaining second data related to said second message, said first processing lifetime and said second processing lifetime overlapping for at least a portion of both lifetimes such that said first and second messages are concurrently processed.

44. The method of claim 43, further comprising:

gathering data in accordance with said first message and said first set of one or more rules; and

gathering data in accordance with said second message and said second set of one or more rules.

45. The method of claim 44, further comprising:

determining, after processing said first message, whether a first data storage area storing said first context information can be reused by determining whether a set of rules included in said first data storage area is said current set of rules;

determining, after processing said second message, whether a second data storage area storing said second context information can be reused by determining whether a set of rules included in said second data storage area is said current set of rules; and

designating said second data storage area for reuse and not designating said first data storage area for reuse.

46. The method of claim 45, further comprising:

receiving a third message;

determining whether there are any available data storage areas designated for reuse; and

selecting one of said available data storage areas if there are any available and otherwise allocating additional storage for storing third context information for said third message, said third context information including a copy of said current set of rules.

47. A computer program product for filtering data from a data stream of a web services application comprising:

executable code that receives a message;

executable code that evaluates, using data from the message, a current set of one or more rules producing a current set of evaluated rules; and

executable code that collects data associated with the message in accordance with the current set of evaluated rules.

48. The computer program product of claim 47, wherein said executable code that evaluates and collects are performed in real-time while said web services application is servicing requests, and the computer program product further comprising:

executable code that performs a service in accordance with said message.

49. The computer program product of claim 47, further comprising:

executable code that specifies a first set of rules; and

executable code that specifies a second set of rules in real-time while said web services application services requests and filters data from said web services application'"'"'s data streami, said second set of rules replacing said first set of rules as said current set of rules.

50. The computer program product of claim 49, wherein said message is a first message and the computer program product further comprising:

executable code that associates a first session structure with said first message from said data stream, said first session structure including a representation of said first set of rules; and

executable code that associates a second session structure with a second message from said data stream, said second session structure including a representation of said second set of rules, a portion of a processing lifetime of said first message overlapping at least a portion of a processing lifetime of said second message, each of said processing lifetimes being a time period during which an associated message is processed to gather data.

51. The computer program product of claim 50, wherein processing of said first message is complete and the computer program product further comprising:

executable code that determines if said first set of rules included in said first session structure is a current set of rules; and

executable code that designates said first session structure for reuse in processing another incoming message if said first set of rules is said current set of rules.

52. The computer program product of claim 51, further comprising:

executable code that receives said second message;

executable code that determines whether there are any existing session structures designated for reuse; and

executable code that selects one of said existing session structures if there are any available for reuse, and otherwise creates a new instance of a session structure for use in processing said second message.

53. The computer program product of claim 52, further comprising:

executable code that uses said current set of rules when creating a new instance of a session structure.

54. The computer program product of claim 48, wherein each rule in said first set of rules has a condition portion and an associated action portion, and the computer program product further comprising:

executable code that extracts one or more data items using said first message;

executable code that evaluates each condition portion in accordance with said one or more data items; and

executable code that, for each condition portion evaluating to true, performs said associated action portion.

55. The computer program product of claim 54, wherein said one or more data items include at least one of:

message origin information, message header information, web service methods, message parameter, message metadata, and authentication and authorization data.

56. The computer program product of claim 54, wherein said associated action portion includes performing at least one of:

outputting data, executing an application, and sending a message.

57. The computer program product of claim 56, wherein said associated action portion includes executing an application and said application is one of:

a customized application and an e-mail application.

58. The computer program product of claim 56, wherein said associated action portion includes sending a message and said message sent is one of:

an e-mail message, and a message in accordance with a predefined protocol.

59. The computer program product of claim 54, wherein said associated action portion includes performing at least one of:

rejecting a request, and transferring control to an application in accordance with at least one compliance condition.

60. The computer program product of claim 59, wherein said at least one compliance condition includes a predetermined number of unauthorized accesses.

61. The computer program product of claim 56, wherein said associated action portion includes outputting data to at least one of:

an event log, a system log, an error log, and a file included in a file system.

62. The computer program product of claim 54, further comprising:

executable code that synchronizes access to a commonly accessed resource in accordance with said associated action portion.

63. The computer program product of claim 54, further comprising:

executable code that concurrently accesses a commonly accessed resource in accordance with said associated action portion.

64. The computer program product of claim 53, comprising executable code that, when processing a received message, only synchronizes with processing of another received message when accessing a structure that includes one or more session objects for reuse and in connection with a globally accessed resource.

65. The computer program product of claim 64, wherein said globally accessed resource is a resource requiring serial access.

66. The computer program product of claim 49, further comprising:

executable code that receives said first set of rules in one or more messages;

executable code that parses said first set of rules producing a representation;

executable code that traverses said representation to construct one or more temporary definitions of objects in accordance with said first set of rules;

executable code that attempts to instantiate one or more objects in accordance with said one or more temporary definitions;

executable code that determines whether an error has occurred in connection with said attempting to instantiate; and

executable code that, if no error has occurred in connection with attempting to instantiate, defines said first set of rules as said current set of rules.

67. The computer program product of claim 66, further comprising:

executable code that, if no error has occurred in connection with attempting to instantiate, discards any existing session structures designated for reuse in connection with processing subsequently received messages.

68. The computer program product of claim 66, wherein said executable code that specifies a second set of rules in real-time includes:

executable code that receives said second set of rules in one or more messages;

executable code that parses said second set of rules producing a representation;

executable code that traverses said representation to construct one or more temporary definitions of objects in accordance with said second set of rules;

executable code that determines whether there has been a change in one or more object definitions in accordance with said one or more temporary definitions and other current definitions associated with said current set of rules;

executable code that if there has been a change in one or more object definitions, performs;

attempting to instantiate one or more objects in accordance with said one or more temporary definitions;

determining whether an error has occurred in connection with said attempting to instantiate; and

if no error has occurred in connection with said attempting to instantiate, defining said second set of rules as said current set of rules.

69. The computer program product of claim 68, wherein there has been a change in a first object definition and at least one other object definition has not changed, and the computer program product further comprising:

executable code that attempts to instantiate only said first object definition that has changed; and

executable code that reuses an existing definition for said at least one other object definition that has not changed.

70. The computer program product of claim 47, wherein said web services application is one of a plurality of applications providing web services.

71. The computer program product of claim 47, wherein said web services application is associated with an application server and a stream sensor, said stream sensor performing including said executable code that evaluates.

72. The computer program product of claim 47, further comprising:

executable code that uses a console to specify said current set of rules.

73. The computer program product of claim 47, further comprising:

executable code that uses a non-interactive input source to specify said current set of rules.

74. The computer program product of claim 72, further comprising:

executable code that selects at least one condition and at least one data item included in said current set of rules.

75. The computer program product of claim 54, wherein said executable code that evaluates each condition portion further comprises:

executable code that performs expression processing in accordance with said one or more data items.

76. The computer program product of claim 75, further comprising:

executable code that performs format processing in connection with outputting data for an action portion of a rule when a condition portion of said rule evaluates to true.

77. The computer program product of claim 47, wherein each rule of said current set of one or more rules includes a condition portion and an action portion, said action portion being performed if said condition portion evaluates to true.

78. The computer program product of claim 77, wherein said condition portion includes a reference to a user-defined routine.

79. The computer program product of claim 77, wherein said action portion includes outputting data to an output destination, and the computer program product further comprising:

executable code that outputs first data to a first output destination executing a first routine in accordance with a first condition; and

executable code that outputs second data to a second output destination executing a second routine in accordance with a second condition, said first output destination being different from said second output destination.

80. The computer program product of claim 79, wherein said first routine outputs first data to a third routine which aggregates said first data and outputs aggregated first data at predetermined time intervals to a third output destination different from said first and second output destinations.

81. The computer program product of claim 47, wherein said message received is one of:

an input message and an output message of said data stream of said web services application.

82. The computer program product of claim 47, wherein a plurality of messages are received and said plurality of messages include at least one input message and at least one output message of said data stream of said web services application.

83. The computer program product of claim 61, wherein said event log is a Windows event log.

84. The computer program product of claim 61, wherein said system log is a UNIX syslog.

85. A computer program product for capturing data from a data stream of an application comprising:

executable code that receives a first message from said data stream of said application;

executable code that stores first context information including a copy of a first set of one or more rules, said first context information describing a processing state of said first message used during a first processing lifetime of said first message, said first set of one or more rules used in obtaining first data related to said first message, said first set of one or more rules being said current set of rules;

executable code that defines a second set of one or more rules at runtime of said application while said application is servicing a request such that any subsequently received messages are processed in accordance with said second set of one or more rules, said second set of one or more rules replacing said first set of one or more rules as said current set;

executable code that receives a second message from said data stream during said first processing lifetime of said first message; and

executable code that stores second context information including a copy of said second set of one or more rules, said second context information describing a processing state of said second message used during a second processing lifetime of said second message, said second set of rules used in obtaining second data related to said second message, said first processing lifetime and said second processing lifetime overlapping for at least a portion of both lifetimes such that said first and second messages are concurrently processed.

86. The computer program product of claim 85, further comprising:

executable code that gathers data in accordance with said first message and said first set of one or more rules; and

executable code that gathers data in accordance with said second message and said second set of one or more rules.

87. The computer program product of claim 86, further comprising:

executable code that determines, after processing said first message, whether a first data storage area storing said first context information can be reused by determining whether a set of rules included in said first data storage area is said current set of rules;

executable code that determines, after processing said second message, whether a second data storage area storing said second context information can be reused by determining whether a set of rules included in said second data storage area is said current set of rules; and

executable code that designates said second data storage area for reuse and not designating said first data storage area for reuse.

88. The computer program product of claim 87, further comprising:

executable code that receives a third message;

executable code that determines whether there are any available data storage areas designated for reuse; and

executable code that selects one of said available data storage areas if there are any available and otherwise allocating additional storage for storing third context information for said third message, said third context information including a copy of said current set of rules.