Analyzing and detecting anomalies in data records using artificial intelligence

• US 7,689,455 B2
• Filed: 05/16/2005
• Issued: 03/30/2010
• Est. Priority Date: 04/07/2005
• Status: Active Grant

• Current Assignee: Mara-Ison Connective Limited
• Original Assignee: Olista Ltd. (Altruist Educational Services Pvt Ltd.)
• Inventors: Dayagi, Yoram, Fligler, Ariel, Haleva, Avraham, Mashal, Sharon
• Primary Examiner(s): Jarrett; Scott L
• Application Number:
  US11/129,290
• Publication Number:
  US 20060229931A1
• Time in Patent Office: 1,779 Days
• Field of Search: 705/10
• US Class Current: 705/7.38
• CPC Class Codes: G06Q 10/0639 Performance analysis of employees; Performance analysis of enterprise or organisation operations
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising

Abstract

A device, system and method for data monitoring, collection and analysis. A method may include creating one or more context insight records representing an indication of a behavior of a service based on raw footprint data related to use of said service by a population of users, and analyzing said one or more context insight records to detect a dynamically defined anomaly of said service.

Claims

1. A method comprising:

creating one or more context insight records representing an indication of a behavior of a service based on raw footprint data related to use of said service by a population of users;

selecting at least one anomaly criteria dimension;

defining possible values for multiple dimensions of said one or more context insight records in response to a result of a scanning of said one or more context insight records;

analyzing, by an analyzer that is a hardware component, said one or more context insight records, in response to said at least one anomaly criteria dimension and to at least one of said possible values defined;

detecting based on said analyzing a dynamically defined anomaly of said service by generating a first generation of candidate anomalies by segmenting said one or more context insight records along a segmentation dimension of said raw footprint data;

generating a consecutive generation of candidate anomalies by modifying, adding or deleting one or more candidate anomalies of said first generation;

said consecutive generation of candidate anomalies further comprising;

modifying said first generation into said consecutive generation by activating a dimension of said one or more context insight records;

selecting, by the hardware analyzing, at least one of each types of dimensions of said context insight records said dimensions comprising;

criteria, segmenting, and testing dimensions;

wherein said criteria dimensions represent characteristics of at least one context insight record population;

wherein said segmenting dimensions divide said at least one context insight record population into several sub-populations;

wherein said testing dimensions comprise criteria by which different sub-populations of the segmenting dimensions are tested;

wherein the generating of at least one of the generations of candidate anomalies is response to, said criteria, segmenting and said testing dimensions selected by the hardware analyzer;

repeating generating consecutive generations of candidate anomalies until a convergence condition holds true; and

assisting in resolving a problem in the service in response to the dynamically defined anomaly.

2. The method of claim 1, further comprising:

inserting into said one or more context insight records a tag indicating a mapping of an item of said raw footprint data to a related behavior of said service.

3. The method of claim 1, further comprising:

dynamically generating a population description and a test to indicate a logical or functional property of said population based on said one or more context insight records.

4. The method of claim 1, further comprising:

dynamically generating a population description and a test to indicate a quality-of-service associated with said population based on said one or more context insight records.

5. The method of claim 1, further comprising:

calculating a fitness parameter indicating a support level for said at least one candidate anomaly in said one or more context insight records based on an anomaly test.

6. The method of claim 1, wherein modifying comprises:

discarding a candidate anomaly based on an exclusion rule.

7. The method of claim 1, further comprising:

associating between a first candidate anomaly having a first property and a second candidate anomaly having a second property, based on a relation between the first property and the second property.

8. The method of claim 1, further comprising:

grouping a plurality of candidate anomalies into a cluster based on a common property of said plurality of anomalies.

9. The method of claim 1, further comprising:

creating a criteria dimension representing a common characteristic of at least one of said candidate anomalies.

10. The method of claim 1, further comprising:

creating a testing dimension representing a criterion for testing at least one sub-group; and

testing at least one of said sub-groups using said testing dimension.

11. The method of claim 1, further comprising:

monitoring changes in said dynamically defined anomaly over time; and

determining a trend of said anomaly based on said changes.

12. The method of claim 1, further comprising:

after detecting said dynamically defined anomaly, waiting a time period and checking whether said anomaly persists.

13. The method of claim 1, further comprising:

collecting said raw footprint data from said population of users of said service.

14. The method of claim 1, further comprising:

splitting a population associated with said dynamically defined anomaly into a plurality of groups; and

researching said anomaly by evaluating a first group of said plurality of groups in relation to a second group of said plurality of groups.

15. The method according to claim 1 wherein the generating of an alert comprises generating an alert if a service performance breaches a service level agreement.

16. The method according to claim 1 wherein the assisting comprises utilizing an analyzer to fix, correct or resolve functional or logical problems.

17. The method according to claim 1 wherein the assisting comprises providing an end-to-end view about an experience of a certain user of the service by creating aggregated data structures that describe the service'"'"'s functional and logical health and providing a focal point with an end-to-end view of the service.

18. An apparatus comprising:

an analyzer, that is a hardware component and is configured to create one or more context insight records representing an indication of a behavior of a service based on raw footprint data related to use of said service by a population of users, to select at least one anomaly criteria dimension;

to define possible values for multiple dimensions in response to a result of a scanning of the one or more context insight records; and

to analyze, in response to the at least one anomaly criteria dimension and to at least one of the possible values defined, said one or more context insight records to detect a dynamically defined anomaly of said service;

wherein during an analysis of said one or more context insight the analyzer is configured to;

generate a first generation of candidate anomalies by segmenting said one or more context insight records along a segmentation dimension of said raw footprint data;

generate a consecutive generation of candidate anomalies by modifying, adding or deleting one or more candidate anomalies of said first generation;

and repeat generating consecutive generations of candidate anomalies until a convergence condition holds true, wherein the system is configured to assist in resolving a problem in the service in response to the dynamically defined anomaly;

wherein the analyzer is configured to activate a dimension of said one or more context insight records to modify said first generation into said consecutive generation;

wherein the analyzer is configured to generate at least one of the generations of candidate anomalies in response to multiple types of dimensions, that comprise criteria dimensions, segmenting dimensions, and testing dimensions, wherein the criteria dimensions represent characteristics of at least one context insight record population;

wherein the segmenting dimensions divide the at least one context insight record population into several sub-populations;

wherein the testing dimensions comprise criteria by which the different sub-populations of the segmenting dimensions may be tested;

wherein at least one dimension of the criteria dimensions, the segmenting dimensions, and the testing dimensions is a dimension selected by the analyzer.

19. The apparatus of claim 18, wherein said analyzer is configured to insert into said one or more context insight records a tag indicating a mapping of an item of said raw footprint data to a related behavior of said service.

20. The apparatus of claim 18, wherein said analyzer is configured to dynamically generate a population description and a test to indicate a logical or functional property of said population based on said one or more context insight records.

21. The apparatus of claim 18, wherein said analyzer is configured to dynamically generate a population description and a test to indicate a quality-of-service associated with said population based on said one or more context insight records.

22. The apparatus of claim 18, wherein said analyzer is configured to activate a dimension of said one or more context insight records to modify said first generation into said consecutive generation.

23. The apparatus of claim 18, wherein said analyzer is configured to calculate a fitness parameter indicating a support level for said at least one candidate anomaly in said one or more context insight records based on an anomaly test.

24. The apparatus of claim 18, wherein said analyzer is configured to discard a candidate anomaly based on an exclusion rule.

25. The apparatus of claim 18, wherein said analyzer is configured to associate between a first candidate anomaly having a first property and a second candidate anomaly having a second property, based on a relation between the first property and the second property.

26. The apparatus of claim 18, wherein said analyzer is configured to group a plurality of candidate anomalies into a cluster based on a common property of said plurality of anomalies.

27. The apparatus of claim 18, wherein said analyzer is configured to create a criteria dimension representing a common characteristic of at least one of said candidate anomalies.

28. The apparatus of claim 18, wherein said analyzer is configured to create a testing dimension representing a criterion for testing at least one sub-group, and to test at least one of said sub-groups using said testing dimension.

29. The apparatus of claim 18, wherein said analyzer is configured to monitor changes in said dynamically defined anomaly over time, and to determine a trend of said anomaly based on said changes.

30. The apparatus of claim 18, wherein said analyzer is configured to wait a time period after detecting said dynamically defined anomaly, and to check whether said anomaly persists.

31. The apparatus of claim 18, comprising a collector that is configured to collect said raw footprint data from said population of users of said service.

32. The apparatus of claim 18, wherein said analyzer is configured to split a population associated with said dynamically defined anomaly into a plurality of groups, and to research said dynamically defined anomaly by evaluating a first group of said plurality of groups in relation to a second group of said plurality of groups.

33. The apparatus of claim 18, wherein the analyzer is configured to generate at least one of the generations of candidate anomalies by generating at least one of the candidate anomalies by activating new dimensions that were not used for previous generations.

34. The apparatus of claim 18, wherein the analyzer is further configured to construct individual objects in response to at least one population of the context insight records, wherein population characteristics of at least one of the context insight records populations are part of the attributes of at least one of the individual objects;

and to evaluate at least one of the individual objects according to an evaluation function for assisting in resolving a problem in the service in response to the dynamically defined anomaly.

35. The apparatus of claim 18, wherein the analyzer is further configured to construct at least one first context insight record from one or more other context insight records wherein the constructing comprise aggregating data of the one or more other context insight records;

wherein the analyzer is configured to analyze at least one first context insight record to detect the dynamically defined anomaly of said service.

36. A machine-readable medium having stored thereon a set of instructions that, if when executed by a machine, cause the machine to perform a method comprising:

creating one or more context insight records representing an indication of a behavior of a service based on raw footprint data related to use of said service by a population of users;

selecting at least one anomaly criteria dimension;

defining possible values for multiple dimensions in response to a result of a scanning of the one or more context insight records;

analyzing said one or more context insight records, in response to said at least one anomaly criteria dimension and to at least one of said possible values defined;

detecting based on said analyzing a dynamically defined anomaly of said service by generating a first generation of candidate anomalies by segmenting said one or more context insight records along a segmentation dimension of said raw footprint data;

generating a consecutive generation of candidate anomalies by modifying, adding or deleting one or more candidate anomalies of said first generation;

wherein generating the consecutive generation comprises;

activating a dimension of said one or more context insight records to modify said first generation into said consecutive generation;

wherein the generating of at least one of the generations of candidate anomalies is response to multiple types of dimensions, that comprise criteria dimensions, segmenting dimensions, and testing dimensions;

wherein the criteria dimensions represent characteristics of at least one context insight record population;

wherein the segmenting dimensions divide the at least one context insight record population into several sub-populations;

wherein the testing dimensions comprise criteria by which the different sub-populations of the segmenting dimensions may be tested;

wherein at least one dimension of the criteria dimensions, the segmenting dimensions, and the testing dimensions is a dimension selected by the hardware analyzer;

repeating generating consecutive generations of candidate anomalies until a convergence condition holds true; and

assisting in resolving a problem in the service in response to the dynamically defined anomaly.

37. The machine-readable medium of claim 36, wherein the instructions further result in:

inserting into said one or more context insight records a tag indicating a mapping of an item of said raw footprint data to a related behavior of said service.

38. The machine-readable medium of claim 36, wherein the instructions result in:

dynamically generating a population description and a test to indicate a logical or functional property of said population based on said one or more context insight records.

39. The machine-readable medium of claim 36, wherein the instructions further result in:

dynamically generating a population description and a test to indicate a quality-of-service associated with said population based on said one or more context insight records.