Real-time adaptive data mining system and method

• US 20030023593A1
• Filed: 05/22/2002
• Published: 01/30/2003
• Est. Priority Date: 05/11/2000
• Status: Abandoned Application

• Current Assignee: JP Morgan Chase Bank, N.A. (JP Morgan Chase & Co)
• Original Assignee: JP Morgan Chase Bank, N.A. (JP Morgan Chase & Co)
• Inventors: Schmidt, Richard
• Application Number:
  US10/151,814
• Publication Number:
  US 20030023593A1
• Time in Patent Office: Days
• US Class Current: 707/6
• CPC Class Codes: G06F 16/2465 Query processing support for facilitating data mining operations in structured databases
• CPC Class Codes: G06F 18/24765 Rule-based classification
• CPC Class Codes: G06F 2216/03 Data mining
• CPC Class Codes: G06N 5/025 Extracting rules from data

Abstract

A set of rules generated from data examples describing a situation can be created and updated according to new information received in real time. The set of rules provides a description of a domain of knowledge based on logical conclusions that can be drawn from the data. The rules are mutually exclusive, and are reduced to a minimized set that completely represents the data to which the rules are exposed. The outcome of the rules is adoptive to changing data and sensitive to shifts in the conclusions that can be drawn about the data. More recently received data is more heavily weighted than prior data to permit a rapid response to information shifts.

Claims

1. A method for formulating a set of rules, comprising:

a) receiving data related to a situation, said data comprising a received attribute value pattern and an associated conclusion;

b) comparing said received attribute value pattern to all other attribute value patterns in said set of rules that are associated with conclusions different than that of said received data to identify matched attribute values between said received attribute pattern and said compared attribute patterns;

c) marking said matched attribute values as irrelevant in said received attribute pattern and said compared attribute patterns; and

repeating a) through c) to form and update said set of rules, with each rule comprising a relevant attribute pattern and an associated rule conclusion.

2. The method for formulating a set of rules according to claim 1, further comprising initially marking said received attribute values as relevant.

3. The method for formulating a set of rules according to claim 1, further comprising placing said received attribute value pattern and associated conclusion in said set of rules if said received attribute value pattern is not already in said set of rules.

4. The method for formulating a set of rules according to claim 1, further comprising:

designating as a first list all attributes of said received attribute pattern prior to b);

making copies of said first list, and any subsequent lists, of said received attribute pattern and said compared attribute patterns prior to b);

replacing after c) all lists in said received attribute pattern and said compared attribute pattern with their respective copies except for the lowest numbered designated list containing at least one said attribute marked relevant; and

when no list in said received attribute pattern and/or said compared attribute pattern contains at least one attribute marked relevant, designating as a second (third, fourth, . . . as appropriate) list all attributes in said received attribute pattern and/or said compared attribute pattern whose values do not match, marking said values as relevant, and replacing all other lists in said received attribute pattern and/or said compared attribute pattern with their copies.

5. The method for formulating a set of rules according to claim 1, further comprising removing redundant rules from said set of rules.

6. The method for formulating a set of rules according to claim 1, wherein said received data can be selectively discarded and further data can be received to thereby increase the relative frequency of occurrence of an infrequently occurring conclusion in said received data.

7. The method for formulating a set of rules according to claim 1, further comprising initializing said set of rules with an initial received attribute pattern and an associated conclusion.

8. The method for formulating a set of rules according to claim 1, further comprising:

determining if said received attribute value pattern matches any other attribute value pattern in said set of rules;

incrementing a conclusion count in said compared rule having a matching attribute value pattern; and

creating a new rule in said set of rules from said received attribute value pattern and said associated conclusion if said received attribute value pattern matches none of said attribute value patterns in said set of rules.

9. The method for formulating a set of rules according to claim 8, wherein said incremented conclusion count in said compared rule is related to said received associated conclusion.

10. The method for formulating a set of rules according to claim 8, further comprising designating a conclusion of a rule as a predominant conclusion for said rule based on said conclusion count.

11. The method for formulating a set of rules according to claim 1, further comprising designating a conclusion of a rule as a predominant conclusion for said rule based on relevant knowledge about said situation.

12. The method for formulating a set of rules according to claim 1, further comprising expanding each rule in said set of rules into a canonical form.

13. The method for formulating a set of rules according to claim 5, further comprising expanding each rule in said set of rules into a canonical form.

14. The method for formulating a set of rules according to claim 8, further comprising:

setting a maximum conclusion count value;

preventing said conclusion count from being incremented to a value greater than said maximum conclusion count value; and

decrementing all other conclusion counts greater than zero if said conclusion count has a value equivalent to said maximum conclusion count value.

15. The method for formulating a set of rules according to claim 14, further comprising designating a conclusion of a rule as a predominant conclusion for said rule based on said conclusion count.

16. The method for formulating a set of rules according to claim 15, further comprising:

determining if said new rule includes a conclusion different from predominant conclusions found in any other rule; and

processing said new rule with each rule in said any of other rules having a different predominant conclusion according to b) and c).

17. The method for formulating a set of rules according to claim 15, further comprising:

determining if there is a change in a predominant conclusion for said compared rule as a result of changes in a conclusion count for said compared rule; and

if said predominant conclusion changes in said compared rule as a result of changes in its conclusion count;

marking all attribute values relevant in rules having a predominant conclusion equal to the conclusion to which said compared rule changed; and

processing according to b) and c) all rules having a predominant conclusion equal to the conclusion to which said compared rule changed.

18. The method for formulating a set of rules according to claim 1, further comprising:

designating a plurality of domains, each containing a set of rules; and

applying a) through c) to each set of rules in each domain.

19. The method for formulating a set of rules according to claim 18, wherein at least one domain can be completely defined upon receipt of sufficient data.

20. The method for formulating a set of rules according to claim 18, wherein application of a) through c) to each set of rules in each domain takes place simultaneously.

21. The method for formulating a set of rules according to claim 18, further comprising developing a set of domain selection rules for determining a subset of domains for which said received data is applicable.

22. The method for formulating a set of rules according to claim 21, further comprising applying a) through c) to said set of domain selection rules to produce a complete and consistent set of domain selection rules.

23. The method for formulating a set of rules according to claim 18, wherein designating said domains is achieved through selection of attributes represented by said domains.

24. The method for formulating a set of rules according to claim 21, wherein:

at least one domain selection rule in said set of domain selection rules has an attribute corresponding to an attribute value in said received attribute pattern; and

said method further comprises selecting a domain for which said received data is applicable based on said at least one domain selection rule having said corresponding attribute.

25. The method for formulating a set of rules according to claim 1, further comprising at least one multi-valued attribute in said received data having more than two possible values.

26. The method for formulating a set of rules according to claim 1, further comprising:

expanding each rule in said set of rules into a canonical form to form a set of canonical rules; and

removing redundant canonical rules from said set of canonical rules.

27. The method for formulating a set of rules according to claim 25, further comprising:

expanding each rule in said set of rules into a canonical form to form a set of canonical rules; and

removing redundant canonical rules from said set of canonical rules.

28. The method for formulating a set of rules according to claim 27, wherein canonical rules that match except for differing values of one said multi-valued attribute can be combined by grouping said differing values of said multi-valued attribute within a single rule.

29. A system for formulating a set of rules, comprising:

a data input for receiving data;

said data comprising sequential datagroups each comprising an attribute value pattern and an associated conclusion related to a situation;

a processor operable to process said data to form said set of rules comprising a rule attribute value pattern and a predominant conclusion;

said processor being further operable to apply each input datagroup to said set of rules to thereby incorporate information related to said situation into said set of rules;

said processor being further operable to identify attribute values from each rule attribute value pattern that are irrelevant to said associated predominant conclusion; and

said processor is further operable to remove redundant rules from said set of rules to provide a complete and consistent minimal rule set.

30. The system for formulating a set of rules according to claim 29, wherein said processor is further operable to selectively discard some of said datagroups to thereby increase a relative frequency of occurrence of an infrequently occurring situation.

31. The system for formulating a set of rules according to claim 29, wherein said processor is operable to select a predominant conclusion for each rule based on a specified criteria.

32. The system for formulating a set of rules according to claim 31, wherein said specified criteria is provided by an expert.

33. The system for formulating a set of rules according to claim 29, wherein said processor is operable to expand said set of rules into a canonical form before said redundant ones of said rules are removed.

34. The system for formulating a set of rules according to claim 29, further comprising:

a comparator module coupled to said processor and operable to provide a comparison between a selected rule attribute value pattern and all other rule attribute value patterns having predominant conclusions different than that of said selected rule attribute value pattern; and

said processor is further operable to identify said attribute values that match as irrelevant in said selected rule attribute pattern and said compared rule attribute patterns.

35. A computer readable memory storing a program code executable to form a set of rules, said program code comprising:

a) a first code section executable to receive data related to a situation, said data comprising a received attribute value pattern and an associated conclusion, said values initially identified as relevant;

b) a second code section executable to compare said received attribute value pattern to all other attribute value patterns in said set of rules that are associated with conclusions different than that of said received data to match attribute values between said received attribute pattern and said compared attribute patterns;

c) a third code section executable to identify said attribute values that match as irrelevant in said received attribute pattern and said compared attribute patterns; and

d) a fourth code section executable to branch to a) thereby permitting repetition of a) through c) to form and update said set of rules, with each rule comprising a relevant attribute pattern and an associated rule conclusion.

36. The program code according to claim 35, further comprising a fifth code section executable to remove redundant rules from said set of rules.

37. A method for forming a set of rules, comprising:

finding all non-redundant fact patterns in a stream of data related to a corresponding set of situations;

identifying at least one attribute in each fact pattern that contributes to a respective conclusion associated with said fact pattern; and

forming said set of rules using said identified attributes and said respective associated conclusions.

38. The method for forming a set of rules according to claim 37, further comprising removing redundancies within said set of rules.

39. The method for forming a set of rules according to claim 37, wherein said stream of data is sampled to have a first conclusion in a reduced ratio with respect to a second conclusion.

40. The method for forming a set of rules according to claim 37, wherein:

each said fact pattern is associated with a group of conclusions; and

said method further comprises selecting a single conclusion from each of said groups as said respective associated conclusion.

41. The method for forming a set of rules according to claim 38, wherein said rules are expanded into a canonical form prior to removing redundancies.

42. A carrier medium containing a program code executable to form a set of rules, said program code comprising:

a first code section executable to receive data related to a situation, said data comprising a received attribute value pattern and an associated conclusion, said values initially identified as relevant;

a second code section executable to compare said received attribute value pattern to all other attribute value patterns in said set of rules that are associated with conclusions different than that of said received data to match attribute values between said received attribute pattern and said compared attribute patterns;

a third code section executable to identify said attribute values that match as irrelevant in said received attribute pattern and said compared attribute patterns; and

a fourth code section executable to cause repeated execution of said first through said third code sections to form and update said set of rules, with each rule comprising a relevant attribute pattern and an associated rule conclusion.

43. A processor operable to execute a program code from a storage memory to form a set of rules, said program code comprising:

a first code section executable to receive data related to a situation, said data comprising a received attribute value pattern and an associated conclusion, said values initially identified as relevant;

a second code section executable to compare said received attribute value pattern to all other attribute value patterns in said set of rules that are associated with conclusions different than that of said received data to match attribute values between said received attribute pattern and said compared attribute patterns;

a third code section executable to identify said attribute values that match as irrelevant in said received attribute pattern and said compared attribute patterns;

a fourth code section executable to cause repeated execution of said first through said third code sections to form and update said set of rules, with each rule comprising a relevant attribute pattern and an associated rule conclusion; and

a fifth code section executable to remove redundant rules from said set of rules

44. A method for formulating a set of rules comprising:

receiving a stream of data records, each data record containing a set of attributes values and an associated conclusion related to a situation;

forming a first set of mutually exclusive attribute value patterns from said data records, each attribute value pattern being associated with a respective conclusion group containing at least one conclusion;

maintaining a conclusion count for each conclusion in said conclusion group;

forming a second set of attribute value patterns from said first set, each attribute value pattern in said second set being associated with a preferred conclusion chosen from said respective associated conclusion group, said attribute value patterns in said second set containing attribute values relevant to said preferred conclusion, said second set of attribute value patterns being formed by;

a) creating in said second set a copy of a selected attribute value pattern with an associated preferred conclusion from said first set;

b) comparing values of said selected attribute value pattern to corresponding values of all other attribute value patterns in said first set having associated preferred conclusions different from said associated preferred conclusion of said selected attribute value pattern thereby identifying any attributes of said selected attribute value pattern that match as irrelevant to said situation;

c) marking said irrelevant attributes from said copied selected attribute value pattern in said second set; and

repeating a), b) and c) for each attribute value pattern in said first set to form said second set of attribute value patterns comprising said set of rules.

45. The method for formulating a set of rules representing said situations according to claim 44, further comprising sampling said stream of data records to increase a relative occurrence frequency of an infrequently occurring conclusion.

46. A method for formulating a set of rules comprising:

receiving data records, each data record containing a set of attributes values forming an attribute value pattern and an associated conclusion representing a situation;

forming from said records a first set of mutually exclusive attribute value patterns, each pattern being associated with a conclusion group containing at least one conclusion, said first set of attribute value patterns being formed by;

a) placing an initial attribute value pattern and associated conclusion into said first set of attribute value patterns, said initial associated conclusion being placed in an associated conclusion group, and initializing a first conclusion count for said initial associated conclusion placed in said first conclusion group;

b) reading another attribute value pattern and associated conclusion from another received data record;

c) comparing said another attribute value pattern to attribute value patterns in said first set of attribute value patterns;

d) adding said another attribute value pattern and associated conclusion into said first set of attribute value patterns if said another attribute value pattern matches none of said attribute value patterns in said first set of attribute value patterns, said another associated conclusion being placed in another conclusion group associated with said another attribute value pattern added to said first set of attribute value patterns, and initializing another conclusion count for said another associated conclusion in said another associated conclusion group;

e) adjusting conclusion counts in said conclusion group associated with a matched attribute value pattern if a match between said another attribute value pattern and an attribute value pattern in said first set of attribute value patterns is found; and

repeating b) through d) thereby forming said first set of mutually exclusive attribute patterns.

47. The method for formulating a set of rules according to claim 46, wherein said adjusting conclusion counts further comprises:

setting a maximum conclusion count value for said conclusion counts;

incrementing a conclusion count for a conclusion in said conclusion group that matches said another conclusion if said conclusion count is less than said maximum count;

decrementing all other conclusion counts greater than zero in said conclusion group if said conclusion count is at said maximum conclusion count value; and

designating as a predominant conclusion of said conclusion group the conclusion associated with its said conclusion count when said conclusion count exceeds all other conclusion counts associated with said conclusion group.

48. The method for formulating a set of rules according to claim 47, further comprising identifying irrelevant attributes in each attribute value pattern in said first set of attribute value patterns if said another attribute value pattern with a conclusion different from any predominant conclusions in said first set of attribute value patterns is added to said first set of attribute value patterns or if said adjustment in said conclusion counts leads to a changed predominant conclusion when there is more than one attribute value pattern in said first set of attribute value patterns.

49. The method for formulating a set of rules according to claim 48, further comprising:

designating as a first list all attributes of an attribute value pattern being added, said attributes of said list being designated as relevant;

comparing said added attribute value pattern to all other attribute value patterns in said first set of attribute value patterns having different associated predominant conclusions;

identifying said irrelevant attributes as those attributes with matching values in corresponding attributes to which they are compared, said irrelevant attributes being designated as irrelevant;

restoring attribute designations from a copy of a list if all attributes of said list in said added attribute value pattern or said compared attribute value pattern are designated as irrelevant; and

creating a new list if all lists of said added attribute value pattern or said compared attribute value pattern are designated as irrelevant, said new list containing only attributes that do not match, designating said attributes of said new list as relevant.

50. The method for formulating a set of rules according to claim 48, further comprising:

replacing said lists designating relevancy of attributes of each attribute value pattern having a predominant conclusion the same as said changed predominant conclusion with a first list of all attributes of said attribute value pattern, said attributes of said list being designated as relevant;

comparing said attribute value patterns associated with conclusions that are the same as said changed predominant conclusion to all other attribute value patterns having different associated predominant conclusions in said first set of attribute value patterns;

identifying said irrelevant attributes as those attributes with matching values in corresponding attributes to which they are compared, said irrelevant attributes being designated as irrelevant;

restoring attribute designations from a copy of a list if all attributes of said list in an attribute value pattern are identified as irrelevant; and

forming a new list for an attribute value pattern of its attributes that do not match if all attributes in all said lists of said compared attribute value pattern are identified as irrelevant.

51. The method for formulating a set of rules according to claim 50, wherein said attribute value pattern having a changed predominant conclusion is compared to all patterns having a predominant conclusion different from said attribute pattern and only retaining copies of said lists of said attribute value pattern having a changed predominant conclusion and said lists of attribute value patterns having predominant conclusions the same as said attribute value pattern before said change.

52. The method for formulating a set of rules according to claim 50, wherein said attribute value patterns having predominant conclusions the same as said changed predominant conclusion, excluding said attribute value pattern having a changed predominant conclusion, are compared to all patterns having a predominant conclusion different from said attribute value pattern;

retaining copies only of said lists for said attribute value patterns for restoring attribute designations; and

stopping for each said attribute pattern when said attribute value pattern list matches a former list.