Data set comparison and net change processing

• US 7,200,602 B2
• Filed: 02/07/2003
• Issued: 04/03/2007
• Est. Priority Date: 02/07/2003
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: International Business Machines Corporation
• Inventors: Jonas, Jeffrey James
• Primary Examiner(s): Pardo; Thuy N.
• Application Number:
  US10/361,069
• Publication Number:
  US 20040162802A1
• Time in Patent Office: 1,516 Days
• Field of Search: 707/3, 707/10, 707/1, 707/101, 707/203, 707/204, 707/6, 707/7, 707/102, 707/205, 707/4, 707/100, 707/103.Y, 707/104.1, 707/201, 713/100, 713/194, 348/239, 382/181, 382/154, 705/10
• US Class Current: 1/1
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: G06F 16/90344 by using string matching techniques
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99944 Object-oriented database structure
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99952 Coherency, e.g. same view to multiple users
• CPC Class Codes: Y10S 707/99953 Recoverability

Abstract

A method and system for comparing and analyzing a first set of data with a second set of data received by a computer while maintaining a persistent key is disclosed. The method and system comprising the step of receiving a first data set and a second data set, each data set comprising at least one record, where each record reflects at least one of a plurality of entities. In this regard, more than one record may reflect the same entity (e.g., an entity representing a specific person). The method, program and system further comprises the step of utilizing an algorithm to compare the second data set to at least a portion of the first data set; identify or assign a persistent key (and perhaps the same persistent key for records reflecting the same entity) for each record in the second data set; and create a database or file (i.e., a journal).

Claims

1. A method for data set comparison and net change processing by a computer comprising the steps of:

(a) receiving a first data set and a second data set, each of the data sets comprising at least one record reflecting at least one of a plurality of entities, the at least one record of the first data set having a persistent key; and

(b) utilizing an algorithm to;

(1) compare the second data set to at least a portion of the first data set;

(2) identify or assign a persistent key of the first data set to a record in the second data set, if the record of the second data set matches the record of the first data set; and

(3) create a journal database or file to include at least one entry reflecting the persistent key and at least one of;

(i) at least one record in the second data set that is determined to not exist in the portion of the first data set;

(ii) at least one record in the second data set that is determined to include at least one change to at least one record in the portion of the first data set reflecting an identical one of the plurality of entities; and

(iii) at least one record in the portion of the first data set that is determined to not exist in the second data set.

2. The method of claim 1 wherein the one of a plurality of entities is at least one of people, personal property, real property, organizations, chemical compounds, organic compounds, proteins, biometric values and atomic structures.

3. The method of claim 1 wherein the step of utilizing an algorithm includes at least one of:

creating a new portion of the second data set;

modifying at least a portion of the second data set; and

organizing the second data set for efficient comparison prior to comparing the second data set to the portion of the first data set.

4. The method of claim 3 wherein the step of modifying at least a portion of the second data set includes one of removing at least a portion of the second data set meeting a user-defined criterion and replacing at least a portion of the second data set meeting a user-defined criterion.

5. The method of claim 3 wherein the step of organizing the second data set for efficient comparison includes sorting the second data set.

6. The method of claim 3 wherein the step of organizing the second data set for efficient comparison includes utilizing a database structure.

7. The method of claim 3 wherein the step of organizing the second data set for efficient comparison includes utilizing a memory array.

8. The method of claim 1 wherein the step of utilizing an algorithm includes organizing the portion of the first data set for efficient comparison prior to comparing the second data set to the portion of the first data set.

9. The method of claim 1 wherein the step of utilizing an algorithm further comprises modifying the first data set to reflect the second data set.

10. The method of claim 1 wherein the step of utilizing an algorithm to compare the second data set and the portion of the first data set includes determining whether at least one record in the in the second data set:

does not exist in the first data set; and

includes at least one change to at least one record in the portion of the first data set that is determined to reflect an identical one of the plurality of entities; and

at least a portion of the first data set does not exist in the second data set.

11. The method of claim 1 wherein the step of utilizing an algorithm to identify or assign a persistent key includes assigning a persistent key that was previously identified or assigned to a record in the portion of the first data set to at least one record in the second data set when the record in the second data set is determined to reflect the identical one of the plurality of entities.

12. The method of claim 1 wherein the step of utilizing an algorithm to create one of a journal database or file further includes:

at least one record in the second data set that does not include a change to at least one record in the portion of the first data set reflecting an identical one of the plurality of entities; and

a date.

13. The method of claim 12 wherein the step of utilizing an algorithm to create a journal database or file to include at least one entry comprising at least one record in the second data set that does not include a change to at least one record in the portion of the first data set reflecting an identical one of the plurality of entities;

and a “

no change”

directive.

14. The method of claim 1 wherein the step of utilizing an algorithm to create a journal database or file to include at least one entry comprising at least one record in the second data set that is determined to not exist in the at least a portion of the first data set;

and an “

add”

directive.

15. The method of claim 1 wherein the step of utilizing an algorithm to create a journal database or file to include at least one entry comprising at least one record in the second data set that is determined to include at least one change to at least one record in the portion of the first data set reflecting an identical one of the plurality of entities;

and a “

modify”

directive.

16. The method of claim 1 wherein the step of utilizing an algorithm to create a journal database or file to include at least one entry comprising at least one record in the portion of the first data set that is determined to not exist in the second data set;

and a “

delete”

directive.

17. The method of claim 16 wherein the step of utilizing an algorithm to create a journal database or file to include at least one entry comprising at least one record in the portion of the first data set that is determined to not exist in the second data set occurs only when the second data set is not determined to be an incremental data set of the portion of the first data set.

18. A method of data set comparison and net change processing by a computer comprising the steps of:

(a) receiving a first data set having a first record;

(b) assigning a persistent key to the first record;

(c) receiving a second data set having a second record;

(d) comparing the second record to the first record;

(e) assigning the persistent key of the first record to the second record if the second record matches the first record; and

(f) recording an entry in a journal pertaining to the comparison of the second record to the first record reflecting the persistent key and at least one of;

(1) the second record in the second data set that is determined to not exist in the first data set;

(2) the second record in the second data set that is determined to match the first record in the first data set and include at least one change to the first record in the first data set; and

(3) the first rrecord in the first data set that is determined to not exist in the second data set.

19. The method of claim 18 further comprising the step of assigning a persistent key to the second record identical to the persistent key assigned to the first record if the second record matches the first record.

20. The method of claim 18 further comprising the step of assigning a persistent key different from the persistent key assigned to the first record if the second record does not match the first record.

21. The method of claim 18 wherein the step of recording an entry in a journal includes recording a changed second record entry in the journal if the second record matches the first record and includes changes to information contained in the first record.

22. The method of claim 21 wherein the step of recording an entry in a journal includes recording a “

modify”

directive in the journal if the second record matches the first record and includes changes to information contained in the first record.

23. The method of claim 18 wherein the step of recording an entry in a journal includes recording the persistent key and a date if the second record matches the first record and does not include changes to information contained in the first record.

24. The method of claim 23 wherein the step of recording an entry in a journal includes recording a “

no change”

directive in the journal if the second record matches the first record and does not include changes to information contained in the first record.

25. The method of claim 18 wherein the step of recording an entry in a journal includes recording the second record in the journal with an “

add”

directive if the second record does not match the first record.

26. The method of claim 18 wherein the step of recording an entry in a journal includes recording the first record in the journal with a “

delete”

directive if;

the first record does not match the second record and the second data set is not an incremental data set of the first data set.

27. The method of claim 18 wherein the first data set includes a plurality of first records.

28. The method of claim 18 wherein the second data set includes a plurality of second records.

29. The method of claim 18 wherein the first record represents one of a plurality of entities.

30. The method of claim 18 further comprising the step of modifying the second data set prior to comparing the second record to the first record.

31. The method of claim 18 further comprising the step of organizing the second data set for efficient comparison prior to comparing the second record to the first record.

32. The method of claim 18 further comprising the step of organizing the first data set for efficient comparison prior to comparing the second record to the first record.

33. The method of claim 18 further comprising the step of modifying the first data set to reflect the second data set.

34. For a system for data set comparison and net change processing and a computer readable medium containing program instructions for execution by a computer for performing the method comprising the steps of:

(a) receiving a first data set and a second data set, each of the data sets comprising at least one record reflecting at least one of a plurality of entities, the records of the first data set having a persistent key; and

(b) utilizing an algorithm to;

(1) compare the second data set to at least a portion of the first data set;

(2) identify or assign a persistent key of a record of the first data set to a record in the second data set, if the record of the second data set matches the record of the first data set; and

(3) create a journal database or file to include at least one entry reflecting the persistent key and at least one of;

(i) at least one record in the second data set that is determined to not exist in the portion of the first data set;

(ii) at least one record in the second data set that is determined to include at least one change to at least one record in the portion of the first data set reflecting an identical one of the plurality of entities; and

(iii) at least one record in the portion of the first data set that is determined to not exist in the second data set.

35. The computer readable medium for performing the method of claim 34 wherein the one of a plurality of entities is at least one of people, personal property, real property, organizations, chemical compounds, organic compounds, proteins, biometric values and atomic structures.

36. The computer readable medium for performing the method of claim 34 wherein the step of utilizing an algorithm includes at least one of:

creating a new portion of the second data set;

modifying at least a portion of the second data set; and

organizing the second data set for efficient comparison prior to comparing the second data set to the portion of the first data set.

37. The computer readable medium for performing the method of claim 36 wherein the step of modifying at least a portion of the second data set includes one of removing at least a portion of the second data set meeting a user-defined criterion and replacing at least a portion of the second data set meeting a user-defined criterion.

38. The computer readable medium for performing the method of claim 36 wherein the step of organizing the second data set for efficient comparison includes sorting the second data set.

39. The computer readable medium for performing the method of claim 36 wherein the step of organizing the second data set for efficient comparison includes utilizing a database structure.

40. The computer readable medium for performing the method of claim 36 wherein the step of organizing the second data set for efficient comparison includes utilizing a memory array.

41. The computer readable medium for performing the method of claim 34 wherein the step of utilizing an algorithm includes organizing the portion of the first data set for efficient comparison prior to comparing the second data set to the portion of the first data set.

42. The computer readable medium for performing the method of claim 34 wherein the step of utilizing an algorithm further comprises modifying the first data set to reflect the second data set.

43. The computer readable medium for performing the method of claim 34 wherein the step of utilizing an algorithm to compare the second data set and the portion of the first data set includes determining whether at least one record in the second data set:

does not exist in the first data set; and

includes at least one change to at least one record in the portion of the first data set that is determined to reflect an identical one of the plurality of entities; and

at least a portion of the first data set does not exist in the second data set.

44. The computer readable medium for performing the method of claim 34 wherein the step of utilizing an algorithm to one of identify or assign a persistent key includes assigning a persistent key that was previously identified or assigned to a record in the portion of the first data set to at least one record in the second data set when the record in the second data set is determined to reflect the identical one of the plurality of entities.

45. The computer readable medium for performing the method of claim 34 wherein the step of utilizing an algorithm to create a journal database or file further includes:

at least one record in the second data set that does not include a change to at least one record in the portion of the first data set reflecting an identical one of the plurality of entities; and

a date.

46. The computer readable medium for performing the method of claim 45 wherein the step of utilizing an algorithm to create a journal database or file to include at least one entry comprising at least one record in the second data set that does not include a change to at least one record in the at least one record in the portion of the first data set reflecting an identical one of the plurality of entities;

and a “

no change”

directive.

47. The computer readable medium for performing the method of claim 34 wherein the step of utilizing an algorithm to create a journal database or file to include at least one entry comprising at least one record in the second data set that is determined to not exist in the portion of the first data set;

and an “

add”

directive.

48. The computer readable medium for performing the method of claim 34 wherein the step of utilizing an algorithm to create a journal database or file to include at least one entry comprising at least one record in the second data set that is determined to include at least one change to at least one record in the at least one record in the portion of the first data set reflecting an identical one of the plurality of entities;

and a “

modify”

directive.

49. The computer readable medium for performing the method of claim 34 wherein the step of utilizing an algorithm to create journal a database or file to include at least one entry comprising at least one record in the portion of the first data set that is determined to not exist in the second data set;

and a “

delete”

directive.

50. The computer readable medium for performing the method of claim 49 wherein the step of utilizing an algorithm to create a journal database or file to include at least one entry comprising at least one record in the portion of the first data set that is determined to not exist in the second data set occurs only when the second data set is not determined to be an incremental data set of the portion of the first data set.

51. For a system for data set comparison and net change processing and a computer readable medium containing program instructions for execution by a computer for performing the method comprising the steps of:

(a) receiving a first data set having a first record;

(b) assigning a persistent key to the first record;

(c) receiving a second data set having a second record;

(d) comparing the second record to the first record;

(e) assigning the persistent key of the first record to the second record if the second record matches the first record; and

(f) recording an entry in a journal pertaining to the comparison of the second record to the first record reflecting the persistent key and at least one of;

(1) the second record in the second data set that is determined to not exist in the first data set;

(2) the second record in the second data set that is determined to match the first record in the first data set and include at least one change to the first record in the first data set; and

(3) the first record in the first data set that is determined to not exist in the second data set.

52. The computer readable medium for performing the method of claim 51 further comprising the step of assigning a persistent key to the second record identical to the persistent key assigned to the first record if the second record matches the first record.

53. The computer readable medium for performing the method of claim 51 further comprising the step of assigning a persistent key different from the persistent key assigned to the first record if the second record does not match the first record.

54. The computer readable medium for performing the method of claim 51 wherein the step of recording an entry in a journal includes recording a changed second record entry in the journal if the second record matches the first record and includes changes to information contained in the first record.

55. The computer readable medium for performing the method of claim 54 wherein the step of recording an entry in a journal includes recording a “

modify”

directive in the journal if the second record matches the first record and includes changes to information contained in the first record.

56. The computer readable medium for performing the method of claim 51 wherein the step of recording an entry in a journal includes recording the persistent key and a date if the second record matches the first record and does not include changes to information contained in the first record.

57. The computer readable medium for performing the method of claim 56 wherein the step of recording an entry in a journal includes recording a “

no change”

directive in the journal if the second record matches the first record and does not include changes to information contained in the first record.

58. The computer readable medium for performing the method of claim 51 wherein the step of recording an entry in a journal includes recording the second record in the journal with an “

add”

directive if the second record does not match the first record.

59. The computer readable medium for performing the method of claim 51 wherein the step of recording an entry in a journal includes recording the first record in the journal with a “

delete”

directive if;

the first record does not match the second record and the second data set is not an incremental data set of the first data set.

60. The computer readable medium for performing the method of claim 51 wherein the first data set includes a plurality of first records.

61. The computer readable medium for performing the method of claim 51 wherein the second data set includes a plurality of second records.

62. The computer readable medium for performing the method of claim 51 wherein the first record represents one of a plurality of entities.

63. The computer readable medium for performing the method of claim 51 further comprising the step of modifying the second data set prior to comparing the second record to the first record.

64. The computer readable medium for performing the method of claim 51 further comprising the step of organizing the second data set for efficient comparison prior to comparing the second record to the first record.

65. The computer readable medium for performing the method of claim 51 further comprising the step of organizing the first data set for efficient comparison prior to comparing the second record to the first record.

66. The computer readable medium for performing the method of claim 51 further comprising the step of modifying the first data set to reflect the second data set.