Data set comparison and net change processing

• US 20040162802A1
• Filed: 02/07/2003
• Published: 08/19/2004
• Est. Priority Date: 02/07/2003
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Stokely-Van Camp Incorporated (PepsiCo Incorporated)
• Inventors: Jonas, Jeffrey James
• Granted Patent: US 7,200,602 B2
• Time in Patent Office: Days
• US Class Current: 707/1
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: G06F 16/90344 by using string matching techniques
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99944 Object-oriented database structure
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99952 Coherency, e.g. same view to multiple users
• CPC Class Codes: Y10S 707/99953 Recoverability

Abstract

A method, program and system for comparing and analyzing a first set of data with a second set of data received by a computer while maintaining a persistent key is disclosed. The method, program and system comprising the step of receiving a first data set and a second data set, each data set comprising at least one record, where each record reflects at least one of a plurality of entities. In this regard, more than one record may reflect the same entity (e.g., an entity representing a specific person). The method, program and system further comprises the step of utilizing an algorithm to: (a) compare the second data set to at least a portion of the first data set; (b) identify or assign a persistent key (and perhaps the same persistent key for records reflecting the same entity) for each record in the second data set; and (c) create a database or file (i.e., a journal) to include any records: (i) in the second data set that: (1) do not exist in the first data set (perhaps with an “add” directive representing additions and the identified or assigned persistent key), (2) include at least one change to at least one record in the first data set (perhaps with a “modify” directive representing modifications and the identified or assigned persistent key), and/or (3) perhaps do not include at least one change to at least one record in the first data set, but reflects the identical one of a plurality of entities as a record in the first data set with a date (perhaps with a “no change” directive representing that the same record in the first data set was submitted in the second data set); and/or (ii) in the first data set that do not exist in the second data set, perhaps: (1) with a “delete” directive representing deletions and the identified persistent key and (2) only if the second data set is not incremental (e.g, only last month'"'"'s changes, rather than a complete data set) of the at least a portion of the first data set.

Claims

1. A method for processing data comprising the steps of:

receiving a first data set and a second data set, each of the data sets comprising at least one record reflecting at least one of a plurality of entities; and

utilizing an algorithm to;

compare the second data set to at least a portion of the first data set;

one of identify and assign a persistent key to each record in the second data set; and

create one of a database and file to include at least one record reflecting at least one of;

at least one record in the second data set that is determined to not exist in the at least a portion of the first data set;

at least one record in the second data set that is determined to include at least one change to at least one record in the at least one record in the at least a portion of the first data set reflecting an identical one of the plurality of entities; and

at least one record in the at least a portion of the first data set that is determined to not exist in the second data set.

2. The method of claim 1 wherein the one of a plurality of entities is at least one of people, personal property, real property, organizations, chemical compounds, organic compounds, proteins, biometric values and atomic structures.

3. The method of claim 1 wherein the step of utilizing an algorithm includes at least one of:

creating a new portion of the second data set;

modifying at least a portion of the second data set; and

organizing the second data set for efficient comparison prior to comparing the second data set to the at least a portion of the first data set.

4. The method of claim 3 wherein the step of modifying at least a portion of the second data set includes one of removing at least a portion of the second data set meeting a user-defined criterion and replacing at least a portion of the second data set meeting a user-defined criterion.

5. The method of claim 3 wherein the step of organizing the second data set for efficient comparison includes sorting the second data set.

6. The method of claim 3 wherein the step of organizing the second data set for efficient comparison includes utilizing a database structure.

7. The method of claim 3 wherein the step of organizing the second data set for efficient comparison includes utilizing a memory array.

8. The method of claim 1 wherein the step of utilizing an algorithm includes organizing the at least a portion of the first data set for efficient comparison prior to comparing the second data set to the at least a portion of the first data set.

9. The method of claim 1 wherein the step of utilizing an algorithm further comprises modifying the first data set to reflect the second data set.

10. The method of claim 1 wherein the step of utilizing an algorithm to compare the second data set and the at least a portion of the first data set includes determining whether at least one record in the at least one record in the

second data set:

does not exist in the first data set; and

includes at least one change to at least one record in the at least a portion of the first data set that is determined to reflect an identical one of the plurality of entities; and

at least a portion of the first data set does not exist in the second data set.

11. The method of claim 1 wherein the step of utilizing an algorithm to one of identify and assign a persistent key includes assigning a persistent key that was previously identified or assigned to a record in the at least a portion of the first data set to at least one record in the second data set when the at least one record in the second data set is determined to reflect the identical one of the plurality of entities.

12. The method of claim 1 wherein the step of utilizing an algorithm to create one of

a database and file further includes:

at least one record in the second data set that does not include a change to at least one record in the at least one record in the at least a portion of the first data set reflecting an identical one of the plurality of entities; and

a date.

13. The method of claim 12 wherein the step of utilizing an algorithm to create one of a database and file to include at least one record comprising at least one record in the second data set that does not include a change to at least one record in the at least one record in the at least a portion of the first data set reflecting an identical one of the plurality of entities includes a no change directive.

14. The method of claim 1 wherein the step of utilizing an algorithm to create one of a database and file to include at least one record comprising at least one record in the second data set that is determined to not exist in the at least a portion of the first data set includes an add directive.

15. The method of claim 1 wherein the step of utilizing an algorithm to create one of a database and file to include at least one record comprising at least one record in the second data set that is determined to include at least one change to at least one record in the at least one record in the at least a portion of the first data set reflecting an identical one of the plurality of entities includes a modify directive.

16. The method of claim 1 wherein the step of utilizing an algorithm to create one of a database and file to include at least one record comprising at least one record in the at least a portion of the first data set that is determined to not exist in the second data set includes a delete directive.

17. The method of claim 16 wherein the step of utilizing an algorithm to create one of a database and file to include at least one record comprising at least one record in the at least a portion of the first data set that is determined to not exist in the second data set occurs only when the second data set is not determined to be an incremental data set of the at least a portion of the first data set.

18. A method of processing data by a computer comprising the steps of:

receiving a first data set having a first record;

assigning a persistent key to the first record;

receiving a second data set having a second record;

comparing the second record to the first record; and

recording an entry in a journal pertaining to the comparison of the second record to the first record.

19. The method of claim 18 further comprising the step of assigning a persistent key to the second record identical to the persistent key assigned to the first record if the second record matches the first record.

20. The method of claim 18 further comprising the step of assigning a persistent key different from the persistent key assigned to the first record if the second record does not match the first record.

21. The method of claim 18 wherein the step of recording an entry in a journal includes recording a changed second record entry in the journal if the second record matches the first record and includes changes to information contained in the first record.

22. The method of claim 21 wherein the step of recording an entry in a journal includes recording a “

modify”

directive in the journal if the second record matches the first record and includes changes to information contained in the first record.

23. The method of claim 18 wherein the step of recording an entry in a journal includes recording the persistent key and a date if the second record matches the first record and does not include changes to information contained in the first record.

24. The method of claim 23 wherein the step of recording an entry in a journal includes recording a “

no change”

directive in the journal if the second record matches the first record and does not include changes to information contained in the first record.

25. The method of claim 18 wherein the step of recording an entry in a journal includes recording the second record in the journal with an “

add”

directive if the second record does not match the first record.

26. The method of claim 18 wherein the step of recording an entry in a journal includes recording the first record in the journal with a “

delete”

directive if;

the first record does not match the second record and

the second data set is not an incremental data set of the first data set.

27. The method of claim 18 wherein the first data set includes a plurality of first records.

28. The method of claim 18 wherein the second data set includes a plurality of second records.

29. The method of claim 18 wherein the first record represents one of a plurality of entities.

30. The method of claim 18 further comprising the step of modifying the second data set prior to comparing the second record to the first record.

31. The method of claim 18 further comprising the step of organizing the second data set for efficient comparison prior to comparing the second record to the first record.

32. The method of claim 18 further comprising the step of organizing the first data set for efficient comparison prior to comparing the second record to the first record.

33. The method of claim 18 further comprising the step of modifying the first data set to reflect the second data set.

34. For a system for processing data and a computer readable medium containing program instructions for execution by a computer for performing the method comprising the steps of:

receiving a first data set and a second data set, each of the data sets comprising at least one record reflecting at least one of a plurality of entities; and

utilizing an algorithm to;

compare the second data set to at least a portion of the first data set;

one of identify and assign a persistent key to each record in the second data set; and

create one of a database and file to include at least one record reflecting at least one of;

at least one record in the second data set that is determined to not exist in the at least a portion of the first data set;

at least one record in the second data set that is determined to include at least one change to at least one record in the at least one record in the at least a portion of the first data set reflecting an identical one of the plurality of entities; and

at least one record in the at least a portion of the first data set that is determined to not exist in the second data set.

35. The computer readable medium for performing the method of claim 34 wherein the one of a plurality of entities is at least one of people, personal property, real property, organizations, chemical compounds, organic compounds, proteins, biometric values and atomic structures.

36. The computer readable medium for performing the method of claim 34 wherein the step of utilizing an algorithm includes at least one of:

creating a new portion of the second data set;

modifying at least a portion of the second data set; and

organizing the second data set for efficient comparison prior to comparing the second data set to the at least a portion of the first data set.

37. The computer readable medium for performing the method of claim 36 wherein the step of modifying at least a portion of the second data set includes one of removing at least a portion of the second data set meeting a user-defined criterion and replacing at least a portion of the second data set meeting a user-defined criterion.

38. The computer readable medium for performing the method of claim 36 wherein the step of organizing the second data set for efficient comparison includes sorting the second data set.

39. The computer readable medium for performing the method of claim 36 wherein the step of organizing the second data set for efficient comparison includes utilizing a database structure.

40. The computer readable medium for performing the method of claim 36 wherein the step of organizing the second data set for efficient comparison includes utilizing a memory array.

41. The computer readable medium for performing the method of claim 34 wherein the step of utilizing an algorithm includes organizing the at least a portion of the first data set for efficient comparison prior to comparing the second data set to the at least a portion of the first data set.

42. The computer readable medium for performing the method of claim 34 wherein the step of utilizing an algorithm further comprises modifying the first data set to reflect the second data set.

43. The computer readable medium for performing the method of claim 34 wherein the step of utilizing an algorithm to compare the second data set and the at least a portion of the first data set includes determining whether at least one record in the at least one record

second data set:

does not exist in the first data set; and

includes at least one change to at least one record in the at least a portion of the first data set that is determined to reflect an identical one of the plurality of entities; and

at least a portion of the first data set does not exist in the second data set.

44. The computer readable medium for performing the method of claim 34 wherein the step of utilizing an algorithm to one of identify and assign a persistent key includes assigning a persistent key that was previously identified or assigned to a record in the at least a portion of the first data set to at least one record in the second data set when the at least one record in the second data set is determined to reflect the identical one of the plurality of entities.

45. The computer readable medium for performing the method of claim 34 wherein the step of utilizing an algorithm to create one of a database and file further includes:

at least one record in the second data set that does not include a change to at least one record in the at least one record in the at least a portion of the first data set reflecting an identical one of the plurality of entities; and

a date.

46. The computer readable medium for performing the method of claim 45 wherein the step of utilizing an algorithm to create one of a database and file to include at least one record comprising at least one record in the second data set that does not include a change to at least one record in the at least one record in the at least a portion of the first data set reflecting an identical one of the plurality of entities includes a no change directive.

47. The computer readable medium for performing the method of claim 34 wherein the step of utilizing an algorithm to create one of a database and file to include at least one record comprising at least one record in the second data set that is determined to not exist in the at least a portion of the first data set includes an add directive.

48. The computer readable medium for performing the method of claim 34 wherein the step of utilizing an algorithm to create one of a database and file to include at least one record comprising at least one record in the second data set that is determined to include at least one change to at least one record in the at least one record in the at least a portion of the first data set reflecting an identical one of the plurality of entities includes a modify directive.

49. The computer readable medium for performing the method of claim 34 wherein the step of utilizing an algorithm to create one of a database and file to include at least one record comprising at least one record in the at least a portion of the first data set that is determined to not exist in the second data set includes a delete directive.

50. The computer readable medium for performing the method of claim 49 wherein the step of utilizing an algorithm to create one of a database and file to include at least one record comprising at least one record in the at least a portion of the first data set that is determined to not exist in the second data set occurs only when the second data set is not determined to be an incremental data set of the at least a portion of the first data set.

51. For a system for processing data and a computer readable medium containing program instructions for execution by a computer for performing the method comprising the steps of:

receiving a first data set having a first record;

assigning a persistent key to the first record;

receiving a second data set having a second record;

comparing the second record to the first record; and

recording an entry in a journal pertaining to the comparison of the second record to the first record.

52. The computer readable medium for performing the method of claim 51 further comprising the step of assigning a persistent key to the second record identical to the persistent key assigned to the first record if the second record matches the first record.

53. The computer readable medium for performing the method of claim 51 further comprising the step of assigning a persistent key different from the persistent key assigned to the first record if the second record does not match the first record.

54. The computer readable medium for performing the method of claim 51 wherein the step of recording an entry in a journal includes recording a changed second record entry in the journal if the second record matches the first record and includes changes to information contained in the first record.

55. The computer readable medium for performing the method of claim 54 wherein the step of recording an entry in a journal includes recording a “

modify”

directive in the journal if the second record matches the first record and includes changes to information contained in the first record.

56. The computer readable medium for performing the method of claim 51 wherein the step of recording an entry in a journal includes recording the persistent key and a date if the second record matches the first record and does not include changes to information contained in the first record.

57. The computer readable medium for performing the method of claim 56 wherein the step of recording an entry in a journal includes recording a “

no change”

directive in the journal if the second record matches the first record and does not include changes to information contained in the first record.

58. The computer readable medium for performing the method of claim 51 wherein the step of recording an entry in a journal includes recording the second record in the journal with an “

add”

directive if the second record does not match the first record.

59. The computer readable medium for performing the method of claim 51 wherein the step of recording an entry in a journal includes recording the first record in the journal with a “

delete”

directive if;

the first record does not match the second record and

the second data set is not an incremental data set of the first data set.

60. The computer readable medium for performing the method of claim 51 wherein the first data set includes a plurality of first records.

61. The computer readable medium for performing the method of claim 51 wherein the second data set includes a plurality of second records.

62. The computer readable medium for performing the method of claim 51 wherein the first record represents one of a plurality of entities.

63. The computer readable medium for performing the method of claim 51 further comprising the step of modifying the second data set prior to comparing the second record to the first record.

64. The computer readable medium for performing the method of claim 51 further comprising the step of organizing the second data set for efficient comparison prior to comparing the second record to the first record.

65. The computer readable medium for performing the method of claim 51 further comprising the step of organizing the first data set for efficient comparison prior to comparing the second record to the first record.

66. The computer readable medium for performing the method of claim 51 further comprising the step of modifying the first data set to reflect the second data set.