System and method for data collection

• US 7,822,710 B1
• Filed: 05/24/2007
• Issued: 10/26/2010
• Est. Priority Date: 05/24/2006
• Status: Expired due to Fees

• Current Assignee: Troux Technologies Incorporated
• Original Assignee: Troux Technologies Incorporated
• Inventors: Kitchin, Anthony, Dennis, Matthew F., Miller, Clint, Williams, David K.
• Primary Examiner(s): Lee; Wilson
• Application Number:
  US11/805,873
• Time in Patent Office: 1,251 Days
• Field of Search: 707100-102, 707200-202, 707609-624, 707644-647, 707678-682
• US Class Current: 707/620
• CPC Class Codes: G06F 16/907 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually

Abstract

Embodiments of the present invention provide methods and systems for extracting data from multiple sources and loading data into a central repository. One embodiment of a method of the present invention includes extracting a first set of data from one or more data sources, correlating the first set of data with a second set of data contained in a central repository, comparing the first set of data with the second set of data, determining a set of operations to be applied to the second set of data and applying the set of operations to the second set of data.

Claims

1. A method of extracting data from one or more data sources to a central repository storing a data model, the data model comprising a set of components and relationships representing entities in an environment and the relationships between these entities, wherein each of the components and relationships in the data model has a type and a set of properties and the values for the type and set of properties of each of the components and relationships in the data model are based on a first set of data extracted at a first time from each of the one or more data sources, comprising:

extracting a second set of data from the one or more data sources at a second time subsequent to the first time;

normalizing and mapping the second set of data into a format associated with the data model in the central repository;

correlating the second set of data extracted at the second time with the components and relationships of the data model stored in the central repository;

comparing the components and relationships of the data model in the central repository with the second set of data; and

updating the data model in the central repository based on the comparison of the components and relationships of the data model, wherein updating the data model comprises

determining a set of operations to update the data central repository, wherein the set of operations is based on the comparison between the second set of data and the component and relationships in the central repository and the set of operations are configured to alter the types of values of the components or relationships in the central repository or to add components or relationships to the central repository to account for changes in the one or more data sources associated with the components or relationships which occurred between the first time and the second time, and

applying the set of operations to the central repository to update the data model.

2. The method of claim 1, wherein the first set of data is associated with a first set of precedences, the second set of data is associated with a second set of precedences and determining a set of operations to be applied to the second set of data is based on the first set of precedences, the second set of precedences and precedence rules.

3. The method of claim 2, further comprising updating the second set of precedences based on the first set of precedences and the set of operations.

4. The method of claim 1, further comprising generating a first set of data structures from the second set of data.

5. The method of claim 4, wherein comparing the components and relationships of the data model in the central repository with the second set of data comprises correlating the first set of data structures with a second set of data structures associated with the data model.

6. The method of claim 1, wherein normalizing and mapping the second set of data comprises producing an XML file containing the current state of the second set of data.

7. The method of claim 1, further comprising dividing the second set of data into a plurality of batches, wherein the second set of data is compared according to the plurality of batches.

8. A non-transitory computer readable medium comprising instructions for extracting data from one or more data sources to a central repository storing a data model, the data model comprising a set of components and relationships representing entities in an environment and the relationships between these entities, wherein each of the components and relationships in the data model has a type and a set of properties and the values for the type and set of properties of each of the components and relationships in the data model are based on a first set of data extracted at a first time from each of the one or more data sources, the instruction including instructions for:

extracting a second set of data from the one or more data sources at a second time subsequent to the first time;

normalizing and mapping the second set of data into a format associated with the data model in the central repository;

correlating the second set of data extracted at the second time with the components and relationships of the data model stored in the central repository;

comparing the components and relationships of the data model in the central repository with the second set of data; and

updating the data model in the central repository based on the comparison of the components and relationships of the data model, wherein updating the data model comprises

determining a set of operations to update the data central repository, wherein the set of operations is based on the comparison between the second set of data and the component and relationships in the central repository and the set of operations are configured to alter the types of values of the components or relationships in the central repository or to add components or relationships to the central repository to account for changes in the one or more data sources associated with the components or relationships which occurred between the first time and the second time, and

applying the set of operations to the central repository to update the data model.

9. The computer readable medium of claim 8, wherein the first set of data is associated with a first set of precedences, the second set of data is associated with a second set of precedences and determining a set of operations to be applied to the second set of data is based on the first set of precedences, the second set of precedences and precedence rules.

10. The computer readable medium of claim 9, further comprising updating the second set of precedences based on the first set of precedences and the set of operations.

11. The computer readable medium of claim 8, further comprising generating a first set of data structures from the second set of data.

12. The computer readable medium of claim 11, wherein comparing the components and relationships of the data model in the central repository with the second set of data comprises correlating the first set of data structures with a second set of data structures associated with the data model.

13. The computer readable medium of claim 8, wherein normalizing and mapping the second set of data comprises producing an XML file containing the current state of the second set of data.

14. The computer readable medium of claim 8, further comprising dividing the second set of data into a plurality of batches, wherein the second set of data is compared according to the plurality of batches.

15. A system, including

one or more data sources;

a central repository coupled to the one or more data sources, the central repository storing a data model, the data model comprising a set of components and relationships representing entities in an environment and the relationships between these entities, wherein each of the components and relationships in the data model has a type and a set of properties and the values for the type and set of properties of each of the components and relationships in the data model are based on a first set of data extracted at a first time from each of the one or more data sources; and

a non-transitory computer readable medium, including instructions for extracting data from the one or more data sources to the central repository, the instruction including instructions for;

extracting a second set of data from the one or more data sources at a second time subsequent to the first time;

normalizing and mapping the second set of data into a format associated with the data model in the central repository;

correlating the second set of data extracted at the second time with the components and relationships of the data model stored in the central repository;

comparing the components and relationships of the data model in the central repository with the second set of data; and

updating the data model in the central repository based on the comparison of the components and relationships of the data model, wherein updating the data model comprises

determining a set of operations to update the data central repository, wherein the set of operations is based on the comparison between the second set of data and the component and relationships in the central repository and the set of operations are configured to alter the types of values of the components or relationships in the central repository or to add components or relationships to the central repository to account for changes in the one or more data sources associated with the components or relationships which occurred between the first time and the second time, and

applying the set of operations to the central repository to update the data model.

16. The system of claim 15, wherein the first set of data is associated with a first set of precedences, the second set of data is associated with a second set of precedences and determining a set of operations to be applied to the second set of data is based on the first set of precedences, the second set of precedences and precedence rules.

17. The system of claim 16, further comprising updating the second set of precedences based on the first set of precedences and the set of operations.

18. The system of claim 15, further comprising generating a first set of data structures from the second set of data.

19. The system of claim 18, wherein comparing the components and relationships of the data model in the central repository with the second set of data comprises correlating the first set of data structures with a second set of data structures associated with the data model.

20. The system of claim 15, wherein normalizing and mapping the second set of data comprises producing an XML file containing the current state of the second set of data.

21. The system of claim 15, further comprising dividing the second set of data into a plurality of batches, wherein the second set of data is compared according to the plurality of batches.