Systems and methods for collection and consolidation of heterogeneous remote business data using dynamic data handling

• US 9,740,757 B1
• Filed: 08/08/2016
• Issued: 08/22/2017
• Est. Priority Date: 08/26/2008
• Status: Active Grant

• Current Assignee: ZeeWise, Inc.
• Original Assignee: ZeeWise, Inc.
• Inventors: Gilder, Clark S., Hix, Josh, Zalewski, Bartosz J.
• Primary Examiner(s): Iwarere, Seye
• Application Number:
  US15/230,719
• Time in Patent Office: 379 Days
• Field of Search: 705 30, 4554141, 725 25
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/256 in federated or virtual databases
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: G06F 40/151 Transformation
• CPC Class Codes: G06Q 40/12 Accounting

Abstract

A method, system and computer program product for controlling collection of data from heterogeneous data sources is disclosed. Data is extracted in a data source independent manner using at least two types of control messages to hide implementation details for the heterogeneous data sources. The method receives from the heterogeneous data sources, a request for a first control message type, which defines data to be extracted from the heterogeneous data sources. Responsive to receiving the request, the method sends a first control messages type to the heterogeneous data sources. The method receives from the heterogeneous data sources, a second control message type that includes data extracted from the heterogeneous data sources that was defined in the first control message type. The method also stores, in a data store, the received data identified in the first control message type and that was extracted from the heterogeneous data sources.

Claims

1. A method, performed by a processor, for controlling the collection of data from a plurality of heterogeneous data sources, wherein the data is extracted in a data source independent manner relative to the plurality of heterogeneous data sources using at least two types of control message to hide implementation details for each of the plurality of heterogeneous data sources, the method comprising:

receiving from at least one of plurality of heterogeneous data sources, a request for a first type of control message, wherein the first type of control message is comprised of data collection information, wherein the data collection information is comprised of a collection definition that defines data to be extracted from the at least one of the plurality of heterogeneous data sources;

responsive to receiving the request, sending one or more of the first type of control messages to at least one of the plurality of heterogeneous data sources;

receiving from the at least one of the plurality of heterogeneous data sources, a second type of control message, wherein the second control message type is comprised of data extracted from the at least one of the plurality of heterogeneous data sources that was defined in the one or more of the first type of control messages; and

storing, in a data store, the received data that was identified in the first type of control message and that was extracted from the at least one of the plurality of heterogeneous data sources.

2. The method of claim 1, wherein the first type of control message is further comprised of data collection scheduling information, wherein the data collection scheduling information is comprised of information indicating a schedule for collecting data from the at least one of the plurality of heterogeneous data sources, and the method further comprises:

receiving from the at least one of the plurality of heterogeneous data sources, a second type of control message, wherein the second type of control message is comprised of data defined in the one or more of the first type of control messages that was extracted from the at least one of the plurality of heterogeneous data sources according to the data collection scheduling information.

3. The method of claim 1, wherein the first type of control message is further comprised of updated data collection information, wherein updated data collection information is comprised of updated information defining data to be extracted from the at least one of the plurality of heterogeneous data sources.

4. The method of claim 1, further comprising

receiving from the at least one of plurality of heterogeneous data sources, a control message of a third type, wherein the control message of the third type is comprised of data collection status information, wherein the data collection status information is comprised of status information about the process of collecting data from the at least one of plurality of heterogeneous data sources;

and

storing, in a data store, the data collection status information.

5. The method of claim 4, wherein the data collection status information is comprised of information about beginning data extraction, ending data extraction, beginning data encryption, ending data encryption, beginning encrypted data transmission or ending encrypted data transmission.

6. The method of claim 1, further comprising:

receiving from the at least one of plurality of heterogeneous data sources, a control message of a fourth type, wherein the control message of the fourth type is comprised of data collection error information, wherein the data collection error information is comprised of information about an error that occurred during the data collection process from the at least one of plurality of heterogeneous data sources; and

storing, in a data store, the data collection error information.

7. The method of claim 1, wherein each of the plurality of heterogeneous data sources is comprised of data from a data source associated with one of a financial accounting application, a line of business application and a point of sale application.

8. The method of claim 7, wherein the financial accounting application is QuickBooks.

9. The method of claim 1, further comprising:

consolidating the received data that was identified in the first type of control message and that was extracted from each of the plurality of heterogeneous data sources; and

storing the consolidated data in a data store; and

further transforming the consolidated data in the consolidated data store.

10. The method of claim 9, further comprising:

using one of a reporting system and business intelligence system to graphically display the consolidated data.

11. The method of claim 9, further comprising:

comparing data from one of the plurality of heterogeneous data sources with an average or peer group value generated from the consolidated data; and

displaying results of the data comparison.

12. The method of claim 1, wherein the data is extracted via an application programming interface (API).

13. A method, performed by a processor, for controlling the collection of data from a plurality of heterogeneous data sources, wherein the data is extracted in a data source independent manner relative to the plurality of heterogeneous data sources using at least two types of control message to hide implementation details for each of the plurality of heterogeneous data sources, the method comprising:

sending, from at least one of plurality of heterogeneous data sources, a request for a first type of control message, wherein the first type of control message is comprised of data collection information, wherein the data collection information is comprised of information defining data to be collected from the at least one of the plurality of heterogeneous data sources;

receiving one or more of the first type of control messages; and

responsive to receiving the one or more first type of control messages, sending a second type of control message, wherein the second control message type is comprised of data defined in the one or more of the first type of control messages that was extracted from the at least one of the plurality of heterogeneous data sources.

14. The method of claim 13, wherein the first type of control message is further comprised of data collection scheduling information, wherein the data collection scheduling information is comprised of information indicating a schedule for collecting data from the at least one of the plurality of heterogeneous data sources, and the method further comprises:

responsive to receiving the one or more first type of control messages, sending a second type of control message, wherein the second control message type is comprised of data defined in the one or more of the first type of control messages that was extracted from the at least one of the plurality of heterogeneous data sources according to the data collection scheduling information.

15. The method of claim 13, wherein the first type of control message is further comprised of updated data collection information, wherein updated data collection information is comprised of updated information defining data to be extracted from the at least one of the plurality of heterogeneous data sources.

16. The method of claim 13, further comprising

sending a control message of a third type, wherein the control message of the third type is comprised of data collection status information, wherein the data collection status information is comprised of status information about the process of collecting data from the at least one of plurality of heterogeneous data sources.

17. The method of claim 16, wherein the data collection status information is comprised of information about beginning data extraction, ending data extraction, beginning data encryption, ending data encryption, beginning encrypted data transmission or ending encrypted data transmission.

18. The method of claim 13, further comprising:

sending a control message of a fourth type, wherein the control message of the fourth type is comprised of data collection error information, wherein the data collection error information is comprised of information about an error that occurred during the data collection process from the at least one of plurality of heterogeneous data sources.

19. The method of claim 13, wherein each of the plurality of heterogeneous data sources is comprised of data from a data source associated with one of a financial accounting application, a line of business application and a point of sale application.

20. The method of claim 19, wherein the financial accounting software application is QuickBooks.

21. The method of claim 13, further comprising:

storing in a shadow data store the data that was identified in the first type of control message and that was extracted from the at least one of the plurality of heterogeneous data sources; and

verifying the data stored in the shadow data store by comparing the data stored in the shadow data store with the data identified in the first type of control message and stored in the at least one of a plurality of heterogeneous data stores.

22. The method of claim 21, further comprising:

comparing the data stored in the shadow data store with the data identified in the first type of control message and that is stored in the at least one a plurality of heterogeneous data sources;

identifying data stored in the shadow data store that is changed data or new data, wherein changed data is comprised on one or more changed data elements, wherein a changed data element is a data element that is stored in both the shadow data store and the at least one a plurality of heterogeneous data sources, wherein the changed data element has a value in the shadow data store that is different than the value of the changed data element in the in at least one a plurality of heterogeneous data sources, and wherein the new data is one or more data elements that are stored in the at least one a plurality of heterogeneous data sources but not in the shadow database, and wherein deleted data is one or more data elements that are stored in the shadow database but not in the at least one a plurality of heterogeneous data sources; and

sending at least one of the changed data, the new data and the deleted data to a server.

23. The method of claim 13, wherein the data is extracted from the application via an application programming interface (API).

24. A system for controlling the collection of data from a plurality of heterogeneous data sources, wherein the data is extracted in a data source independent manner relative to the plurality of heterogeneous data sources using at least two types of control message to hide implementation details for each of the plurality of heterogeneous data sources, the system comprising:

at least one processor; and

at least one memory storing computer program code, wherein the at least one memory and stored computer program code are configured, with the at least one processor, to cause the system to at least;

receive from at least one of plurality of heterogeneous data sources, a request for a first type of control message, wherein the first type of control message is comprised of data collection information, wherein the data collection information is comprised of a collection definition that defines data to be extracted from the at least one of the plurality of heterogeneous data sources;

responsive to receiving the request, send one or more of the first type of control messages to at least one of the plurality of heterogeneous data sources;

receive, from the at least one of the plurality of heterogeneous data sources, a second type of control message, wherein the second control message type is comprised of data extracted from the at least one of the plurality of heterogeneous data sources that was defined in the one or more of the first type of control messages; and

store, in a data store, the received data that was identified in the first type of control message and that was extracted from the at least one of the plurality of heterogeneous data sources.

25. The system of claim 24, wherein the first type of control message is further comprised of data collection scheduling information, wherein the data collection scheduling information is comprised of information indicating a schedule for collecting data from the at least one of the plurality of heterogeneous data sources, and wherein the at least one memory and stored computer program code are configured, with the at least one processor, to further cause the system to:

receive from the at least one of the plurality of heterogeneous data sources, a second type of control message, wherein the second type of control message is comprised of data defined in the one or more of the first type of control messages that was extracted from the at least one of the plurality of heterogeneous data sources according to the data collection scheduling information.

26. The system of claim 24, wherein the first type of control message is further comprised of updated data collection information, wherein updated data collection information is comprised of updated information defining data to be extracted from the at least one of the plurality of heterogeneous data sources.

27. The system of claim 24, wherein the at least one memory and stored computer program code are configured, with the at least one processor, to further cause the system to:

consolidate the received data that was identified in the first type of control message and that was extracted from each of the plurality of heterogeneous data sources; and

store the consolidated data in a data store; and

further transform the consolidated data in the consolidated data store.

28. The system of claim 27, wherein the at least one memory and stored computer program code are configured, with the at least one processor, to further cause the system to:

use one of a reporting system and business intelligence system to graphically display the consolidated data.

29. The system of claim 27, wherein the at least one memory and stored computer program code are configured, with the at least one processor, to further cause the system to:

compare data from one of the plurality of heterogeneous data sources with an average or peer group value generated from the consolidated data; and

display results of the data comparison.

30. A system for controlling the collection of data from a plurality of heterogeneous data sources, wherein the data is extracted in a data source independent manner relative to the plurality of heterogeneous data sources using at least two types of control message to hide implementation details for each of the plurality of heterogeneous data sources, the system comprising:

at least one processor; and

at least one memory storing computer program code, wherein the at least one memory and stored computer program code are configured, with the at least one processor, to cause the system to at least;

send, from at least one of plurality of heterogeneous data sources, a request for a first type of control message, wherein the first type of control message is comprised of data collection information, wherein the data collection information is comprised of information defining data to be collected from the at least one of the plurality of heterogeneous data sources;

receive one or more of the first type of control messages; and

responsive to receiving the one or more first type of control messages, send a second type of control message, wherein the second control message type is comprised of data defined in the one or more of the first type of control messages that was extracted from the at least one of the plurality of heterogeneous data sources.