Clinical data aggregation architecture and platform

• US 10,642,854 B2
• Filed: 10/09/2017
• Issued: 05/05/2020
• Est. Priority Date: 10/07/2016
• Status: Active Grant

• Current Assignee: Pankaj Manon, Sudeep Pattnaik
• Original Assignee: Pankaj Manon, Sudeep Pattnaik
• Inventors: Pattnaik, Sudeep, Manon, Pankaj
• Primary Examiner(s): Alam, Shahid A
• Assistant Examiner(s): Lodhi, Andalib F
• Application Number:
  US15/727,930
• Publication Number:
  US 20180101584A1
• Time in Patent Office: 939 Days
• CPC Class Codes: G06F 16/244 Grouping and aggregation
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/86 Mapping to a database
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

A clinical data aggregation system and method, comprising ingesting, transforming and storing data in a clinical data lake. The present invention uses an Extract Load and Transform (ELT) rather than traditional Extract Transform and Load (ETL) design principle. The data hub platform leverages modern noSQL databases which makes the platform highly flexible to configure studies with any design complexity with relative ease.

Claims

1. A system for loading and mapping source data into a predetermined standard format, the system comprising:

a data hub configured to receive the source data from one or more external databases and to store the received source data;

a graphical user interface comprised of multiples screens for displaying on a display screen, the graphical user interface having;

i) an area having a user selectable list of data formats of the source data;

ii) an area configured to allow a user to input a plurality of key identifiers for the source data, wherein each key identifier identifies a predetermined data item or items of the source data;

iii) an area for setting mapping rules that are operative to map a plurality of data items to a plurality of target fields or variables of the predetermined standard format;

iv) an area for inputting a plurality of expressions, wherein each of the expressions are used to transform a predetermined data item or items of the source data into a target value;

a hardware processing system, the hardware processing system programmed with instructions for executing on the hardware processing system for;

     1) extracting data items of the source data from the one or more external databases;

     2) loading the extracted data items into the data hub as pairs of key identifier and value for each data item or items; and

     3) transforming the loaded source data using the mapping rules into a predetermined standard format.

2. The system of claim 1, wherein the key is a unique identifier and the value is a value identified by the key.

3. The system of claim 1, wherein the graphical user interface has an area for defining the characteristics of the incoming file.

4. The system of claim 1, wherein the hardware processing system is programmed with instructions for executing on the hardware processing system to automatically configure key identifiers for source data that are in a standard data format.

5. The system of claim 1, wherein the hardware processing system is programmed with instructions for executing on the hardware processing system to:

1) compare loaded source data with stored ontology data; and

     2) map data items from the source data to variables of the predetermined standard format.

6. The system of claim 1, wherein the hardware processing system is programmed with instructions for executing on the hardware processing system to compare the source data with a model of the predetermined standard format using pattern matching algorithms to automatically map the source data to the predetermined standard format.

7. The system of claim 6, wherein the hardware processing system is configured to learn previously applied mapping rules and to automatically apply said mapping rules to future occurrences of similar data transformations.

8. The system of claim 1, wherein the hardware processing system is programmed with instructions for executing on the hardware processing system to compare the metadata of the source data with the variables of the predetermined standard format using pattern matching algorithms to automatically map the source data to the predetermined standard format.

9. The system of claim 1, wherein the hardware processing system is programmed with instructions for executing on the hardware processing system to use configurable matching algorithms to compare metadata of the source data with a stored mapping library to automatically map the source data to the predetermined standard format.

10. The system of claim 9, wherein the stored mapping library is a library of mapping between domains of the predetermined standard format with source data metadata.

11. A system for loading and mapping source data into a predetermined standard format, the system comprising:

a data hub configured to receive the source data from one or more external databases and to store the received source data;

an inbound data manager, as a component of the data hub, that acts as an interface between the data hub and the one or more external databases and is configured to load the source data to the data hub with no transformation to the source data;

a graphical user interface comprised of multiple screens for displaying on a display screen, the graphical user interface having;

i) a first area having a user selectable list of types of source data that can be loaded by the system;

ii) a second area having a user selectable list of data formats of the source data;

iii) a third area configured to allow a user to input a plurality of key identifiers for the source data, wherein each key identifier identifies a predetermined data item or items of the source data;

iv) a fourth area for setting mapping rules by mapping a plurality of data items to a plurality of target fields or variables of the predetermined standard format;

v) a fifth area for inputting a plurality of expressions, wherein each of the expressions are used to transform a predetermined data item or items of the source data into a target value;

a hardware processing system, the hardware processing system programmed with instructions for executing on the hardware processing system for;

     1) extracting data items of the source data from the one or more external databases;

     2) loading the extracted data items of the source data without transformation into the data hub as pairs of key identifier and value for each data item or items;

     3) transforming the loaded source data using the mapping rules into a predetermined standard format;

     4) comparing loaded source data with stored ontology data; and

     5) mapping data items from the source data to variables of the predetermined standard format.

12. The system of claim 11, wherein the key is a unique identifier and the value is a value identified by the key.

13. The system of claim 11, wherein the graphical user interface has an area for defining the characteristics of the incoming file.

14. The system of claim 11, wherein the hardware processing system is programmed with instructions for executing on the hardware processing system to automatically configure key identifiers for source data that are in a standard data format.

15. The system of claim 11, wherein the hardware processing system is programmed with instructions for executing on the hardware processing system to compare the source data with a model of the predetermined standard format using pattern matching algorithms to automatically map the source data to the predetermined standard format.

16. The system of claim 15, wherein the hardware processing system is configured to learn previously applied mapping rules and to automatically apply said mapping rules to future occurrences of similar data transformations.

17. The system of claim 11, wherein the hardware processing system is programmed with instructions for executing on the hardware processing system to compare the metadata of the source data with the variables of the predetermined standard format using pattern matching algorithms to automatically map the source data to the predetermined standard format.

18. The system of claim 11, wherein the hardware processing system is programmed with instructions for executing on the hardware processing system to use configurable matching algorithms to compare metadata of the source data with a stored mapping library to automatically map the source data to the predetermined standard format.

19. The system of claim 18, wherein the stored mapping library is a library of mapping between domains of the predetermined standard format with source data metadata.

20. A system for loading and mapping source data into a predetermined standard format, the system comprising:

a data hub configured to receive the source data from one or more external databases and to store the received source data in a noSQL database format;

an inbound data manager, as a component of the data hub, that acts as an interface between the data hub and the one or more external databases and is configured to load the source data to the data hub with no transformation to the source data;

a graphical user interface comprised of multiple screens for displaying on a display screen, the graphical user interface having;

i) a first area having a user selectable list of types of source data that can be loaded by the system;

ii) a second area having a user selectable list of data formats of the source data;

iii) a third area configured to allow a user to input a plurality of key identifiers for the source data, wherein each key identifier identifies a predetermined data item or items of the source data;

iv) a fourth area for setting configurable mapping rules by mapping a plurality of data items to a plurality of target fields or variables of the predetermined standard format;

v) a fifth area for inputting a plurality of expressions, wherein each of the expressions are used to transform a predetermined data item or items of the source data into a target value;

a hardware processing system, the hardware processing system programmed with instructions for executing on the hardware processing system for;

     1) extracting data items of the source data from the one or more external databases;

     2) loading the extracted data items of the source data without transformation into the data hub as pairs of key identifier and value for each data item or items;

     3) transforming the loaded source data using the configurable mapping rules and expressions through a built-in rules engine into a predetermined standard format;

     4) comparing loaded source data with stored ontology data and mapping data items from the source data to variables of the predetermined standard format;

     5) comparing the source data with a model of the predetermined standard format and using pattern matching algorithms to automatically map the source data to the predetermined standard format; and

     6) learning previously applied mapping rules and automatically applying said mapping rules to future occurrences of similar data transformations.

21. The system of claim 20, wherein the key is a unique identifier and the value is a valued identified by the key.

22. The system of claim 20, wherein the graphical user interface has an area for defining the characteristics of the incoming file.

23. The system of claim 20, wherein the hardware processing system is programmed with instructions for executing on the hardware processing system to automatically configure key identifiers for source data that are in a standard data format.

24. The system of claim 20, wherein the hardware processing system is programmed with instructions for executing on the hardware processing system to compare the source data with a model of the predetermined standard format using pattern matching algorithms to automatically map the source data to the predetermined standard format.

25. The system of claim 20 wherein the hardware processing system is programmed with instructions for executing on the hardware processing system to use configurable matching algorithms to compare metadata of the source data with a stored mapping library to automatically map the source data to the predetermined standard format.

26. The system of claim 25, wherein the stored mapping library is a library of mapping between domains of the predetermined standard format with source data metadata.