DECLARATIVE EXTERNAL DATA SOURCE IMPORTATION, EXPORTATION, AND METADATA REFLECTION UTILIZING HTTP AND HDFS PROTOCOLS

• US 20160092476A1
• Filed: 09/24/2015
• Published: 03/31/2016
• Est. Priority Date: 09/26/2014
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Stojanovic, Alexander Sasha, Kreider, Mark, Savolainen, Douglas C.
• Granted Patent: US 10,976,907 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/211 Schema design and management
• CPC Class Codes: G06F 16/2219 Large Object storage; Management thereof
• CPC Class Codes: G06F 16/2379 Updates performed during online database operations; commit processing
• CPC Class Codes: G06F 16/9038 Presentation of query results
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06T 11/206 Drawing of charts or graphs
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]

Abstract

Techniques are disclosure for a data enrichment system that enables declarative external data source importation and exportation. A user can specify via a user interface input for identifying different data sources from which to obtain input data. The data enrichment system is configured to import and export various types of sources storing resources such as URL-based resources and HDFS-based resources for high-speed bi-directional metadata and data interchange. Connection metadata (e.g., credentials, access paths, etc.) can be managed by the data enrichment system in a declarative format for managing and visualizing the connection metadata.

Claims

1. A method comprising:

receiving, by a computing system of a data enrichment system, a request to access a data source from the data enrichment system, wherein the request includes identification information of the data source and a source type of the data source;

determining, by the computing system, using the identification information, that the data enrichment system does not have a connection to the data source;

rendering a user interface to receive connection information for the data source, the connection information based on the source type;

receiving, by the computing system, via the user interface, a set of connection parameters, the set of connection parameters including a connection name, a connection type, a data source location, and credential information;

storing the set of connection parameters in a data repository system accessible to the data enrichment system;

establishing, by the computing system, using the set of connection parameters, a connection between the data enrichment system and the data source; and

generating, by the computing system, a profile for a data set accessed from the data source via the connection.

2. The method of claim 1, further comprising:

generating, by the data enrichment system, a data service corresponding to the request;

storing, using a data structure in the data repository system, the set of connection parameters in association with the data service; and

generating, by the data service, a transformed data set based on the data set accessed from the data source, wherein the transformed data set is generated using the profile.

3. The method of claim 1, wherein the source type includes a cloud-based storage system, a distributed storage system, a web service, or a uniform resource locator (URL).

4. The method of claim 1, further comprising:

querying, using the identification information, the data repository system to determine whether the data enrichment system has a connection to the data source;

upon determining that the data enrichment system has a connection to the data source, accessing from the data repository system, connection information for the connection; and

rendering the user interface to indicate the connection information for the connection to the data source.

5. The method of claim 1, wherein the credential information includes an access key and a password defined for access to the data source.

6. The method of claim 1, further comprising:

accessing, using the identification information, the data repository system to determine that the data enrichment system has a plurality of connections to the data source;

rendering a user interface that identifies each of the plurality of connections to the data source; and

receiving, via the user interface that identifies each of the plurality of connections, input indicating a selection of one of the plurality of connections.

7. The method of claim 1, wherein storing the set of connection parameters and the credential information includes:

generating one or more data structures defining a data source, the one or more data structures including the set of connection parameters and the credential information, wherein the one or more data structures are stored in the data repository system.

8. A data enrichment system comprising:

a plurality of data sources;

a data repository system; and

a cloud computing infrastructure system comprising;

one or more processors communicatively coupled to the plurality of data sources over at least one communication network; and

a memory coupled to the one or more processors, the memory storing instructions to provide a data enrichment service, wherein the instructions, when executed by the one or more processors, cause the one or more processors to;

receive a request to access a data source of the plurality of data sources, wherein the request includes identification information of the data source and a source type of the data source;

determine, using the identification information, that the data enrichment system does not have a connection to the data source;

render a user interface to receive connection information for the data source, the connection information based on the source type;

receive, via the user interface, a set of connection parameters, the set of connection parameters including a connection name, a connection type, a data source location, and credential information;

store the set of connection parameters in a data repository system accessible to the data enrichment system;

establish, using the set of connection parameters, a connection between the data enrichment system and the data source; and

generate a profile for a data set accessed from the data source via the connection

9. The data enrichment system of claim 8, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to:

generate a data service corresponding to the request;

store, using a data structure in the data repository system, the set of connection parameters in association with the data service; and

generate, by the data service, a transformed data set based on the data set accessed from the data source, wherein the transformed data set is generated using the profile.

10. The data enrichment system of claim 8, wherein the source type includes a cloud-based storage system, a distributed storage system, a web service, or a uniform resource locator (URL).

11. The data enrichment system of claim 8, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to:

query, using the identification information, the data repository system to determine whether the data enrichment system has a connection to the data source;

upon determining that the data enrichment system has a connection to the data source, access from the data repository system, connection information for the connection; and

render the user interface to indicate the connection information for the connection to the data source.

12. The data enrichment system of claim 8, wherein the credential information includes an access key and a password defined for access to the data source.

13. The data enrichment system of claim 8, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to:

access, using the identification information, the data repository system to determine that the data enrichment system has a plurality of connections to the data source;

render a user interface that identifies each of the plurality of connections to the data source; and

receive, via the user interface that identifies each of the plurality of connections, input indicating a selection of one of the plurality of connections.

14. The data enrichment system of claim 8, wherein storing the set of connection parameters and the credential information includes:

generating one or more data structures defining a data source, the one or more data structures including the set of connection parameters and the credential information, wherein the one or more data structures are stored in the data repository system.

15. A non-transitory computer readable storage medium including instructions stored thereon which, when executed by one or more processors, cause the one or more processors to:

receive, by a computing system of a data enrichment system, a request to access a data source from the data enrichment system, wherein the request includes identification information of the data source and a source type of the data source;

determine, by the computing system, using the identification information, that the data enrichment system does not have a connection to the data source;

render a user interface to receive connection information for the data source, the connection information based on the source type;

receive, by the computing system, via the user interface, a set of connection parameters, the set of connection parameters including a connection name, a connection type, a data source location, and credential information;

store the set of connection parameters in a data repository system accessible to the data enrichment system;

establish, by the computing system, using the set of connection parameters, a connection between the data enrichment system and the data source; and

generate, by the computing system, a profile for a data set accessed from the data source via the connection.

16. The non-transitory computer readable storage medium of claim 15, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to:

generate, by the data enrichment system, a data service corresponding to the request;

store, using a data structure in the data repository system, the set of connection parameters in association with the data service; and

generate, by the data service, a transformed data set based on the data set accessed from the data source, wherein the transformed data set is generated using the profile.

17. The non-transitory computer readable storage medium of claim 15, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to:

query, using the identification information, the data repository system to determine whether the data enrichment system has a connection to the data source;

upon determining that the data enrichment system has a connection to the data source, access from the data repository system, connection information for the connection; and

render the user interface to indicate the connection information for the connection to the data source.

18. The non-transitory computer readable storage medium of claim 15, wherein the credential information includes an access key and a password defined for access to the data source.

19. The non-transitory computer readable storage medium of claim 15, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to:

access, using the identification information, the data repository system to determine that the data enrichment system has a plurality of connections to the data source;

render a user interface that identifies each of the plurality of connections to the data source; and

receive, via the user interface that identifies each of the plurality of connections, input indicating a selection of one of the plurality of connections.

20. The non-transitory computer readable storage medium of claim 15, wherein storing the set of connection parameters and the credential information includes:

generating one or more data structures defining a data source, the one or more data structures including the set of connection parameters and the credential information, wherein the one or more data structures are stored in the data repository system.