Method for connecting a relational data store's meta data with Hadoop

• US 10,540,330 B1
• Filed: 03/29/2013
• Issued: 01/21/2020
• Est. Priority Date: 02/25/2013
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Tiwari, Sameer, Bhandarkar, Milind Arun
• Primary Examiner(s): Pham, Hung Q
• Application Number:
  US13/853,438
• Time in Patent Office: 2,489 Days
• Field of Search: None
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: G06F 16/11 File system administration, e.g. details of archiving or snapshots file system backup G06F11/14
• CPC Class Codes: G06F 16/113 Details of archiving lifecycle management in storage systems G06F3/0649; backup systems G06F11/1446
• CPC Class Codes: G06F 16/148 File search processing
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/1858 Parallel file systems, i.e. file systems supporting multiple processors
• CPC Class Codes: G06F 16/217 Database tuning G06F16/2282 takes precedence; database performance monitoring G06F11/3409
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/24524 Access plan code generation and invalidation; Reuse of access plans
• CPC Class Codes: G06F 16/2453 Query optimisation
• CPC Class Codes: G06F 16/24532 of parallel queries
• CPC Class Codes: G06F 16/24542 Plan optimisation
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/43 Querying
• CPC Class Codes: G06F 16/907 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: H04L 65/60 Network streaming of media packets
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H05K 999/99 dummy group

Abstract

A system for sharing a metadata store between a relational database and an unstructured data source is disclosed. The unstructured data source may comprise a Hadoop system with a Hadoop Distributed Files System.

Claims

1. A system for managing metadata, the system comprising:

a network resource configured to store database metadata and to provide a shared catalog service to one or more data sources, wherein the shared catalog manages metadata for the one or more data sources, and wherein the one or more data sources include at least one data source comprising unstructured data and at least one data source comprising structured data;

a first data source in communication with the network resource to access the shared catalog service, wherein the first data source comprises a memory storing unstructured data; and

a second data source in communication with the network resource to access the shared catalog service, wherein the second data source comprises memory storing structured data,

wherein in the event that a query is submitted to the system, the system determines that the query requires information from one or more of the first data source and the second data source, the first data source is configured to retrieve a first database schema corresponding to the first data source from the network resource via the shared catalog service and the first database schema is applied to the unstructured data comprised in the first data source, and the second data source is configured to retrieve a second database schema corresponding to the second data source from the network resource via the shared catalog service and the second database schema is applied to the structured data comprised in the second data source, and

wherein a response to the query is generated using the unstructured data of the first data source and the structured data of the second data source.

2. The system as recited in claim 1, wherein the first data source communicates with the shared catalog service through an adaptor.

3. The system as recited in claim 1, wherein the unstructured data is stored in a distributed file system.

4. The system as recited in claim 1, wherein the second data source comprises a relational database.

5. The system as recited in claim 1, wherein database metadata comprises a database schema.

6. The system as recited in claim 5, wherein the database schema comprises database table definitions.

7. The system of as recited in claim 5, wherein the first data source applies the first database schema to unstructured data.

8. The system as recited in claim 5, wherein a database scheme corresponding to the first database schema and the second database schema is defined by both the first data source and the second data source.

9. The system as defined in claim 5, wherein the first database schema and the second database schema are stored in a relational database.

10. The system as defined in claim 9, wherein the first database schema is accessed using a Java Database Connectivity (“

JDBC”

) connection.

11. The system as defined in claim 10, wherein the first data source accesses the first database schema through an adapter, wherein the adapter is configured to convert an Apache Thrift®

connection to the JDBC connection.

12. The system as recited in claim 1, wherein the first data source, the second data source, and the shared catalog service operate on a computer comprising a processor and a non-transitory computer readable medium.

13. The system as recited in claim 1, wherein the database metadata comprises table definitions, functions, users, views, and indexes.

14. The system of claim 1, further comprising:

a client device,

wherein in the event that the first database schema and the second database schema are the same and that the first database schema is applied to the unstructured data of the first data source and the second database schema is applied to the structured data of the second data source, the client device can manipulate or interface with the unstructured data of the first data source and the structured data of the second data source.

15. The system of claim 14, wherein the client device can manipulate or interface with the unstructured data of the first data source and the structured data of the second data source using a SQL command.

16. The system of claim 1, wherein the database metadata comprises a plurality of database schemas.

17. The system of claim 16, wherein a third data source communicates with the network resource via the shared catalog service to retrieve a third database schema corresponding to a configuration of data stored at the third data source from among the plurality of database schemas.

18. The system of claim 1, wherein one or more of the first data source and the second data source are in communication with the network resource via an adaptor that receives one or more calls for a corresponding database schema, the one or more calls being made using a protocol that is native to the corresponding one of the first data source and the second data source, and wherein the adaptor converts the one or more calls to a protocol associated with shared catalog service.

19. The system of claim 1, wherein application of the first database schema to the unstructured data comprised in the first data source enables a Relationship Database Management System (RDBMS) command to be run against the unstructured data.

20. The system of claim 1, wherein the shared catalog service accesses the first database schema and the second database schema using a Relationship Database Management System (RDBMS) query.

21. The system of claim 1, wherein the first data source retrieves the first database schema based at least in part on querying the network resource for the first database schema, and wherein the second data source retrieves the second database schema based at least in part on querying the network resource for the second database schema.

22. The method of claim 1, wherein the second data source makes one or more calls directly to the network resource, and the first data source makes one or more calls to the network resource using an adaptor that receives the one or more calls from the first data source using a protocol that is native to the first data source.

23. The method of claim 22, wherein the adaptor converts the one or more calls from the first data source to a protocol used by the network resource.

24. A method for establishing a system for managing metadata, the method comprising:

receiving a query at a first data source;

determining that the query requires information from one or more of the first data source and the second data source, wherein the first data source in communication with a shared catalog service via a network resource, wherein the shared catalog manages metadata for the one or more data sources, and wherein the one or more data sources include at least one data source comprising unstructured data and at least one data source comprising structured data, wherein the network resource is configured to store database metadata and to provide the shared catalog service, wherein the shared catalog service is in communication with a second data source, wherein the first data source comprises unstructured data, and wherein the second data source comprises structured data;

retrieving, by one or more processors, a first database schema corresponding to the first data source from a network resource via the shared catalog service;

applying, by one or more processors, the first database schema to unstructured data of the first data source;

retrieving, by one or more processors, a second database schema corresponding to the second data source from the network resource via the shared catalog service;

applying, by one or more processors, the second database schema to structured data of the second data source;

processing, by one or more processors, the request; and

generating a response to a query using the unstructured data of the first data source and the structured data of the second data source.

25. The method of claim 24, wherein the generating the response to the query using the unstructured data of the first data source and the structured data of the second data source comprises performing a JOIN operation between the unstructured data and the structured data and returning a result of the JOIN operation.

26. A computer program product for establishing a system for managing metadata, comprising a non-transitory computer readable medium having program instructions embodied therein for:

receiving a query at a first data source;

determining that the query requires information from one or more of the first data source and the second data source, wherein the first data source in communication with a shared catalog service, wherein the shared catalog service is provided by a network resource, wherein the shared catalog manages metadata for the one or more data sources, and wherein the one or more data sources include at least one data source comprising unstructured data and at least one data source comprising structured data, wherein the shared catalog service is in communication with a second data source, wherein the first data source comprises unstructured data, and wherein the second data source comprises structured data;

retrieving a first database schema from the network resource via the shared catalog service;

applying the first database schema to unstructured data of the first data source;

retrieving a second database schema from the network resource via the shared catalog service;

applying the second database schema to structured data of the second data source;

processing the request; and

generating a response to a query using the unstructured data of the first data source and the structured data of the second data source.