QUERY-LEVEL ACCESS TO EXTERNAL PETABYTE-SCALE DISTRIBUTED FILE SYSTEMS

• US 20140040310A1
• Filed: 08/01/2012
• Published: 02/06/2014
• Est. Priority Date: 08/01/2012
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: HANCKEL, Robert, SWART, Garret
• Granted Patent: US 9,589,036 B2
• Time in Patent Office: Days
• US Class Current: 707/770
• CPC Class Codes: G06F 16/168 Details of user interfaces specifically adapted to file systems, e.g. browsing and visualisation, 2d or 3d GUIs query results presentation G06F16/156
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/24532 of parallel queries
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/25 Integrating or interfacing systems involving database management systems
• CPC Class Codes: G06F 16/252 between a Database Management System and a front-end application
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor

Abstract

A system to implement query-level access by a database engine to an external distributed file system by identifying a results file location of one or more results files on the external distributed file system, and storing the results file locations in external table files on the database engine for subsequent use during retrieval of data from the results files. The database engine serves to process queries where the query specifies the external table (which in turn references locations of the results files). Execution of the query streams data from the external distributed file system into the database engine. The data from the external distributed file system is not stored in the external table files on the database engine; rather, the external table files specify a location of code or operational directives which, when executed, streams results from the external distributed file system to at least one parallel query engine.

Claims

1. A computer implemented method for query-level access by a database engine to an external distributed file system, the method comprising:

identifying a results file location of one or more results files on the external distributed file system;

storing, in one or more external table files on the database engine, the results file location of at least one of the results files;

storing, in the one or more external table files on the database engine, metadata specifying a location of code to execute operational directives;

executing a query, the query specifying a table corresponding to the at least one of the external table files; and

retrieving data from the external distributed file system using the query.

2. The method of claim 1, wherein the query is an SQL query.

3. The method of claim 1, further comprising streaming results of the query to at least one parallel query engine.

4. The method of claim 3, wherein the metadata specifies a preprocessor codec.

5. The method of claim 4, wherein the streaming comprises executing the preprocessor codec over at least a portion of the results of the query.

6. The method of claim 1, wherein the results file location is a pathname to a file in the external distributed file system.

7. The method of claim 1, further comprising a running a map-reduce process on an external database system to populate data into one or more of the results files.

8. A computer system for query-level access by a database engine to an external distributed file system, comprising:

a computer processor to execute a set of program code instructions; and

a memory to hold the program code instructions, in which the program code instructions comprises program code to perform,

identifying a results file location of one or more results files on the external distributed file system;

storing, in one or more external table files on the database engine, the results file location of at least one of the results files;

storing, in the one or more external table files on the database engine, metadata specifying a location of code to execute operational directives;

executing a query, the query specifying a table corresponding to the at least one of the external table files; and

retrieving data from the external distributed file system using the query.

9. The computer system of claim 8, wherein the query is an SQL query.

10. The computer system of claim 8, further comprising streaming results of the query to at least one parallel query engine.

11. The computer system of claim 10, wherein the metadata specifies a preprocessor codec.

12. The computer system of claim 11, wherein the streaming comprises executing the preprocessor codec over at least a portion of the results of the query.

13. The computer system of claim 8, wherein the results file location is a pathname to a file in the external distributed file system.

14. The computer system of claim 8, further comprising a running a map-reduce process on an external database system to populate data into one or more of the results files.

15. A computer program product embodied in a non-transitory computer readable medium, the computer readable medium having stored thereon a sequence of instructions which, when executed by a processor causes the processor to execute a process to perform query-level access by a database engine to an external distributed file system, the process comprising:

identifying a results file location of one or more results files on the external distributed file system;

storing, in one or more external table files on the database engine, the results file location of at least one of the results files;

storing, in the one or more external table files on the database engine, metadata specifying a location of code to execute operational directives;

executing a query, the query specifying a table corresponding to the at least one of the external table files; and

retrieving data from the external distributed file system using the query.

16. The computer program product of claim 15, wherein the query is an SQL query.

17. The computer program product of claim 15, further comprising streaming results of the query to at least one parallel query engine.

18. The computer program product of claim 17, wherein the metadata specifies a preprocessor codec.

19. The computer program product of claim 18, wherein the streaming comprises executing the preprocessor codec over at least a portion of the results of the query.

20. The computer program product of claim 15, wherein the results file location is a pathname to a file in the external distributed file system.