Accelerated query execution within a storage array

• US 9,665,621 B1
• Filed: 03/14/2013
• Issued: 05/30/2017
• Est. Priority Date: 03/14/2013
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Nanda, Nihar, Hushon, John Daniel, Reiner, David Stephen
• Primary Examiner(s): Allen, Brittany N
• Application Number:
  US13/804,703
• Time in Patent Office: 1,538 Days
• Field of Search: 707607
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2453 Query optimisation
• CPC Class Codes: G06F 16/24544 Join order optimisation
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/2456 Join operations
• CPC Class Codes: G06F 16/24561 Intermediate data storage techniques for performance improvement
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP

Abstract

A data storage system comprises a storage array and an associated indexer that may be implemented in whole or in part within the storage array. The storage array comprises storage devices and a controller. The indexer associated with the storage array is configured to generate indexes over data stored in the storage array. The controller comprises a query processing control point configured to utilize the indexes generated by the indexer to accelerate execution of queries directed to the data stored in the storage array. The indexes may be stored in a high-speed memory of the storage array, such as flash memory. The queries executed in the storage array may comprise analytic queries that would otherwise normally be executed in an analytic data store external to the storage array. By providing index-based processing of the queries within the storage array, query processing throughput of the data storage system is substantially improved.

Claims

1. An apparatus comprising:

a storage array comprising a plurality of storage devices and a controller;

an indexer implemented in the storage array and configured to generate indexes over data stored in the storage array; and

an analytic data store;

wherein the controller comprises a query processing control point configured to utilize the indexes generated by the indexer to accelerate execution of queries directed to the data stored in the storage array;

wherein at least a subset of the queries directed to the data stored in the storage array comprise analytic queries generated at least in part utilizing the analytic data store;

wherein the query processing control point is configured to draw information about the data stored in the storage array directly from the indexes and generate a response to at least a portion of a given one of the analytic queries using the information drawn directly from the indexes;

wherein the query processing control point comprises a query director implemented in the storage array using a container of the storage array;

wherein the query director processes queries of at least first and second different types and is configured to differentiate between queries received via at least one component of the analytic data store as being one or more of the first type queries, and queries received via one or more database connectivity connectors as being one or more of the second type queries;

wherein the query director is further configured to;

direct the one or more of the first type queries to the indexes instead of the data, and to direct the one or more of the second type queries to the data instead of the indexes; and

interact with the analytic data store by receiving the one or more of the first type queries from at least one external table of the analytic data store that is configured to generate the one or more of the first type queries to be applied against the indexes via the query director;

wherein the query director is operatively coupled to one or more logs of the storage array for implementation of updates to the indexes within the storage array;

wherein the storage array comprises a compute-capable storage array having integrated compute capabilities and a file system configured to store the indexes; and

wherein the indexer and the controller are implemented in software executed within the storage array.

2. The apparatus of claim 1 wherein the storage array further comprises a memory accessible to the controller and configured to store the indexes.

3. The apparatus of claim 1 wherein the analytic queries are directed against an operational data store comprising the storage array.

4. The apparatus of claim 1 wherein the query processing control point is configured to accelerate execution of the given analytic query by utilizing the information drawn directly from the indexes to identify particular portions of the data stored in the storage array, and generating a response to at least a portion of the given analytic query using the particular portions of the data.

5. The apparatus of claim 1 wherein the query processing control point is configured to accelerate execution of the given analytic query by generating a plurality of subqueries for the given analytic query, processing each of the subqueries utilizing the indexes to obtain intermediate results, combining the intermediate results to form a final result, and returning the final result as a response to the given analytic query.

8. The apparatus of claim 1 wherein the indexer is configured to generate the indexes over at least one unstructured data source so as to permit join processing to be performed over unstructured but indexed data or over combinations of structured data and unstructured but indexed data.

9. The apparatus of claim 1 wherein functionality of the indexer is at least in part federated so as to permit at least a portion of one or more of the indexes to be built utilizing multiple distinct data sources including one or more data sources external to the storage array.

10. The apparatus of claim 1 wherein the indexes are replicated within each of a plurality of storage arrays associated with respective geographically-dispersed physical locations of a data storage system.

11. The apparatus of claim 1 further comprising a query router arranged between one or more applications and at least one of the storage array and an associated database management system.

12. The apparatus of claim 3 wherein the operational data store against which the analytic queries are directed comprises a production online transaction processing (OLTP) system.

13. A data storage system comprising the apparatus of claim 1.

6. An apparatus comprising:

a storage array comprising a plurality of storage devices and a controller; and

an indexer implemented in the storage array and configured to generate indexes over data stored in the storage array;

wherein the controller comprises a query processing control point configured to utilize the indexes generated by the indexer to accelerate execution of queries directed to the data stored in the storage array;

wherein the indexes comprise one or more join indexes and one or more content indexes;

wherein functionality of the indexer is at least in part federated so as to permit at least a portion of one or more of the indexes to comprise data from multiple distinct data sources including one or more data sources external to the storage array;

wherein the query processing control point comprises a query director implemented in the storage array using a container of the storage array;

wherein the query director processes queries of at least first and second different types and is configured to differentiate between queries received via at least one component of an analytic data store as being one or more of the first type queries, and queries received via one or more database connectivity connectors as being one or more of the second type queries;

wherein the query director is further configured to;

direct the one or more of the first type queries to the indexes instead of the data, and to direct the one or more of the second type queries to the data instead of the indexes; and

interact with the analytic data store by receiving the one or more of the first type queries from at least one external table of the analytic data store that is configured to generate the one or more of the first type queries to be applied against the indexes via the query director;

wherein the query director is operatively coupled to one or more logs of the storage array for implementation of updates to the indexes within the storage array;

wherein the storage array comprises a compute-capable storage array having integrated compute capabilities and a file system configured to store the indexes; and

wherein the indexer and the controller are implemented in software executed within the storage array.

7. The apparatus of claim 6 wherein the query processing control point is configured to accelerate execution of a given one of the queries by performing join processing over both the join and content indexes.

20. The apparatus of claim 6 wherein the indexer is configured to generate the indexes over at least one unstructured data source so as to permit join processing to be performed over unstructured but indexed data or over combinations of structured data and unstructured but indexed data.

21. The apparatus of claim 6 wherein the indexes are replicated within each of a plurality of storage arrays associated with respective geographically-dispersed physical locations of a data storage system.

22. The apparatus of claim 6 wherein functionality of the indexer is at least in part federated so as to permit at least a portion of one or more of the indexes to be built utilizing multiple distinct data sources including one or more data sources external to the storage array.

23. The apparatus of claim 6 further comprising a query router arranged between one or more applications and at least one of the storage array and an associated database management system.

26. The apparatus of claim 6 wherein the storage array further comprises a memory accessible to the controller and configured to store the indexes.

14. A method comprising:

generating indexes over data stored in a storage array comprising a plurality of storage devices, wherein an indexer configured to generate the indexes is implemented in the storage array; and

utilizing the indexes to accelerate execution of analytic queries directed to the data stored in the storage array by;

drawing information about the data stored in the storage array directly from the indexes; and

generating a response to at least a portion of a given one of the analytic queries using the information drawn directly from the indexes;

wherein the generating and utilizing are performed at least in part by a controller comprising a query processing control point implemented within the storage array; and

wherein the query processing control point comprises a query director implemented in the storage array using a container of the storage array;

wherein the query director processes queries of at least first and second different types and is configured to differentiate between queries received via at least one component of an analytic data store as being one or more of the first type queries, and queries received via one or more database connectivity connectors as being one or more of the second type queries;

wherein the query director is further configured to;

direct the one or more of the first type queries to the indexes instead of the data, and to direct the one or more of the second type queries to the data instead of the indexes; and

interact with the analytic data store by receiving the one or more of the first type queries from at least one external table of the analytic data store that is configured to generate the one or more of the first type queries to be applied against the indexes via the query director;

wherein the query director is operatively coupled to one or more logs of the storage array for implementation of updates to the indexes within the storage array;

wherein the storage array comprises a compute-capable storage array having integrated compute capabilities and a file system configured to store the indexes; and

wherein the indexer and the controller are implemented in software executed within the storage array.

15. The method of claim 14 further comprising updating the indexes at least one of periodically and responsive to changes in stored data.

16. The method of claim 14 wherein utilizing the indexes to accelerate execution of analytic queries directed to the data stored in the storage array comprises:

utilizing the information drawn directly from the indexes to identify particular portions of the data stored in the storage array; and

generating a response to at least a portion of the given analytic query using the particular portions of the data.

17. The method of claim 14 wherein utilizing the indexes to accelerate execution of analytic queries directed to the data stored in the storage array comprises:

receiving the given analytic query;

generating a plurality of subqueries for the given analytic query;

processing each of the subqueries utilizing the indexes to obtain intermediate results;

combining the intermediate results to form a final result; and

returning the final result as a response to the given analytic query.

18. A computer program product comprising a processor-readable storage medium having encoded therein executable code of one or more software programs, wherein the one or more software programs when executed by a processor of the storage array implement the method of claim 14.

19. The method of claim 14 wherein the indexes comprise one or more join indexes and one or more content indexes.

24. The method of claim 14 wherein the analytic queries are directed against an operational data store comprising the storage array.

25. The method of claim 24 wherein the operational data store against which the analytic queries are directed comprises a production online transaction processing (OLTP) system.