Persistent query dispatch and execution architecture

• US 10,212,257 B2
• Filed: 05/14/2016
• Issued: 02/19/2019
• Est. Priority Date: 05/14/2015
• Status: Active Grant

• Current Assignee: Deephaven Data Labs, LLC
• Original Assignee: Deephaven Data Labs, LLC
• Inventors: Teodorescu, Radu, Caudy, Ryan, Kent, IV, David R., Wright, Charles, Dorfman, Nathan, Ries, Brian
• Primary Examiner(s): Willoughby, Alicia M
• Application Number:
  US15/155,005
• Publication Number:
  US 20160335323A1
• Time in Patent Office: 1,011 Days
• Field of Search: None
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/1469 Backup restoration techniques
• CPC Class Codes: G06F 12/0261 using reference counting
• CPC Class Codes: G06F 12/084 with a shared cache
• CPC Class Codes: G06F 12/1483 using an access-table, e.g. matrix or list
• CPC Class Codes: G06F 15/17331 Distributed shared memory [DSM], e.g. remote direct memory access [RDMA]
• CPC Class Codes: G06F 16/113 Details of archiving lifecycle management in storage systems G06F3/0649; backup systems G06F11/1446
• CPC Class Codes: G06F 16/144 Query formulation
• CPC Class Codes: G06F 16/162 Delete operations erasing in storage systems G06F3/0652
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/221 Column-oriented storage; Management thereof
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2237 Vectors, bitmaps or matrices
• CPC Class Codes: G06F 16/2246 Trees, e.g. B+trees
• CPC Class Codes: G06F 16/2255 Hash tables
• CPC Class Codes: G06F 16/2264 Multidimensional index structures
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/2282 Tablespace storage structures; Management thereof
• CPC Class Codes: G06F 16/2291 : User-Defined Types; Storage management thereof
• CPC Class Codes: G06F 16/23 : Updating
• CPC Class Codes: G06F 16/2308 : Concurrency control transaction processing G06F9/466
• CPC Class Codes: G06F 16/2358 : Change logging, detection, and notification replication G06F16/27
• CPC Class Codes: G06F 16/2365 : Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/2372 : Updates performed during offline database operations
• CPC Class Codes: G06F 16/2379 : Updates performed during online database operations; commit processing
• CPC Class Codes: G06F 16/242 : Query formulation
• CPC Class Codes: G06F 16/2428 : Query predicate definition using graphical user interfaces, including menus and forms G06F16/2423 takes precedence
• CPC Class Codes: G06F 16/245 : Query processing
• CPC Class Codes: G06F 16/2453 : Query optimisation
• CPC Class Codes: G06F 16/24534 : Query rewriting; Transformation
• CPC Class Codes: G06F 16/24535 : of sub-queries or views
• CPC Class Codes: G06F 16/24537 : of operators
• CPC Class Codes: G06F 16/24539 : using cached or materialised query results
• CPC Class Codes: G06F 16/2455 : Query execution
• CPC Class Codes: G06F 16/24553 : of query operations
• CPC Class Codes: G06F 16/2456 : Join operations
• CPC Class Codes: G06F 16/24561 : Intermediate data storage techniques for performance improvement
• CPC Class Codes: G06F 16/2457 : with adaptation to user needs
• CPC Class Codes: G06F 16/24575 : using context
• CPC Class Codes: G06F 16/248 : Presentation of query results
• CPC Class Codes: G06F 16/254 : Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/27 : Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/278 : Data partitioning, e.g. horizontal or vertical partitioning
• CPC Class Codes: G06F 16/285 : Clustering or classification
• CPC Class Codes: G06F 16/9024 : Graphs; Linked lists G06F16/9027 takes precedence
• CPC Class Codes: G06F 16/907 : Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/951 : Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9535 : Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 : Presentation of query results
• CPC Class Codes: G06F 16/9566 : URL specific, e.g. using aliases, detecting broken or misspelled links
• CPC Class Codes: G06F 16/9574 : of access to content, e.g. by caching
• CPC Class Codes: G06F 17/40 : Data acquisition and logging for input to computer G06F3/00
• CPC Class Codes: G06F 21/00 : Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
• CPC Class Codes: G06F 21/6209 : to a single file or object, e.g. in a secure envelope, encrypted and accessed using a key, or with access control rules appended to the object itself
• CPC Class Codes: G06F 2201/805 : Real-time
• CPC Class Codes: G06F 2201/84 : Using snapshots, i.e. a logical point-in-time copy of the data
• CPC Class Codes: G06F 2212/1052 : Security improvement
• CPC Class Codes: G06F 2212/154 : Networked environment
• CPC Class Codes: G06F 2212/163 : Server or database system
• CPC Class Codes: G06F 2212/60 : Details of cache memory
• CPC Class Codes: G06F 3/0481 : based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0482 : Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0483 : Interaction with page-structured environments, e.g. book metaphor
• CPC Class Codes: G06F 3/04847 : Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/0485 : Scrolling or panning
• CPC Class Codes: G06F 3/04895 : Guidance during keyboard input operation, e.g. prompting
• CPC Class Codes: G06F 3/0605 : by facilitating the interaction with a user or administrator
• CPC Class Codes: G06F 3/0656 : Data buffering arrangements
• CPC Class Codes: G06F 3/067 : Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 40/117 : Tagging; Marking up details of markup languages G06F40/143; Designating a block; Setting of attributes style sheets, e.g. eXtensible Stylesheet Language Transformation [XSLT], G06F40/154
• CPC Class Codes: G06F 40/134 : Hyperlinking
• CPC Class Codes: G06F 40/166 : Editing, e.g. inserting or deleting
• CPC Class Codes: G06F 40/174 : Form filling; Merging
• CPC Class Codes: G06F 40/177 : of tables; using ruled lines
• CPC Class Codes: G06F 40/18 : of spreadsheets form-filling G06F40/174
• CPC Class Codes: G06F 40/183 : Tabulation, i.e. one-dimensional positioning
• CPC Class Codes: G06F 40/216 : using statistical methods
• CPC Class Codes: G06F 40/274 : Converting codes to words; Guess-ahead of partial word inputs
• CPC Class Codes: G06F 8/30 : Creation or generation of source code
• CPC Class Codes: G06F 8/41 : Compilation
• CPC Class Codes: G06F 8/427 : Parsing
• CPC Class Codes: G06F 8/60 : Software deployment
• CPC Class Codes: G06Q 40/04 : Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
• CPC Class Codes: H04L 12/18 : for broadcast or conference , e.g. multicast
• CPC Class Codes: H04L 51/046 : Interoperability with other network applications or services
• CPC Class Codes: H04L 51/212 : using filtering or selective blocking
• CPC Class Codes: H04L 61/5069 : for group communication, multicast communication or broadcast communication
• CPC Class Codes: H04L 63/101 : Access control lists [ACL]
• CPC Class Codes: H04L 63/102 : Entity profiles
• CPC Class Codes: H04L 67/01 : Protocols
• CPC Class Codes: H04L 67/1001 : for accessing one among a plurality of replicated servers
• CPC Class Codes: H04L 67/141 : Setup of application sessions admission control or resource allocation in data switching networks H04L47/70
• CPC Class Codes: H04L 67/34 : involving the movement of software or configuration parameters network booting or remote initial program loading [RIPL] G06F9/4416
• CPC Class Codes: H04L 67/56 : Provisioning of proxy services store-and-forward switching systems in data switching networks H04L12/54
• CPC Class Codes: H04L 67/566 : Grouping or aggregating service requests, e.g. for unified processing
• CPC Class Codes: H04L 67/568 : Storing data temporarily at an intermediate stage, e.g. caching
• CPC Class Codes: H04L 67/5681 : Pre-fetching or pre-delivering data based on network characteristics
• CPC Class Codes: H04L 69/16 : Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]

Abstract

The disclosed subject matter includes systems, methods, and computer readable medium for improving performance of a computer data system. An electronic request for a remote query processor (RQP) can be sent from the persistent query controller to a remote query dispatcher (RQD) executing on a query server computer. The request can include parameters for configuring the RQP and an operating environment for the RQP. The RQD can automatically attempt to allocate an isolated operating environment for the RQP and to prepare the RQP on the query server computer. When the RQP is prepared, performing: providing the controller with an address assignment of the RQP; automatically connecting from the controller to the RQP via a network; transmitting a persistent database query electronically from the controller to the RQP; publishing persistent database query configuration information including a query state and the RQP address assignment; and connecting from a client to the RQP.

Claims

1. A computer data system having a persistent query dispatch and execution architecture, the system comprising:

one or more processors each being a hardware processor;

computer readable storage coupled to the one or more processors, the computer readable storage having stored thereon instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including;

sending an electronic request for a remote query processor from a persistent query controller to a remote query dispatcher executing on a query server computer, wherein the request includes parameters for configuring the remote query processor and an operating environment for the remote query processor, the parameters including a parameter selected from the group consisting of;

a shell variable to be set on the operating environment for the remote query processor, and

a classpath specifying a location of code to be used by the remote query processor;

attempting, at the remote query dispatcher, to allocate an isolated operating environment for the remote query processor and to start execution of the remote query processor on the query server computer;

when the remote query processor is started, performing operations including;

providing the persistent query controller with an address assignment of the remote query processor or of a proxy machine in communication with the remote query processor, the address assignment identifying a specific address of the query server computer or of the proxy machine available to connect electronically over an electronic communications network;

automatically connecting from the persistent query controller to the remote query processor via the electronic communications network;

transmitting a persistent database query electronically from the persistent query controller to the remote query processor;

publishing persistent database query configuration information including a state of the persistent database query and the address assignment of the remote query processor;

connecting from a client to the remote query processor via the electronic communications network;

executing the persistent database query;

after the executing the persistent database query has started, receiving, from a second client different than the client, an instruction to the persistent query controller to modify code executed for the persistent database query;

determining whether to allow the connection by the client to the remote query processor based on access control information specific to the persistent database query; and

when the connection by the client is allowed;

sending a request to perform an administrative operation with respect to the persistent database query from the client to the persistent query controller, the administrative operation being an instruction to modify one or more of the parameters for configuring the remote query processor and the operating environment for the remote query processor, and

determining whether the client is authorized to perform the administrative operation based on the access control information.

2. The system of claim 1, wherein the operations further comprise:

when the connection by the client is allowed;

receiving at the client at least a portion of a current result of the persistent database query from the remote query processor.

3. The system of claim 1, wherein the operations further comprise:

when the connection by the client is allowed;

filtering, based on access control information, a current result of the persistent database query requested by the client from the remote query processor; and

sending at least a portion of the filtered current result of the persistent database query to the client.

4. The system of claim 1, wherein the operations further comprise:

sending, from the second client different than the client, an instruction to the persistent query controller to start, stop, restart, modify parameters, or modify code of the persistent database query.

5. The system of claim 1, wherein the operations further include:

when the connection by the client is allowed;

receiving a result of the persistent database query at the client;

displaying at least a portion of the result at the client via a graphical user interface and/or a console;

receiving at least a portion of an updated result of the persistent database query from the remote query processor; and

responsive to the receiving the at least a portion of the updated result, updating the graphical user interface and/or console to display the at least a portion of the updated result.

6. The system of claim 1, wherein the client is another remote query processor.

7. The system of claim 1, wherein the operations further include:

determining whether the remote query processor rejects the request for a remote query processor from the persistent query controller;

when the remote query dispatcher rejects the request, publishing an indication of the rejection;

detecting, by the remote query processor or remote query dispatcher, an error in the execution of the persistent database query; and

when the remote query processor or remote query dispatcher detects an error in the execution of the persistent database query, publishing an indication of the error.

8. The system of claim 1, wherein the operations further include:

when the connection by the client is allowed;

transmitting an additional query task electronically from the client to the remote query processor;

executing, at the remote query processor, the additional query task; and

receiving at least a portion of a result of the additional query task at the client.

9. The system of claim 1, wherein the operations further include:

periodically providing a liveness indication from the persistent query controller to the remote query dispatcher, and

when the liveness indication is not received after a limited amount of time, stopping the remote query processor.

10. A method for improving performance of a computer data system through control of a persistent query dispatch and execution architecture, the method comprising:

sending an electronic request for a remote query processor from a persistent query controller to a remote query dispatcher executing on a query server computer, wherein the request includes parameters for configuring the remote query processor and an operating environment for the remote query processor, the parameters including a parameter selected from the group consisting of;

a shell variable to be set on the operating environment for the remote query processor, and

a classpath specifying a location of code to be used by the remote query processor;

automatically attempting, at the remote query dispatcher, to allocate an isolated operating environment for the remote query processor and to prepare the remote query processor on the query server computer;

when the remote query processor is prepared, performing operations including;

providing the persistent query controller with an address assignment of the remote query processor or of a proxy machine in communication with the remote query processor, the address assignment identifying a specific address of the query server computer or of the proxy machine available to connect electronically over an electronic communications network;

automatically connecting from the persistent query controller to the remote query processor via the electronic communications network;

transmitting a persistent database query electronically from the persistent query controller to the remote query processor;

publishing persistent database query configuration information including a state of the persistent database query and the address assignment of the remote query processor;

connecting from a client to the remote query processor via the electronic communications network;

executing the persistent database query;

after the executing the persistent database query has started, receiving, from a second client different than the client, an instruction to the persistent query controller to modify code executed for the persistent database query;

determining whether to allow the connection by the client to the remote query processor based on access control information specific to the persistent database query; and

when the connection by the client is allowed;

sending a request to perform an administrative operation with respect to the persistent database query from the client to the persistent query controller, the administrative operation being an instruction to modify one or more of the parameters for configuring the remote query processor and the operating environment for the remote query processor, and

determining whether the client is authorized to perform the administrative operation based on the access control information.

11. The method of claim 10, further comprising:

when the connection by the client is allowed;

receiving at the client at least a portion of a current result of the persistent database query from the remote query processor.

12. The method of claim 10, wherein the operations further comprise:

when the connection by the client is allowed;

filtering, based on access control information, a current result of the persistent database query requested by the client from the remote query processor; and

sending at least a portion of the filtered current result of the persistent database query to the client.

13. The method of claim 10, further comprising:

sending, from the second client different than the client, an instruction to the persistent query controller to start, stop, or restart the persistent database query.

14. The method of claim 10, further comprising:

when the connection by the client is allowed;

receiving a result of the persistent database query at the client;

displaying at least a portion of the result at the client via a graphical user interface and/or a console;

receiving at least a portion of an updated result of the persistent database query from the remote query processor; and

responsive to the receiving the at least a portion of the updated result, updating the graphical user interface and/or console to display the at least a portion of the updated result.

15. The method of claim 10, wherein the client is another remote query processor.

16. The method of claim 10, further comprising:

determining whether the remote query processor rejects the request for a remote query processor from the persistent query controller;

when the remote query dispatcher rejects the request, publishing an indication of the rejection;

detecting, by the remote query processor or remote query dispatcher, an error in the execution of the persistent database query; and

when the remote query processor or remote query dispatcher detects an error in the execution of the persistent database query, publishing an indication of the error.

17. The method of claim 10, further comprising:

when the connection by the client is allowed;

transmitting an additional query task electronically from the client to the remote query processor;

executing, at the remote query processor, the additional query task; and

receiving at least a portion of a result of the additional query task at the client.

18. The method of claim 10, further comprising:

periodically providing a liveness indication from the persistent query controller to the remote query dispatcher, and

when the liveness indication is not received after a limited amount of time, stopping the remote query processor.

19. A nontransitory computer readable medium having stored thereon software instructions that, when executed by one or more processors, cause the processors to perform operations including:

sending an electronic request for a remote query processor from a persistent query controller to a remote query dispatcher executing on a query server computer, wherein the request includes parameters for configuring the remote query processor and an operating environment for the remote query processor, the parameters including a parameter selected from the group consisting of;

a shell variable to be set on the operating environment for the remote query processor, and

a classpath specifying a location of code to be used by the remote query processor;

automatically attempting, at the remote query dispatcher, to allocate an isolated operating environment for the remote query processor and to run-o€

the remote query processor on the query server computer;

when the remote query processor is running, performing operations including;

providing the persistent query controller with an address assignment of the remote query processor or of a proxy machine in communication with the remote query processor, the address assignment identifying a specific address of the query server computer or of the proxy machine available to connect electronically over an electronic communications network;

automatically connecting from the persistent query controller to the remote query processor via the electronic communications network;

transmitting a persistent database query electronically from the persistent query controller to the remote query processor;

publishing persistent database query configuration information including a state of the persistent database query and the address assignment of the remote query processor;

connecting from a client to the remote query processor via the electronic communications network;

executing the persistent database query;

after the executing the persistent database query has started, receiving, from a second client different than the client, an instruction to the persistent query controller to modify code executed for the persistent database query;

determining whether to allow the connection by the client to the remote query processor based on access control information specific to the persistent database query; and

when the connection by the client is allowed;

sending a request to perform an administrative operation with respect to the persistent database query from the client to the persistent query controller, the administrative operation being an instruction to modify one or more of the parameters for configuring the remote query processor and the operating environment for the remote query processor, and

determining whether the client is authorized to perform the administrative operation based on the access control information.

20. The computer readable medium of claim 19, wherein the operations further comprise:

when the connection by the client is allowed;

receiving at the client at least a portion of a current result of the persistent database query from the remote query processor.

21. The computer readable medium of claim 19, wherein the operations further comprise:

when the connection by the client is allowed;

filtering, based on access control information, a current result of the persistent database query requested by the client from the remote query processor; and

sending at least a portion of the filtered current result of the persistent database query to the client.

22. The computer readable medium of claim 19, wherein the operations further comprise:

sending, from the second client different than the client, an instruction to the persistent query controller to start, stop, or restart the persistent database query.

23. The computer readable medium of claim 19, wherein the operations further include:

when the connection by the client is allowed;

receiving a result of the persistent database query at the client;

displaying at least a portion of the result at the client via a graphical user interface and/or a console;

receiving at least a portion of an updated result of the persistent database query from the remote query processor; and

responsive to the receiving the at least a portion of the updated result, updating the graphical user interface and/or console to display the at least a portion of the updated result.

24. The computer readable medium of claim 19, wherein the client is another remote query processor.

25. The computer readable medium of claim 19, wherein the operations further include:

determining whether the remote query processor rejects the request for a remote query processor from the persistent query controller;

when the remote query dispatcher rejects the request, publishing an indication of the rejection;

detecting, by the remote query processor or remote query dispatcher, an error in the execution of the persistent database query; and

when the remote query processor or remote query dispatcher detects an error in the execution of the persistent database query, publishing an indication of the error.

26. The computer readable medium of claim 19, wherein the operations further include:

determining whether the attempting to start a remote query processor was successful; and

when the attempting was not successful, repeating the attempting a limited number of times.

27. The computer readable medium of claim 19, wherein the operations further include:

periodically providing a liveness indication from the persistent query controller to the remote query dispatcher, and

when the liveness indication is not received after a limited amount of time, stopping the remote query processor.