Computer data distribution architecture

• US 10,003,673 B2
• Filed: 03/20/2017
• Issued: 06/19/2018
• Est. Priority Date: 05/14/2015
• Status: Active Grant

• Current Assignee: Deephaven Data Labs, LLC
• Original Assignee: Illumon LLC
• Inventors: Caudy, Ryan, Kent, IV, David R., Wright, Charles, Teodorescu, Radu
• Primary Examiner(s): Barot, Bharat N
• Application Number:
  US15/464,314
• Publication Number:
  US 20170195461A1
• Time in Patent Office: 456 Days
• Field of Search: 709201-203, 709212-219
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/1469 Backup restoration techniques
• CPC Class Codes: G06F 12/0261 using reference counting
• CPC Class Codes: G06F 12/084 with a shared cache
• CPC Class Codes: G06F 12/1483 using an access-table, e.g. matrix or list
• CPC Class Codes: G06F 15/17331 Distributed shared memory [DSM], e.g. remote direct memory access [RDMA]
• CPC Class Codes: G06F 16/113 Details of archiving lifecycle management in storage systems G06F3/0649; backup systems G06F11/1446
• CPC Class Codes: G06F 16/144 Query formulation
• CPC Class Codes: G06F 16/162 Delete operations erasing in storage systems G06F3/0652
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/221 Column-oriented storage; Management thereof
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2237 Vectors, bitmaps or matrices
• CPC Class Codes: G06F 16/2246 Trees, e.g. B+trees
• CPC Class Codes: G06F 16/2255 Hash tables
• CPC Class Codes: G06F 16/2264 Multidimensional index structures
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/2282 Tablespace storage structures; Management thereof
• CPC Class Codes: G06F 16/2291 : User-Defined Types; Storage management thereof
• CPC Class Codes: G06F 16/23 : Updating
• CPC Class Codes: G06F 16/2308 : Concurrency control transaction processing G06F9/466
• CPC Class Codes: G06F 16/2358 : Change logging, detection, and notification replication G06F16/27
• CPC Class Codes: G06F 16/2365 : Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/2372 : Updates performed during offline database operations
• CPC Class Codes: G06F 16/2379 : Updates performed during online database operations; commit processing
• CPC Class Codes: G06F 16/242 : Query formulation
• CPC Class Codes: G06F 16/2428 : Query predicate definition using graphical user interfaces, including menus and forms G06F16/2423 takes precedence
• CPC Class Codes: G06F 16/245 : Query processing
• CPC Class Codes: G06F 16/2453 : Query optimisation
• CPC Class Codes: G06F 16/24534 : Query rewriting; Transformation
• CPC Class Codes: G06F 16/24535 : of sub-queries or views
• CPC Class Codes: G06F 16/24537 : of operators
• CPC Class Codes: G06F 16/24539 : using cached or materialised query results
• CPC Class Codes: G06F 16/2455 : Query execution
• CPC Class Codes: G06F 16/24553 : of query operations
• CPC Class Codes: G06F 16/2456 : Join operations
• CPC Class Codes: G06F 16/24561 : Intermediate data storage techniques for performance improvement
• CPC Class Codes: G06F 16/2457 : with adaptation to user needs
• CPC Class Codes: G06F 16/24575 : using context
• CPC Class Codes: G06F 16/248 : Presentation of query results
• CPC Class Codes: G06F 16/254 : Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/27 : Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/278 : Data partitioning, e.g. horizontal or vertical partitioning
• CPC Class Codes: G06F 16/285 : Clustering or classification
• CPC Class Codes: G06F 16/9024 : Graphs; Linked lists G06F16/9027 takes precedence
• CPC Class Codes: G06F 16/907 : Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/951 : Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9535 : Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 : Presentation of query results
• CPC Class Codes: G06F 16/9566 : URL specific, e.g. using aliases, detecting broken or misspelled links
• CPC Class Codes: G06F 16/9574 : of access to content, e.g. by caching
• CPC Class Codes: G06F 17/40 : Data acquisition and logging for input to computer G06F3/00
• CPC Class Codes: G06F 21/00 : Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
• CPC Class Codes: G06F 21/6209 : to a single file or object, e.g. in a secure envelope, encrypted and accessed using a key, or with access control rules appended to the object itself
• CPC Class Codes: G06F 2201/805 : Real-time
• CPC Class Codes: G06F 2201/84 : Using snapshots, i.e. a logical point-in-time copy of the data
• CPC Class Codes: G06F 2212/1052 : Security improvement
• CPC Class Codes: G06F 2212/154 : Networked environment
• CPC Class Codes: G06F 2212/163 : Server or database system
• CPC Class Codes: G06F 2212/60 : Details of cache memory
• CPC Class Codes: G06F 3/0481 : based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0482 : Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0483 : Interaction with page-structured environments, e.g. book metaphor
• CPC Class Codes: G06F 3/04847 : Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/0485 : Scrolling or panning
• CPC Class Codes: G06F 3/04895 : Guidance during keyboard input operation, e.g. prompting
• CPC Class Codes: G06F 3/0605 : by facilitating the interaction with a user or administrator
• CPC Class Codes: G06F 3/0656 : Data buffering arrangements
• CPC Class Codes: G06F 3/067 : Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 40/117 : Tagging; Marking up details of markup languages G06F40/143; Designating a block; Setting of attributes style sheets, e.g. eXtensible Stylesheet Language Transformation [XSLT], G06F40/154
• CPC Class Codes: G06F 40/134 : Hyperlinking
• CPC Class Codes: G06F 40/166 : Editing, e.g. inserting or deleting
• CPC Class Codes: G06F 40/174 : Form filling; Merging
• CPC Class Codes: G06F 40/177 : of tables; using ruled lines
• CPC Class Codes: G06F 40/18 : of spreadsheets form-filling G06F40/174
• CPC Class Codes: G06F 40/183 : Tabulation, i.e. one-dimensional positioning
• CPC Class Codes: G06F 40/216 : using statistical methods
• CPC Class Codes: G06F 40/274 : Converting codes to words; Guess-ahead of partial word inputs
• CPC Class Codes: G06F 8/30 : Creation or generation of source code
• CPC Class Codes: G06F 8/41 : Compilation
• CPC Class Codes: G06F 8/427 : Parsing
• CPC Class Codes: G06F 8/60 : Software deployment
• CPC Class Codes: G06Q 40/04 : Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
• CPC Class Codes: H04L 12/18 : for broadcast or conference , e.g. multicast
• CPC Class Codes: H04L 51/046 : Interoperability with other network applications or services
• CPC Class Codes: H04L 51/212 : using filtering or selective blocking
• CPC Class Codes: H04L 61/5069 : for group communication, multicast communication or broadcast communication
• CPC Class Codes: H04L 63/101 : Access control lists [ACL]
• CPC Class Codes: H04L 63/102 : Entity profiles
• CPC Class Codes: H04L 67/01 : Protocols
• CPC Class Codes: H04L 67/1001 : for accessing one among a plurality of replicated servers
• CPC Class Codes: H04L 67/141 : Setup of application sessions admission control or resource allocation in data switching networks H04L47/70
• CPC Class Codes: H04L 67/34 : involving the movement of software or configuration parameters network booting or remote initial program loading [RIPL] G06F9/4416
• CPC Class Codes: H04L 67/56 : Provisioning of proxy services store-and-forward switching systems in data switching networks H04L12/54
• CPC Class Codes: H04L 67/566 : Grouping or aggregating service requests, e.g. for unified processing
• CPC Class Codes: H04L 67/568 : Storing data temporarily at an intermediate stage, e.g. caching
• CPC Class Codes: H04L 67/5681 : Pre-fetching or pre-delivering data based on network characteristics
• CPC Class Codes: H04L 69/16 : Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]

Abstract

Described are methods, systems and computer readable media for computer data distribution architecture.

Claims

1. A memory-efficient and processor-efficient computer system for reliable implementation of a table data cache proxy, the system comprising:

a plurality of data server computers each being programmed with a table data service accessible via an electronic messaging protocol;

a table data cache proxy enabled (TDCP-enabled) server computer coupled to the one or more data server computers, the TDCP-enabled server computer having a plurality of TDCP clients, the TDCP-enabled server computer caching data from the plurality of data server computers and aggregating subscriptions of the TDCP clients to the plurality of data server computers;

the TDCP-enabled server computer comprising;

one or more hardware processors;

a cache memory device having at least a portion being a shared memory portion; and

a computer readable data storage device coupled to the one or more hardware processors, the computer readable data storage device having stored thereon software instructions that, when executed by the one or more hardware processors, cause the one or more hardware processors to perform operations including;

receiving, from a first TDCP client of the plurality of TDCP clients, a first electronic message requesting table data;

determining whether a shared memory cache stored in the shared memory portion of the cache memory device contains a cached copy of the requested table data;

when the shared memory cache contains a cached copy of the requested table data, transmitting, to the first TDCP client, one or more second electronic messages comprising a reference indicating a location where the cached copy is stored in the shared memory portion of the cache memory device in response to the first electronic message, the data received from at least one of the plurality of data server computers and stored in the cache memory device being authoritative due to a data model of the plurality of data server computers;

when the shared memory cache does not contain a cached copy of the requested table data;

selecting one or more data server computers from the plurality of data server computers as one or more appropriate data server computers to provide the requested table data,

transmitting one or more third electronic messages to the one or more appropriate data server computers requesting the requested table data,

receiving one or more fourth electronic messages from the one or more appropriate data server computers in response to the third electronic messages,

filtering the received fourth electronic messages and storing a result of the filtering in the shared memory portion of the cache memory device.

2. A memory-efficient and processor-efficient computer system for reliable implementation of a table data cache proxy, the system comprising:

a table data cache proxy enabled (TDCP-enabled) server computer coupled to a plurality of data server computers, the plurality of data server computers each being programmed with a table data service accessible via an electronic messaging protocol, the TDCP-enabled server computer having a plurality of TDCP clients, the TDCP-enabled server computer caching data from the plurality of data server computers and aggregating subscriptions of the TDCP clients to the plurality of data server computers;

the TDCP-enabled server computer comprising;

one or more hardware processors;

a cache memory device having at least a portion being a shared memory portion; and

a computer readable data storage device coupled to the one or more hardware processors, the computer readable data storage device having stored thereon software instructions that, when executed by the one or more hardware processors, cause the one or more hardware processors to perform operations including;

receiving, from a first TDCP client of the plurality of TDCP clients, a first electronic message requesting table data;

determining whether a shared memory cache stored in the shared memory portion of the cache memory device contains a cached copy of the requested table data;

when the shared memory cache contains a cached copy of the requested table data, transmitting, to the first TDCP client, one or more second electronic messages comprising a reference indicating a location where the cached copy is stored in the shared memory portion of the cache memory device in response to the first electronic message, the data received from at least one of the plurality of data server computers and stored in the cache memory device being authoritative due to a data model of the plurality of data server computers;

when the shared memory cache does not contain a cached copy of the requested table data, requesting the requested data from a server of the plurality of data servers.

3. The system of claim 2, wherein the operations further include:

when the shared memory cache does not contain a cached copy of the requested table data, transmitting a request to subscribe for updates from the one or more appropriate data server computers;

receiving a subscription update from one or more of the plurality of data server computers; and

electronically distributing the received subscription update to one or more of the plurality of TDCP clients.

4. The system of claim 2,

wherein the first electronic message requests data from a data block and comprises a minimum response length;

and

wherein, when additional data above the minimum response length is available within the block, the additional data is included in response to the first electronic message to prevent redundant subsequent requests.

5. The system of claim 2, wherein the first electronic message includes a table location discovery request requesting a list of table locations for a given table key, the table key uniquely identifying a table.

6. The system of claim 2, wherein the first electronic message is a table location metadata retrieval request requesting table-location-level metadata including the size of a table.

7. The system of claim 2, wherein the first electronic message is a column location metadata retrieval request requesting column-location-level metadata.

8. The system of claim 2, wherein column data is laid out in an optimized data layout to enable high performance data access, the column-location-level metadata including information indicating the optimized data layout.

9. The system of claim 2, wherein the first electronic message is a column file size retrieval request requesting the size of a column file.

10. The system of claim 2, wherein the first electronic message is a column file data retrieval request requesting at least a portion of a block of binary data from a column file.

11. The system of claim 2,

wherein the plurality of data server computers comprises at least two data server computers providing the same data;

and

wherein the TDCP-enabled server is configured to reconnect to any of the data server computers, and upon reconnection any in-progress requests are re-sent and any subscriptions are renewed.

12. The system of claim 2, wherein the operations further include:

when the first electronic message requests a block of binary data, determining whether a future request for another block of binary data is likely based on a number of previously received requests; and

when the future request is determined to be likely, prefetching the another block of binary data including transmitting one or more sixth electronic messages to the one or more appropriate data servers requesting the another block of binary data.

13. The system of claim 2, the operations further including:

when the shared memory cache does not contain a cached copy of the requested table data;

selecting one or more data server computers from the plurality of data server computers as one or more appropriate data server computers to provide the requested table data,

the requesting including transmitting one or more third electronic messages to the one or more appropriate data server computers requesting the requested table data,

receiving one or more fourth electronic messages from the one or more appropriate data server computers in response to the third electronic messages,

filtering the received fourth electronic messages and storing a result of the filtering in the shared memory portion of the cache memory device, and

transmitting, to the first TDCP client computer, one or more fifth electronic messages comprising a reference indicating a location where the result of the filtering is stored in the shared memory portion of the cache memory device in response to the first electronic message.

14. A system comprising:

one or more hardware processors of a table data cache proxy enabled (TDCP-enabled) server, the TDCP-enabled server being coupled to one or more data servers;

a computer readable data storage device coupled to the one or more hardware processors, the computer readable data storage device having stored thereon software instructions that, when executed by the one or more hardware processors, cause the one or more hardware processors to perform operations including;

receiving a first electronic message requesting table data;

determining whether a cache of the TDCP-enabled server contains a cached copy of the requested table data;

when the cache of the TDCP-enabled server contains a cached copy of the requested table data, transmitting one or more second electronic messages providing the cached copy of the requested table data from the cache in response to the first electronic message, the data received from at least one of the one or more data servers and stored in the cache being authoritative due to a data model of the one or more data servers

when the cache of the TDCP-enabled server does not contain a cached copy of the requested data, requesting the requested data from a server of the plurality of data servers.

15. The system of claim 14,

wherein the TDCP-enabled server further comprises a shared memory in which at least a portion of the cache is stored,

wherein the one or more second electronic messages comprise an indication where the cached copy is stored in the shared memory of the TDCP-enabled server, and

wherein the one or more fifth electronic messages comprise indications where the requested table data is stored in the shared memory of the TDCP-enabled server.

16. The system of claim 14, wherein the first electronic message is received via a network from a remote client, the one or more second electronic messages are transmitted to the remote client via the network, and the one or more fifth electronic messages are transmitted to the remote client via the network.

17. The system of claim 14, wherein the first electronic message is received via an inter process communication (IPC) mechanism from a client, the one or more second electronic messages are transmitted to the client via the IPC mechanism, and the one or more fifth electronic messages are transmitted to the client via the IPC mechanism.

18. The system of claim 14,

wherein the first electronic message requests data from a data block and comprises a minimum response length;

and

wherein, when additional data above the minimum response length is available within the block, the additional data is included in response to the first electronic message to prevent redundant subsequent requests.

19. The system of claim 14, wherein the first electronic message is one of:

a table location discovery request requesting a list of table locations for a given table key, the table key uniquely identifying a table;

a table location metadata retrieval request requesting table-location-level metadata including the size of a table;

a column location metadata retrieval request requesting column-location-level metadata, wherein column data is laid out in an optimized data layout to enable high performance data access, the column-location-level metadata including information indicating the optimized data layout;

a column file size retrieval request requesting the size of a column file; and

a column file data retrieval request requesting at least a portion of a block of binary data from a column file.

20. The system of claim 14,

wherein the one or more data servers is a plurality of data servers comprising at least two data sources providing the same data;

and

wherein the TDCP-enabled server is configured to reconnect to any of the data servers, and upon reconnection any in-progress requests are re-sent and any subscriptions are renewed.

21. The system of claim 14, wherein the operations further include:

when the first electronic message requests a block of binary data, determining whether a future request for another block of binary data is likely based on a number of received requests; and

when the future request is determined to be likely, prefetching the another block of binary data including transmitting one or more sixth electronic messages to the one or more appropriate data servers requesting the another block of binary data.

22. The system of claim 14, the operations further including:

when, based on the determining, the cache of the TDCP-enabled server does not contain a cached copy of the requested data;

determining one or more appropriate data servers of the one or more data servers to request the requested table data from,

the requesting including transmitting one or more third electronic messages to the one or more appropriate data servers requesting the requested table data,

receiving one or more fourth electronic messages from the one or more appropriate servers in response to the one or more third electronic messages, and

filtering the received one or more fourth electronic messages and, based on a result of the filtering, transmitting one or more fifth electronic messages providing the requested table data in response to the first electronic message.

23. A method comprising:

receiving, at a table data cache proxy enabled (TDCP-enabled) server, a first electronic message requesting table data, the TDCP-enabled server caching data;

determining that a cache of the TDCP-enabled server contains a cached copy of the requested table data;

after the determining that the cache of the TDCP-enabled server contains a cached copy of the requested table data, transmitting one or more second electronic messages providing the cached copy of the requested table data from the cache in response to the first electronic message, the data stored in the cache being authoritative due to a data model;

receiving, at the TDCP-enabled server, another electronic message requesting second table data;

determining that the cache of the TDCP-enabled server does not contain a cached copy of the requested second table data; and

after the determining that the cache of the TDCP-enabled server does not contain a cached copy of the requested second table data, requesting the requested second table data.

24. The method of claim 23,

wherein the one or more data servers comprises a plurality of data servers;

and

wherein, when the first electronic message requests one or more data locations, the filtering comprises combining at least a portion of each of two or more of the one or more fourth electronic messages to create a composite of data from two or more of the plurality of data servers as the result of the filtering.

25. The method of claim 23, wherein the first electronic message is one of:

a table location discovery request requesting a list of table locations for a given table key, the table key uniquely identifying a table;

a table location metadata retrieval request requesting table-location-level metadata including the size of a table;

a column location metadata retrieval request requesting column-location-level metadata, wherein column data is laid out in an optimized data layout to enable high performance data access, the column-location-level metadata including information indicating the optimized data layout;

a column file size retrieval request requesting the size of a column file; and

a column file data retrieval request requesting at least a portion of a block of binary data from a column file.

26. The method of claim 23,

wherein the one or more data servers comprises a plurality of data servers comprising a first and second data server providing the same data;

and

wherein the TDCP-enabled server is configured to reconnect to any of the data servers, and upon reconnection any in-progress requests are re-sent and any subscriptions are renewed.

27. The method of claim 23, further comprising:

determining whether a future request for another block of binary data is likely based on a number of received requests; and

when the future request is determined to be likely, prefetching the another block of binary data including transmitting one or more sixth electronic messages to the one or more appropriate data servers requesting the another block of binary data.

28. The method of claim 23, further comprising:

when the cache of the TDCP-enabled server does not contain a cached copy of the requested data, transmitting a request to subscribe for updates from the one or more appropriate data servers;

receiving a subscription update from one or more of the one or more data servers; and

electronically distributing the received subscription update to one or more subscribers of the TDCP-enabled server.

29. The method of claim 23, further comprising:

after the determining that the cache of the TDCP-enabled server does not contain a cached copy of the requested second table data;

determining one or more appropriate data servers of the one or more data servers to request the requested second table data from,

the requesting including transmitting one or more third electronic messages to the one or more appropriate data servers requesting the requested second table data,

receiving one or more fourth electronic messages from the one or more appropriate servers in response to the one or more third electronic messages, and

filtering the received one or more fourth electronic messages and, based on a result of the filtering, transmitting one or more fifth electronic messages providing the requested second table data in response to the another electronic message.

30. A nontransitory computer readable medium having stored thereon software instructions that, when executed by one or more processors, cause the one or more processors to perform operations including:

receiving, at a table data cache proxy enabled (TDCP-enabled) server, a first electronic message requesting table data, the TDCP-enabled server being coupled to one or more data servers having a table data service accessible via an electronic messaging protocol, the TDCP-enabled server caching data from the one or more data servers;

determining whether a cache of the TDCP-enabled server contains a cached copy of the requested table data;

when the cache of the TDCP-enabled server contains a cached copy of the requested table data, transmitting one or more second electronic messages providing the cached copy of the requested table data from the cache in response to the first electronic message, the data received from at least one of the one or more data servers and stored in the cache being authoritative due to a data model of the one or more data servers; and

when the cache of the TDCP-enabled server does not contain a cached copy of the requested data, requesting the requested data from a server of the one or more of data servers.

31. The nontransitory computer readable medium of claim 30,

wherein the one or more data servers comprises a plurality of data servers;

and

wherein, when the first electronic message requests one or more data locations, the filtering comprises combining at least a portion of each of two or more of the one or more fourth electronic messages to create a composite of data from two or more of the plurality of data servers as the result of the filtering.

32. The nontransitory computer readable medium of claim 30, wherein the first electronic message is one of:

a table location discovery request requesting a list of table locations for a given table key, the table key uniquely identifying a table;

a table location metadata retrieval request requesting table-location-level metadata including the size of a table;

a column location metadata retrieval request requesting column-location-level metadata, wherein column data is laid out in an optimized data layout to enable high performance data access, the column-location-level metadata including information indicating the optimized data layout;

a column file size retrieval request requesting the size of a column file; and

a column file data retrieval request requesting at least a portion of a block of binary data from a column file.

33. The nontransitory computer readable medium of claim 30,

wherein the one or more data servers comprises a plurality of data servers comprising a first and second data server providing the same data;

and

wherein the TDCP-enabled server is configured to reconnect to any of the data servers, and upon reconnection any in-progress requests are re-sent and any subscriptions are renewed.

34. The nontransitory computer readable medium of claim 30, the operations further comprising:

determining whether a future request for another block of binary data is likely based on a number of received requests; and

when the future request is determined to be likely, prefetching the another block of binary data including transmitting one or more sixth electronic messages to the one or more appropriate data servers requesting the another block of binary data.

35. The nontransitory computer readable medium of claim 30, the operations further comprising:

when, based on the determining, the cache of the TDCP-enabled server does not contain a cached copy of the requested data;

determining one or more appropriate data servers of the one or more data servers to request the requested table data from,

transmitting one or more third electronic messages to the one or more appropriate data servers requesting the requested table data,

receiving one or more fourth electronic messages from the one or more appropriate servers in response to the one or more third electronic messages, and

filtering the received one or more fourth electronic messages and, based on a result of the filtering, transmitting one or more fifth electronic messages providing the requested table data in response to the first electronic message.