Real-time distributed in memory search architecture

• US 9,223,875 B2
• Filed: 12/02/2014
• Issued: 12/29/2015
• Est. Priority Date: 12/02/2013
• Status: Active Grant

• Current Assignee: Finch Computing LLC (Qbase, LLC)
• Original Assignee: Qbase, LLC
• Inventors: Lightner, Scott, Weckesser, Franz
• Primary Examiner(s): Shanmugasundaram, Kannan
• Application Number:
  US14/557,827
• Publication Number:
  US 20150154297A1
• Time in Patent Office: 392 Days
• Field of Search: 707/770, 707/674, 707/686
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9538 Presentation of query results

Abstract

Disclosed here are distributed computing system connection configurations having multiple connection bandwidth and latency tiers. Also disclosed are connection configurations including a suitable number of network segments, where network segments may be connected to external servers and clusters including search managers, analytics agents, search conductors, dependency managers, supervisors, and partitioners, amongst others. In one or more embodiments, modules may be connected to the network segments using a desired bandwidth and latency tier. Disclosed here are hardware components suitable for running one or more types of modules on one or more suitable nodes. One or more suitable hardware components included in said clusters include CPUs, Memory, and Hard Disk, amongst others.

Claims

1. A system comprising one or more nodes hosting an in-memory database, the system comprising:

a plurality of storage nodes comprising non-transitory machine-readable storage medium storing one or more partitions of a collection, wherein the collection stored by each respective storage node contains one or more records of a database, and wherein the storage medium of each respective storage node comprises main memory;

one or more partitioner nodes partitioning one or more collections of database records and distributing the one or more collections to the one or more storage nodes according to a schema file responsive to receiving the one or more collections;

a search manager node comprising a processor generating one or more search conductor queries using a search query received from a user node, transmitting the one or more search conductor queries to one or more search conductor nodes according to the search query, and forward one or more sets of search results to one or more analytics agent nodes according to the search query responsive to receive the one or more sets of search results;

an analytics agent node comprising a processor executing one or more analytics algorithms responsive to receiving a set of search results from the search manager node;

a search conductor node comprising a processor querying the collection of the database records of a storage node according to a search conductor query in response to receiving the search conductor query from the search manager, and transmitting the set of one or more search results to the search manager node in response to identifying the one or more search results of the set, wherein each respective search result corresponds to a database record containing data satisfying the search conductor query;

a plurality of network segments comprising one or more connections between one or more nodes communicating over each respective network segment, wherein a first network segment comprises the search manager, the search conductor, and the analytics agent;

a dependency manager comprising non-transitory machine-readable storage medium storing one or more configuration package files and a dependency tree, wherein the one or more configuration package files attempts to minimize cost based on actual need, and a processor configured to determine a configuration package file for a failed node and transmit the configuration package file to a supervisor node based on the dependency tree in response to a request for the configuration package file from the supervisor node;

wherein a third network segment further comprises the one or more connections communicating the request for the configuration package file; and

the one or more supervisor nodes comprising the processor continuously monitoring one or more statuses associated with the one or more nodes of the system, and configured to transmit the configuration package file to the failed node of the one or more nodes in response to receiving a status indicating a failed resource of the failed node.

2. The system according to claim 1,

wherein each respective network segment comprises the one or more connections based upon a resource demand associated with a communication type, and

wherein the communications type is selected from the group consisting of search query data, one or more collections of database records, status data, and configuration package files.

3. The system according to claim 2, wherein the resource demand of each respective communication type is determined according to a latency demand and a bandwidth demand, wherein the latency demand is based on a permissible amount of time for the communication, and wherein the bandwidth demand is based on an amount of data for the communication.

4. The system according to claim 2, wherein the first network segment comprises the one or more connections communicating search query data.

5. The system according to claim 4, wherein the first network segment comprises a physical connection using InfiniBand.

6. The system according to claim 4, wherein a second network segment comprises the one or more connections communicating the one or more collections from each respective partitioner to the one or more storage nodes.

7. The system according to claim 6, wherein the second network segment comprises a second physical connection using InfiniBand.

8. The system according to claim 7, wherein the second network segment comprises a second physical connection using Ethernet.

9. The system according to claim 6, wherein a the third network segment comprises the one or more connections communicating each respective status of each respective node to the one or more supervisor nodes and the configuration package file transmitted from the supervisor node to the failed node.

10. The system according to claim 9, wherein the third network segment comprises a third physical connection using Ethernet.