Implementation of a web scale data fabric

• US 9,208,240 B1
• Filed: 01/21/2015
• Issued: 12/08/2015
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: State Farm Mutual Automobile Insurance Company (State Farm)
• Original Assignee: State Farm Mutual Automobile Insurance Company (State Farm)
• Inventors: Anton, Alex, Sanidas, Tim G., Perschall, Jeff, Bernico, Michael, Cook, Michael K., Calvo, Lynn, Kanneganti, V. Rao
• Primary Examiner(s): Maung, Zarni
• Application Number:
  US14/601,899
• Time in Patent Office: 321 Days
• Field of Search: 709/203, 709223-229, 709/238, 709/250
• US Class Current: 1/1
• CPC Class Codes: G06F 16/113 Details of archiving lifecycle management in storage systems G06F3/0649; backup systems G06F11/1446
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/256 in federated or virtual databases
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
• CPC Class Codes: G06Q 40/08 Insurance
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/51 Discovery or management thereof, e.g. service location protocol [SLP] or web services
• CPC Class Codes: H04L 69/329 in the application layer [OSI layer 7]
• CPC Class Codes: H04L 9/40 Network security protocols

Abstract

Methods and systems for processing machine accelerated and augmented customer data using a Web-Scale Data Fabric (WSDF). According to embodiments, the data may be received as data transfer objects from a set of business operations client applications. The data transfer objects may be analyzed using complex event processing (CEP) and, based on the analyzing, rules specific to the business operations client application may be applied. The methods and systems may semantically classify text specific to the business operations client application. A federated database (FD) may archive the receive data transfer objects as well as analysis data specific to the business operations client application.

Claims

1. A system for processing business operations transactions and associated augmented customer data, the system comprising:

a plurality of computer servers interconnected with a software defined network (SDN) via a plurality of network switches, controllers, and network interfaces, and facilitated by an operating system (OS) comprising a network operating system (NOS), and a resource negotiator (RN), the plurality of computer servers configured for economical large scale computation and data storage with resilience despite underpinning commodity hardware failure and grow-shrink capacity changes of nodes and associated interconnectivity, wherein the plurality of computer servers are configured to;

implement commodity hardware for economy measured by ownership cost, and

perform computation and store data within the computer grid;

direct attached storage (DAS) comprising just a bunch of disks (JBOD) configured for storage economy measured in total cost of ownership;

an information retrieval (IR) component that facilitates resilient low latency indexing and query-using search on indexed federated database content;

random access memory (RAM) coupled to the DAS to provide storage capacity for the plurality of computer servers;

a central processing unit (CPU);

a co-processor coupled to the CPU to provide computation capacity for the plurality of computer servers;

wherein;

the SDN is configured to connect to an external computer network (ECN) for external client input and output,

the NOS and the RN are configured to interface with the SDN to perform a physical-to-virtual network connectivity mapping between the plurality of computer servers and the ECN for negotiated bandwidth and latency conducive to at least one of computation, data receipt, and storage,

the RN is configured to interface with the plurality of computer servers to perform physical-to-virtual parallel computation with negotiated computational capacity on data that is stored on the DAS or cached in the RAM,

the NOS is configured to facilitate physical-to-virtual network connectivity with managed bandwidth and latency,

the IR component comprises an indexer component configured to ingest data and a search server component configured to interface with at least one client application and process search queries made via the at least one client application, and

the RN is configured to implement a resource-management platform.

2. The system of claim 1, wherein the operating system (OS) further comprises a distributed file system (DFS) and a grid node operating system (GNOS), and wherein the DFS and the GNOS are configured to interface with the plurality of computer servers to maintain resilient storage of data received from the external computer network (ECN) or computationally derived from the received data.

3. The system of claim 2, wherein the grid node operating systems (GNOS) is further configured to interface with the plurality of computer servers to perform physical-to-virtual parallel computation with negotiated computational capacity on data that is stored on the direct access storage (DAS) or cached in the RAM.

4. The system of claim 2, wherein data stored on the direct attached storage (DAS) and cached on the random access memory (RAM) is available for computation through the distributed file system (DFS) either local to one of the plurality of computer servers or across the plurality of computer servers via the software defined network (SDN) interconnect configured for this purpose, and wherein the resource negotiator (RN) is configured to manage computation and corresponding SDN interconnect demands.

5. The system of claim 1, further comprising a federated database (FD) that comprises a distributed in-memory cache (DIMC) and a plurality of log-structured merge tree (LSM-Tree) databases configured for resilient high-throughput data accumulation, wherein:

the FD is configured to interface with client applications and machine sources via the ECN to provide consistent data persistence for retrieved data, and interface with the SDN to configure interconnectivity among the plurality of computer servers, the client applications, and the machine sources,

the DIMC is configured to interface with the plurality of computer servers to provide transient and temporary data persistence, and interface with the LSM-Tree to provide data resiliency, and

the LSM-Tree is configured to provide permanent data persistence, implement a non-relational database, and interface with the DFS to provide storage optimization for persisted data.

6. The system of claim 5, further comprising a stream processor that comprises a plurality of message brokers (MB) and a plurality of complex event processors (CEP) configured for resilient high-throughput-low-latency data processing, wherein:

the plurality of MBs implement an advanced message queuing protocol (AMQP), provide reliable delivery of messages independent of data structure, interface with client applications and machine sources via the ECN for receipt of data and with the LSM-Tree for persistence of received data, and

the plurality of CEPs further comprise the DIMC, a plurality of semantic classifiers (SC) configured for semantic content management, and a plurality of in-memory rules engines.

7. The system of claim 5, further comprising:

a data-local processor (DLP) that enables resilient high-throughput-low-latency data parallel processing localized to each of the plurality of computer servers, wherein the DLP is configured to facilitate a High Performance Computing (HPC) with Message Passing Interface (MPI)-based computation, a Many-Task Computing (MTC) with Map Reduce (MR)-based computation, a math algorithm processing functionality for predictive analytics and machine learning, a natural language processing functionality for business meaningful categorization of text data, and wherein the SDN is configured to manage network traffic associated with the HPC and MTC.

8. The system of claim 1, further comprising:

an analytics system configured to provide a transient data aggregate analysis and a federated database (FD) data content aggregate analysis.

9. The system of claim 1, wherein:

the direct attached storage (DAS) is configured for storage economy measured in total cost of ownership per terabyte;

the random access memory (RAM) is configured for computational speed and storage economy measured in total cost of ownership per gigabyte;

the central processing unit (CPU) comprises a mid-range multi-core CPU for computational economy measured in total cost of ownership per flop; and

the co-processor is configured for computation speed and economy measured in total cost of ownership per flop.

10. The system of claim 1, wherein the co-processor provides computation capacity via at least one of high performance computing (HPC) and many-task computing (MTC).

11. The system of claim 10, wherein the high performance computing (HPC) provides large computing power measured in teraflops over short periods of time measured in seconds to days.

12. The system of claim 10, wherein the many-task computing (MTC) provides computing power distributed over a large number of tasks measured in hundreds over short periods of time measured in seconds to days.

13. A method of processing business operations transactions and associated augmented customer data using a plurality of computer servers interconnected with a software defined network (SDN) via a plurality of network switches, controllers, and network interfaces, and facilitated by an operating system (OS) comprising a network operating system (NOS), and a resource negotiator (RN), the plurality of computer servers configured for economical large scale computation and data storage with resilience despite underpinning commodity hardware failure and grow-shrink capacity changes of nodes and associated interconnectivity, the method comprising:

implementing commodity hardware for economy measured by ownership cost;

performing computation and store data within the computer grid;

facilitating, using an information retrieval (IR) component, resilient low latency indexing and query-using search on indexed federated database content;

providing storage capacity for the plurality of computer servers using random access memory (RAM) coupled to direct attached storage (DAS);

providing computation capacity for the plurality of computer servers using a co-processor coupled to a central processing unit (CPU);

connecting, using the SDN, to an external computer network (ECN) for external client input and output;

interfacing with the SDN by the NOS and the RN to perform a physical-to-virtual network connectivity mapping between the plurality of computer servers and the ECN for negotiated bandwidth and latency conducive to at least one of computation, data receipt, and storage;

interfacing with the plurality of computer servers by the RN to perform physical-to-virtual parallel computation with negotiated computational capacity on data that is stored on the DAS or cached in the RAM;

facilitating, by the NOS, physical-to-virtual network connectivity with managed bandwidth and latency;

ingesting data by an indexer component of the IR component;

interfacing by a search server component of the IR to interface with at least one client application and process search queries made via the at least one client application; and

implementing a resource-management platform by the RN.

14. The method of claim 13, wherein the operating system (OS) further comprises a distributed file system (DFS) and a grid node operating system (GNOS), and wherein the method further comprises:

interfacing, by the DFS and GNOS, with the plurality of computer servers to maintain resilient storage of data received from the external computer network (ECN) or computationally derived from the received data.

15. The method of claim 14, further comprising:

interfacing, by the grid node operating systems (GNOS), with the plurality of computer servers to perform physical-to-virtual parallel computation with negotiated computational capacity on data that is stored on the direct access storage (DAS) or cached in the RAM.

16. The method of claim 13, further comprising a federated database (FD) that comprises a distributed in-memory cache (DIMC) and a plurality of log-structured merge tree (LSM-Tree) databases configured for resilient high-throughput data accumulation, wherein:

interfacing, by a federated database (FD), (i) with client applications and machine sources via the ECN to provide consistent data persistence for retrieved data, and (ii) with the SDN to configure interconnectivity among the plurality of computer servers, the client applications, and the machine sources;

interfacing, by a distributed in-memory cache (DIMC), (i) with the plurality of computer servers to provide transient and temporary data persistence, and (ii) with a plurality of log-structured merge tree (LSM-Tree) to provide data resiliency, wherein the LSM tree is configured to provide permanent data persistence, implement a non-relational database, and interface with the DFS to provide storage optimization for persisted data.

17. The method of claim 13, further comprising:

providing a transient data aggregate analysis and a federated database (FD) data content aggregate analysis.

18. The method of claim 13, wherein:

the direct attached storage (DAS) is configured for storage economy measured in total cost of ownership per terabyte;

the random access memory (RAM) is configured for computational speed and storage economy measured in total cost of ownership per gigabyte;

the central processing unit (CPU) comprises a mid-range multi-core CPU for computational economy measured in total cost of ownership per flop; and

the co-processor is configured for computation speed and economy measured in total cost of ownership per flop.

19. The method of claim 13, wherein the co-processor provides computation capacity via at least one of high performance computing (HPC) and many-task computing (MTC).

20. The method of claim 19, wherein the high performance computing (HPC) provides large computing power measured in teraflops over short periods of time measured in seconds to days.