Data fabric service system architecture
First Claim
1. A method comprising:
- receiving, by a data intake and query system, a search query;
defining, by the data intake and query system, a search scheme for applying the search query on a plurality of distributed data storage systems including an internal data storage system of the data intake and query system and an external data storage system communicatively coupled to the data intake and query system over a computer network, wherein the internal data storage system stores data as a plurality of time-indexed events including respective segments of raw machine data;
transferring, by the data intake and query system, a portion of the search scheme to a search service for obtaining partial search results from the external data storage system;
producing, by the data intake and query system, partial search results by applying a portion of the search scheme to the internal data storage system;
transmitting, by the data intake and query system over the computer network, the partial search results to at least one worker node communicatively coupled to the external data storage system over the computer network, the at least one worker node being communicatively coupled to the search service;
receiving, by the data intake and query system from the search service, search results of the search query obtained by application of the search scheme to the plurality of distributed data storage systems including the internal data storage system and the external data storage system;
wherein the search results obtained by the data intake and query system from the search service are based on a combination of the partial search results of the internal data storage system and the partial search results obtained from the external data storage system combined by the at least one worker node; and
causing output of the search results or data indicative of the search results by a display device to a user.
2 Assignments
0 Petitions
Accused Products
Abstract
Disclosed is a technique that can be performed in a distributed computer network. The technique can include a data index and query system that receives search query, defines a search scheme for applying the search query on distributed data storage systems including an internal data storage system of the data index and query system and an external data storage system. The internal data storage system stores data as time-indexed events including respective segments of raw machine data. The data index and query system can transfer a portion of the search scheme to a search service, which can return search results obtained by application of the search scheme to the distributed data storage systems including the internal data storage system and the external data storage system. Lastly, the search results or data indicative of the search results can be output on a display device to the user.
-
Citations
25 Claims
-
1. A method comprising:
-
receiving, by a data intake and query system, a search query; defining, by the data intake and query system, a search scheme for applying the search query on a plurality of distributed data storage systems including an internal data storage system of the data intake and query system and an external data storage system communicatively coupled to the data intake and query system over a computer network, wherein the internal data storage system stores data as a plurality of time-indexed events including respective segments of raw machine data; transferring, by the data intake and query system, a portion of the search scheme to a search service for obtaining partial search results from the external data storage system; producing, by the data intake and query system, partial search results by applying a portion of the search scheme to the internal data storage system; transmitting, by the data intake and query system over the computer network, the partial search results to at least one worker node communicatively coupled to the external data storage system over the computer network, the at least one worker node being communicatively coupled to the search service; receiving, by the data intake and query system from the search service, search results of the search query obtained by application of the search scheme to the plurality of distributed data storage systems including the internal data storage system and the external data storage system; wherein the search results obtained by the data intake and query system from the search service are based on a combination of the partial search results of the internal data storage system and the partial search results obtained from the external data storage system combined by the at least one worker node; and causing output of the search results or data indicative of the search results by a display device to a user. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)
-
-
12. A method comprising:
-
receiving, by a search service, data representing a portion of a search scheme defined by a data intake and query system for a plurality of worker nodes to collect a plurality of partial search results over a computer network from a plurality of distributed data storage systems including a plurality of external data storage systems and an internal data storage system of the data intake and query system, wherein the search scheme comprises a plurality of search phases; defining, by the search service, an executable search process based on the received portion of the search scheme defined by the data intake and query system, the executable search process being defined as a directed acyclic graph to include a first phase causing the plurality of worker nodes to extract and collect the plurality of partial search results from the plurality of external data storage systems, a second phase causing the plurality of worker nodes to collect the plurality of partial search results from the internal data storage system, and a third phase causing the search service to collect the plurality of partial search results of the plurality of external data storage systems and the plurality of partial search results of the internal data storage system from the plurality of worker nodes; executing, by the search service, the executable search process to cause the plurality of worker nodes to extract and collect the plurality of partial search results from the plurality of external data storage systems and collect the plurality of partial search results from the internal data storage system over the computer network; receiving, by the search service, a plurality of aggregate partial search results based on the plurality of partial search results of the plurality of external data storage systems and the plurality of partial search results of the internal data storage system; and sending, by the search service, the plurality of aggregate partial search results to the data intake and query system.
-
-
13. A method comprising:
-
receiving, by a search service, data representing a portion of a search scheme defined by a data intake and query system for a plurality of worker nodes to collect partial search results over a computer network from a plurality of distributed data storage systems including a plurality of external data storage systems and an internal data storage system of the data intake and query system; defining, by the search service, an executable search process based on the received portion of the search scheme defined by the data intake and query system; executing, by the search service, the executable search process to cause the plurality of worker nodes to collect a plurality of partial search results from the plurality of external data storage systems and collect a plurality of partial search results from the internal data storage system over the computer network and produce a plurality of aggregate partial search results by combining the plurality of partial search results of the plurality of external data storage systems with the plurality of partial search results of the internal data storage system as an arrangement of time-ordered events; receiving, by the search service, the plurality of aggregate partial search results; and sending, by the search service, the plurality of aggregate partial search results to the data intake and query system. - View Dependent Claims (14, 15, 16, 17, 18, 19, 20)
-
-
21. A method comprising:
-
receiving, by a worker node over a computer network, search instructions defined by a search service for the worker node to collect a plurality of partial search results from a plurality of distributed data storage systems including a plurality of partial search results of an external data storage system and a plurality of partial search results of an internal data storage system of a data intake and query system; collecting, by the worker node over a computer network, the plurality of partial search results of the external data storage system searched in accordance with the search instructions; collecting, by the worker node over the computer network, the plurality of partial search results of the internal data storage system; producing, by the worker node, a plurality of aggregate partial search results from the plurality of partial search results of the external data storage system and the plurality of partial search results of the internal data storage system by combining the plurality of partial search results of the external data storage system with the plurality of partial search results of the internal data storage system as an arrangement of time-ordered events; and providing, by the worker node over the computer network to the search service, the plurality of aggregate partial search results in response to the search instructions. - View Dependent Claims (22, 23, 24)
-
-
25. A method comprising:
-
receiving, by a first worker node over a computer network, search instructions defined by a search service for the first worker node to collect a plurality of partial search results from a plurality of distributed data storage systems including a first plurality of partial search results of a first external data storage system and a second plurality of partial search results of an internal data storage system of a data intake and query system; collecting, by the first worker node over a computer network, the first plurality of partial search results of the first external data storage system searched in accordance with the search instructions; collecting, by the first worker node over the computer network, the second plurality of partial search results of the internal data storage system; collecting, by the first worker node over the computer network, a third plurality of partial search results extracted from a second external data storage system by a second worker node, the second worker node being communicatively coupled over the computer network to the second external data storage system; sending, by the first worker node over the computer network, at least a portion of the first plurality of partial search results to the second worker node; producing, by the first worker node, a plurality of aggregate partial search results from the first plurality of partial search results of the first external data storage system, the second plurality of partial search results of the internal data storage system, and the third plurality of partial search results of the second external data storage system; and providing, by the first worker node over the computer network to the search service, the plurality of aggregate partial search results in response to the search instructions.
-
Specification