Method and apparatus for retrieving and combining summarized log data in a distributed log data processing system
First Claim
Patent Images
1. A method for processing log data, the method comprising:
- receiving raw log data in a log data analyzer from a plurality of log-producing devices, the raw log data including a plurality of messages;
parsing the raw log data to generate structured query language (SQL) statements representing the raw log data, including;
extracting fields from the raw log data based on a log data message type for each message, the log data message type indicating that a respective message is associated with a pre-defined type of event on a log-producing device that produced the respective message;
generating the SQL statements from the extracted fields, wherein the extracted fields are incorporated into the SQL statements; and
inserting the SQL statements into a database table designated to store the generated SQL statements on a volatile storage device, the database table being different from database tables for storing the raw log data;
summarizing the SQL statements, including;
at pre-specified time intervals, copying the SQL statements stored in the database table stored on the volatile storage device to one or more database tables stored on a non-volatile storage device;
identifying, from the SQL statements stored in the one or more database tables on the non-volatile storage device, a set of one or more SQL statements each including one or more fields of commonality and one or more fields of uniqueness, each field of commonality being a field storing a value that is commonly incorporated in the set of SQL statements, each field of uniqueness being a field storing values that are different among the set of SQL statements; and
creating summarized data, the summarized data including a new statement condensed from the identified set of one or more SQL statements based on the one or more fields of commonality and one or more fields of uniqueness; and
generating a report based on the summarized data, wherein the method is performed by one or more computers.
15 Assignments
0 Petitions
Accused Products
Abstract
A system can receive raw log data from log-producing devices. The system can parse the raw log data to generate parsed log data, including extracting fields from the raw log data, generating structured query language (SQL) statements from the extracted fields, and inserting the SQL statements in a database. The system can summarize the parsed log data, including identifying one or more SQL statements based on fields of commonality and fields of uniqueness, and creating summarized data that include a new statement condensed from the identified SQL statements. The system can then generate a report based on the summarized data.
-
Citations
19 Claims
-
1. A method for processing log data, the method comprising:
-
receiving raw log data in a log data analyzer from a plurality of log-producing devices, the raw log data including a plurality of messages; parsing the raw log data to generate structured query language (SQL) statements representing the raw log data, including; extracting fields from the raw log data based on a log data message type for each message, the log data message type indicating that a respective message is associated with a pre-defined type of event on a log-producing device that produced the respective message; generating the SQL statements from the extracted fields, wherein the extracted fields are incorporated into the SQL statements; and inserting the SQL statements into a database table designated to store the generated SQL statements on a volatile storage device, the database table being different from database tables for storing the raw log data; summarizing the SQL statements, including; at pre-specified time intervals, copying the SQL statements stored in the database table stored on the volatile storage device to one or more database tables stored on a non-volatile storage device; identifying, from the SQL statements stored in the one or more database tables on the non-volatile storage device, a set of one or more SQL statements each including one or more fields of commonality and one or more fields of uniqueness, each field of commonality being a field storing a value that is commonly incorporated in the set of SQL statements, each field of uniqueness being a field storing values that are different among the set of SQL statements; and creating summarized data, the summarized data including a new statement condensed from the identified set of one or more SQL statements based on the one or more fields of commonality and one or more fields of uniqueness; and generating a report based on the summarized data, wherein the method is performed by one or more computers. - View Dependent Claims (2, 3, 4, 5, 6, 7)
-
-
8. A data processing system for processing log data comprising:
-
a management station including one or more processors; and a log data analyzer including one or more processors, the log data analyzer being connected to the management station via a data communications link and configured to perform operations comprising; receiving raw log data from a plurality of log-producing devices, the raw log data including a plurality of messages; parsing the raw log data to generate structured query language (SQL) statements representing the raw log data, including; extracting fields from the raw log data based on a log data message type for each message, the log data message type indicating that a respective message is associated with a pre-defined type of event on a log-producing device that produced the respective message; generating the SQL statements from the extracted fields, wherein the extracted fields are incorporated into the SQL statements; and inserting the SQL statements into a database table designated to store the generated SQL statements, the database table being different from database tables for storing the raw log data; summarizing the SQL statements, including; identifying, from the SQL statements inserted into the database table, a set of one or more SQL statements based on one or more fields of commonality and one or more fields of uniqueness, each field of commonality being a field storing a value that is commonly incorporated in the set of SQL statements, each field of uniqueness being a field storing values that are different among the set of SQL statements; and creating summarized data, the summarized data including a new statement condensed from the set of one or more SQL statement based on the one or more fields of commonality and one or more fields of uniqueness; and generating a report based on the summarized data. - View Dependent Claims (9, 10, 11, 12, 13, 14)
-
-
15. A non-transitory storage device storing computer software, the computer software operable to cause one or more computers to perform operations comprising:
-
receiving raw log data in a log data analyzer from a plurality of log-producing devices, the raw log data including a plurality of messages; parsing the raw log data to generate structured query language (SQL) statements representing the raw log data, including; extracting fields from the raw log data based on a log data message type for each message, the log data message type indicating that a respective message is associated with a pre-defined type of event on a log-producing device that produced the respective message; generating the SQL statements from the extracted fields, wherein the extracted fields are incorporated into the SQL statements; and inserting the SQL statements into a database table designated to store the generated SQL statements the database table being different from database tables for storing the raw log data; summarizing the SQL statements, including; identifying, from the SQL statements inserted into the database table, a set of one or more SQL statements based on one or more fields of commonality and one or more fields of uniqueness, each field of commonality being a field storing a value that is commonly incorporated in the set of SQL statements, each field of uniqueness being a field storing values that are different among the set of SQL statements; and creating summarized data, the summarized data including a new statement condensed from the identified set of one or more SQL statements based on the one or more fields of commonality and one or more fields of uniqueness; and generating a report based on the summarized data. - View Dependent Claims (16, 17, 18, 19)
-
Specification