Importation, presentation, and persistent storage of data
First Claim
Patent Images
1. A computer data system comprising:
- one or more processors;
computer readable storage coupled to the one or more processors, the computer readable storage having stored thereon instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including;
starting with a processor a log tailer process for reading data from one or more log files and transmitting the data read from the one or more log files;
the log tailer process reading the data from the one or more log files;
the log tailer process transmitting the data to one or more data import servers, the one or more data import servers being separate from the log tailer process;
at the one or more data import servers, writing the data to one or more columns of an associated table location in cache memory buffers;
confirming, at the one or more data import servers, a current number of rows of data written to the cache memory buffers;
synchronizing, at the one or more data import servers, the column data from the cache memory buffers into a persistent table data store;
updating, at the one or more data import servers, table location metadata;
writing formatted data into the one or more log files by applying schema-specified, generated code to format the data;
parsing at least a portion of the formatted data with a listener generated from a schema;
generating the schema-specified, generated code used to format the data, wherein the table location metadata comprises a version identifier of another schema used to generate the schema-specified, generated code;
andwriting a check point for persisted data from the persistent table data store for use in resuming import after a system fault.
3 Assignments
0 Petitions
Accused Products
Abstract
Described are methods, systems and computer readable media for the importation, presentation, and persistent storage of data.
495 Citations
29 Claims
-
1. A computer data system comprising:
-
one or more processors; computer readable storage coupled to the one or more processors, the computer readable storage having stored thereon instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including; starting with a processor a log tailer process for reading data from one or more log files and transmitting the data read from the one or more log files; the log tailer process reading the data from the one or more log files; the log tailer process transmitting the data to one or more data import servers, the one or more data import servers being separate from the log tailer process; at the one or more data import servers, writing the data to one or more columns of an associated table location in cache memory buffers; confirming, at the one or more data import servers, a current number of rows of data written to the cache memory buffers; synchronizing, at the one or more data import servers, the column data from the cache memory buffers into a persistent table data store; updating, at the one or more data import servers, table location metadata; writing formatted data into the one or more log files by applying schema-specified, generated code to format the data; parsing at least a portion of the formatted data with a listener generated from a schema; generating the schema-specified, generated code used to format the data, wherein the table location metadata comprises a version identifier of another schema used to generate the schema-specified, generated code; and writing a check point for persisted data from the persistent table data store for use in resuming import after a system fault. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9)
-
-
10. A method for importing and presenting data for computer system use, the method comprising:
-
Starting, with a processor, a log tailer process for reading data from one or more log files and transmitting the data read from the one or more log files; the log tailer process reading the data from the one or more log files; the log tailer process transmitting the data to one or more data import servers, the one or more data import servers being separate from the log tailer process; at the one or more data import servers, writing the data to one or more columns of an associated table location in cache memory buffers; confirming, at the one or more data import servers, a current number of rows of data written to the cache memory buffers; writing formatted data into the one or more log files by applying schema-specified, generated code to format the data; parsing at least a portion of the formatted data with a listener generated from a schema; updating, at the one or more data import servers, table location metadata; generating the schema-specified, generated code used to format the data, wherein the table location metadata comprises a version identifier of another schema used to generate the schema-specified, generated code; and writing a check point for persisted data from a persistent table data store for use in resuming import after a system fault. - View Dependent Claims (11, 12, 13, 14, 15, 16, 17, 18)
-
-
19. A nontransitory computer readable medium having stored thereon software instructions that, when executed by one or more processors, cause the one or more processors to perform operations including:
-
starting with a processor a log tailer process for reading data from one or more log files and transmitting the data read from the one or more log files; the log tailer process reading the data from the one or more log files; the log tailer process transmitting the data to one or more data import servers, the one or more data import servers being separate from the log tailer process; at the one or more data import servers, writing the data to one or more columns of an associated table location in cache memory buffers; synchronizing, at the one or more data import servers, the column data from the cache memory buffers into a persistent table data store; writing formatted data into the one or more log files by applying schema-specified, generated code to format the data; parsing at least a portion of the formatted data with a listener generated from a schema; updating, at the one or more data import servers, table location metadata; generating the schema-specified, generated code used to format the data, wherein the table location metadata comprises a version identifier of another schema used to generate the schema-specified, generated code; and writing a check point for persisted data from the persistent table data store for use in resuming import after a system fault. - View Dependent Claims (20, 22, 23, 24, 25, 26)
-
-
21. The nontransitory computer readable medium of 20, wherein the advertising is made without waiting for the check point to be written.
-
27. A computer data system comprising:
-
one or more processors; computer readable storage coupled to the one or more processors, the computer readable storage having stored thereon instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including; starting a log tailer process for reading data from one or more log files and transmitting data read from log files; the log tailer process reading data from the one or more log files; the log tailer process transmitting the data to one or more data import servers; at the one or more data import servers, writing the data to one or more columns of an associated table location in cache memory buffers; confirming, at the one or more data import servers, a current number of rows of data written to the cache memory buffers; synchronizing, at the one or more data import servers, the column data from the cache memory buffers into a persistent table data store; updating, at the one or more data import servers, table location metadata; writing a check point for the persisted data for use in resuming import after a system fault; generating a listener based on a schema; generating formatting code based on the same schema; receiving data; formatting the received data by executing the generated formatting code to format the data for output to the one or more log files; writing the formatted data to the one or more log files; and parsing, with the listener, at least a portion of the formatted data read from the one or more log files.
-
-
28. A method for importing and presenting data for computer system use, the method comprising:
-
starting with a processor a log tailer process for reading data from one or more log files and transmitting data read from log files; the log tailer process reading data from the one or more log files; the log tailer process transmitting the data to one or more data import servers; at the one or more data import servers, writing the data to one or more columns of an associated table location in cache memory buffers; confirming, at the one or more data import servers, a current number of rows of data written to the cache memory buffers; writing a check point for the persisted data for use in resuming import after a system fault; generating a listener based on a schema; generating formatting code based on the same schema; receiving data; formatting the received data by executing the generated formatting code to format the data for output to the one or more log files; writing the formatted data to the one or more log files; and parsing, with the listener, at least a portion of the formatted data read from the one or more log files.
-
-
29. A nontransitory computer readable medium having stored thereon software instructions that, when executed by one or more processors, cause the one or more processors to perform operations including:
-
starting a log tailer process for reading data from one or more log files and transmitting data read from log files; the log tailer process reading data from the one or more log files; the log taller process transmitting the data to one or more data import servers; at the one or more data import servers, writing the data to one or more columns of an associated table location in cache memory buffers; synchronizing, at the one or more data import servers, the column data from the cache memory buffers into a persistent table data store; writing a check point for the persisted data for use in resuming import after a system fault generating a listener based on a schema; generating formatting code based on the same schema; receiving data; formatting the received data by executing the generated formatting code to format the data for output to the one or more log files; writing the formatted data to the one or more log files; and parsing, with the listener, at least a portion of the formatted data read from the one or more log files.
-
Specification