Optimized logging of data elements to a data storage device
First Claim
1. In a computer system, a method for logging data produced by a data source to a hardware device by a data logger operating in a separate thread than the data source, the method comprising the steps of:
- retrieving from a set of empty buffers, by the data source, a first buffer pointing to a first page in memory;
storing in the first page in memory, by the data source, data records to be logged to the hardware device;
placing the first buffer in a queue of full buffers;
retrieving from the set of empty buffers, by the data source, a second buffer pointing to a second page in memory;
storing in the second page in memory, by the data source, data records to be logged to the hardware device;
placing the second buffer in the queue of full buffers;
retrieving, by the data logger, the first and second buffers from the queue of full buffers;
packaging the first and second buffers into a data block; and
logging on the hardware device the data stored in the memory pages corresponding to the buffers contained in the data block.
2 Assignments
0 Petitions
Accused Products
Abstract
An invention for optimizing the logging of data elements to a hardware device is described. Using this invention, a large stream of data can be written to a hardware device at a rate that approaches the limits of the physical characteristics of the hardware device. To achieve this efficiency, the performance of a logging operation is divided between a data source and a data logging software processes which operate in different threads or processes. The data source collects pieces of data to be written to the hardware device into a larger buffer retrieved from a pool of empty buffers. When a buffer becomes full, the buffer is placed at the end of a full buffer queue. The data logger, operating asynchronously, retrieves a full buffer from the queue and then writes the data to the hardware. In this fashion, the hardware data logging device is able to perform optimally while creating or expanding the file.
82 Citations
32 Claims
-
1. In a computer system, a method for logging data produced by a data source to a hardware device by a data logger operating in a separate thread than the data source, the method comprising the steps of:
-
retrieving from a set of empty buffers, by the data source, a first buffer pointing to a first page in memory;
storing in the first page in memory, by the data source, data records to be logged to the hardware device;
placing the first buffer in a queue of full buffers;
retrieving from the set of empty buffers, by the data source, a second buffer pointing to a second page in memory;
storing in the second page in memory, by the data source, data records to be logged to the hardware device;
placing the second buffer in the queue of full buffers;
retrieving, by the data logger, the first and second buffers from the queue of full buffers;
packaging the first and second buffers into a data block; and
logging on the hardware device the data stored in the memory pages corresponding to the buffers contained in the data block. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13)
retrieving from the set of empty buffers, by the second data source, a third buffer pointing to a third page in memory;
storing in the third page in memory, by the second data source, data records to be logged to the hardware device; and
placing the third buffer in the queue of full buffers;
retrieving from the set of empty buffers, by the second data source, a fourth buffer pointing to a fourth page in memory;
storing in the fourth page in memory, by the second data source, data records to be logged to the hardware device; and
placing the fourth buffer in the queue of full buffers;
retrieving, by the data logger, the third and fourth buffers from the queue of full buffers;
packaging the third and fourth buffers into a second data block; and
logging on the hardware device the data stored in the memory pages corresponding to the third and fourth buffers contained in the second data block.
-
-
12. The method of claim 1, wherein the computer system further comprises a second hardware controller, a second hardware device, and a second data logger, the method further comprising the steps of:
-
retrieving from the set of empty buffers, by the data source, a third buffer pointing to a third page in memory;
storing in the third page in memory, by the data source, data records to be logged to the second hardware device;
placing the third buffer in the queue of full buffers;
retrieving from the set of empty buffers, by the data source, a fourth buffer pointing to a fourth page in memory;
storing in the fourth page in memory, by the data source, data records to be logged to the second hardware device;
placing the fourth buffer in a queue of full buffers;
retrieving, by the second data logger, the third and fourth buffers from the queue of full buffers;
packaging the third and fourth buffers into a second data block; and
logging on the second hardware device the data stored in the memory pages corresponding to the buffers contained in the second data block.
-
-
13. A computer-readable medium having computer-executable instructions representing the method of claim 1.
-
14. In a computer system, a method for logging a stream of data produced by a data source to a hardware device by a data logger operating in a separate thread than the data source, the method comprising the steps of:
-
retrieving from a set of empty buffers a first buffer pointing to a first block of memory;
filling the first block of memory with data records from the stream of data;
placing the first buffer in a queue of full buffers;
retrieving from the set of empty buffers a second buffer pointing to a second block of memory;
filling the second block of memory with data records from the stream of data;
placing the second buffer in the queue of full buffers;
retrieving the first and second buffers from the queue of full buffers;
packaging the contents of the first and second buffers into a larger data block;
passing the larger data block to a controller for logging the data stored in the first and second blocks of memory on the hardware device; and
logging the data stored in the first and second blocks of memory on the hardware device. - View Dependent Claims (15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25)
-
-
26. In a computer system, a method for logging data produced by a data source to a hard disk drive by a data logger operating in a separate thread than the data source, the method comprising the steps of:
-
retrieving from a set of empty buffers, by the data source, a first buffer pointing to a first page in memory;
storing in the first page in memory, by the data source, data records to be logged to the hard disk drive;
placing the first buffer in a queue of full buffers;
retrieving from the set of empty buffers, by the data source, a second buffer pointing to a second page in memory;
storing in the second page in memory, by the data source, data records to be logged to the hardware device;
placing the second buffer in the queue of full buffers;
retrieving the first and second buffers from the queue of full buffers;
packaging the addresses contained in the first and second buffers in a parameter array;
passing the parameter array to a hard drive controller;
logging on the hard disk drive the data stored in the memory pages corresponding to the addresses contained in the parameter array; and
returning the first and second buffers to the set of empty buffers. - View Dependent Claims (27, 28, 29, 30, 31, 32)
-
Specification