Dynamic monitoring of ability to reassemble streaming data across multiple channels based on history
First Claim
1. A method, in a data processing system, for processing streaming data at high sustained data rates, comprising:
- receiving a plurality of data elements, in the data processing system, over a plurality of non-sequential communication channels;
writing the plurality of data elements directly to a file system of the data processing system in an unassembled manner;
determining whether to perform a data scrubbing operation or not based on history information indicative of whether data elements in the plurality of data elements are being received in a substantially sequential manner;
performing a data scrubbing operation, in response to a determination to perform data scrubbing, to determine if there are any missing data elements that are not present in the plurality of data elements written to the file system; and
assembling the plurality of data elements into a plurality of data streams associated with the plurality of non-sequential communication channels in response to results of the data scrubbing indicating that there are no missing data elements.
0 Assignments
0 Petitions
Accused Products
Abstract
Mechanisms are provided for processing streaming data at high sustained data rates. These mechanisms receive a plurality of data elements over a plurality of non-sequential communication channels and write the plurality of data elements directly to the file system of the data processing system in an unassembled manner. The mechanisms determining whether to perform a data scrubbing operation or not based on history information indicative of whether data elements in the plurality of data elements are being received in a substantially sequential manner. The mechanisms perform a data scrubbing operation, in response to a determination to perform data scrubbing, to identify any missing data elements in the plurality of data elements written to the file system and assemble the plurality of data elements into a plurality of data streams in response to results of the data scrubbing indicating that there are no missing data elements.
51 Citations
13 Claims
-
1. A method, in a data processing system, for processing streaming data at high sustained data rates, comprising:
-
receiving a plurality of data elements, in the data processing system, over a plurality of non-sequential communication channels; writing the plurality of data elements directly to a file system of the data processing system in an unassembled manner; determining whether to perform a data scrubbing operation or not based on history information indicative of whether data elements in the plurality of data elements are being received in a substantially sequential manner; performing a data scrubbing operation, in response to a determination to perform data scrubbing, to determine if there are any missing data elements that are not present in the plurality of data elements written to the file system; and assembling the plurality of data elements into a plurality of data streams associated with the plurality of non-sequential communication channels in response to results of the data scrubbing indicating that there are no missing data elements. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13)
-
Specification