Reassembling streaming data across multiple packetized communication channels

• US 8,335,238 B2
• Filed: 12/23/2008
• Issued: 12/18/2012
• Est. Priority Date: 12/23/2008
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Arimilli, Ravi K., Chaudhary, Piyush
• Primary Examiner(s): Rinehart, Mark
• Assistant Examiner(s): Crompton, Christopher R
• Application Number:
  US12/342,193
• Publication Number:
  US 20100158048A1
• Time in Patent Office: 1,456 Days
• Field of Search: 370/476
• US Class Current: 370/476
• CPC Class Codes: G06F 11/106 Correcting systematically all correctable errors, i.e. scrubbing
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04L 69/14 Multichannel or multilink protocols
• CPC Class Codes: H04N 21/234318 by decomposing into objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/4347 Demultiplexing of several video streams
• CPC Class Codes: H04N 21/44016 involving splicing one content stream with another content stream, e.g. for substituting a video clip
• CPC Class Codes: H04N 21/631 Multimode Transmission, e.g. transmitting basic layers and enhancement layers of the content over different transmission paths or transmitting with different error corrections, different keys or with different transmission protocols
• CPC Class Codes: H04N 21/8453 by locking or enabling a set of features, e.g. optional functionalities in an executable program
• CPC Class Codes: H04N 21/8547 involving timestamps for synchronizing content
• CPC Class Codes: H04N 5/765 Interface circuits between an apparatus for recording and another apparatus
• CPC Class Codes: H04N 5/782 on tape

Abstract

Mechanisms are provided for processing streaming data at high sustained data rates. These mechanisms receive a plurality of data elements over a plurality of non-sequential communication channels and write the plurality of data elements directly to the file system of the data processing system in an unassembled manner. The mechanisms further perform a data scrubbing operation to determine if there are any missing data elements that are not present in the plurality of data elements written to the file system and assemble the plurality of data elements into a plurality of data streams associated with the plurality of non-sequential communication channels in response to results of the data scrubbing indicating that there are no missing data elements. In addition, the mechanisms release the assembled plurality of data streams for access via the file system.

Claims

1. A method, in a data processing system comprising at least one processor, for processing streaming data at high sustained data rates, comprising:

receiving, in the data processing system, a plurality of data elements over a plurality of non-sequential communication channels;

writing the plurality of data elements directly to a file system of a storage system associated with the data processing system in an unassembled manner such that data elements received over at least one of the non-sequential communication channels are written in an intermixed manner with data elements received over at least one other non-sequential communication channel, in the file system of the storage system, wherein the data elements are written directly to the file system without validity or completeness checks being performed prior to writing the data elements directly to the file system;

performing, by the data processing system, a data scrubbing operation to determine if there are any missing data elements that are not present in the plurality of data elements written to the file system of the storage system;

assembling, by the data processing system, the plurality of data elements into a plurality of data streams associated with the plurality of non-sequential communication channels in response to results of the data scrubbing indicating that there are no missing data elements; and

releasing, by the data processing system, the assembled plurality of data streams for access via the file system of the storage system.

2. The method of claim 1, wherein performing the data scrubbing operation comprises performing the data scrubbing operation on a metadata directory associated with the plurality of data elements to determine if metadata indicates that metadata for portions of data are missing in the metadata directory.

3. The method of claim 1, further comprising:

in response to the data scrubbing operation indicating that there is a missing data element, transmitting a request to a source of the missing data element requesting that the missing data element be retransmitted.

4. The method of claim 1, wherein the data scrubbing operation is performed in response to a determination to perform the data scrubbing operation based on history information indicative of whether data elements in the plurality of data elements are being received in a substantially sequential manner, wherein the data scrubbing operation is performed in response to a determination that the data scrubbing operation should be performed based on the history information.

5. The method of claim 4, wherein the history information comprises data indicating an identifier of a last received data element.

6. The method of claim 5, wherein determining to perform data scrubbing comprises determining whether the plurality of data elements are being received in a substantially sequential manner based on whether a current data element is within a predetermined range of data elements from a last received data element identified by the history information, and wherein the data scrubbing operation is performed in response to the plurality of data elements not being received in a substantially sequential manner.

7. The method of claim 4, wherein the history information is communicated to one or more switches in a data network over which the plurality of data elements are transmitted to the data processing system, and wherein the one or more switches use the history information to select a path through the data network over which to transmit data to the data processing system.

8. The method of claim 1, wherein the plurality of data elements comprises portions of data elements, and wherein each portion of data elements is received from a processor in a plurality of processors.

9. The method of claim 8, wherein the plurality of processors are part of a distributed computing cluster operating on message passing interface (MPI) jobs.

10. A method, in a data processing system comprising at least one processor, for processing streaming data at high sustained data rates, comprising:

receiving, in the data processing system, a plurality of data elements over a plurality of non-sequential communication channels from a plurality of processors;

writing the plurality of data elements directly to a file system of a storage system associated with the data processing system in an unassembled manner;

performing, by the data processing system, a data scrubbing operation to determine if there are any missing data elements that are not present in the plurality of data elements written to the file system of the storage system;

assembling, by the data processing system, the plurality of data elements into a plurality of data streams associated with the plurality of non-sequential communication channels in response to results of the data scrubbing indicating that there are no missing data elements; and

releasing, by the data processing system, the assembled plurality of data streams for access via the file system of the storage system, wherein the plurality of processors are part of a distributed computing cluster operating on message passing interface (MPI) jobs, and wherein the plurality of data elements are transmitted by the plurality of processors in response to the processors encountering an MPI barrier operation.

11. A computer program product comprising a computer recordable device having a computer readable program recorded thereon, wherein the computer readable program, when executed on a computing device comprising at least one processor, causes the computing device to:

receive, by the computing device, a plurality of data elements over a plurality of non-sequential communication channels;

write, by the computing device, the plurality of data elements directly to a file system of the data processing system in an unassembled manner such that data elements received over at least one of the non-sequential communication channels are written in an intermixed manner with data elements received over at least one other non-sequential communication channel, in the file system, wherein the data elements are written directly to the file system without validity or completeness checks being performed prior to writing the data elements directly to the file system;

perform, by the computing device, a data scrubbing operation to determine if there are any missing data elements that are not present in the plurality of data elements written to the file system;

assemble, by the computing device, the plurality of data elements into a plurality of data streams associated with the plurality of non-sequential communication channels in response to results of the data scrubbing indicating that there are no missing data elements; and

release, by the computing device, the assembled plurality of data streams for access via the file system.

12. The computer program product of claim 11, wherein the computer readable program causes the computing device to perform the data scrubbing operation by performing the data scrubbing operation on a metadata directory associated with the plurality of data elements to determine if metadata indicates that metadata for portions of data are missing in the metadata directory.

13. The computer program product of claim 11, wherein the computer readable program further causes the computing device to:

transmit, in response to the data scrubbing operation indicating that there is a missing data element, a request to a source of the missing data element requesting that the missing data element be retransmitted.

14. The computer program product of claim 11, wherein the data scrubbing operation is performed in response to a determination to perform the data scrubbing operation based on history information indicative of whether data elements in the plurality of data elements are being received in a substantially sequential manner, wherein the data scrubbing operation is performed only in response to a determination that the data scrubbing operation should be performed based on the history information.

15. The computer program product of claim 14, wherein the history information comprises data indicating an identifier of a last received data element.

16. The computer program product of claim 15, wherein the computer readable program causes the computing device to determine to perform data scrubbing by determining whether the plurality of data elements are being received in a substantially sequential manner based on whether a current data element is within a predetermined range of data elements from a last received data element identified by the history information, and wherein the data scrubbing operation is performed in response to the plurality of data elements not being received in a substantially sequential manner.

17. The computer program product of claim 14, wherein the history information is communicated to one or more switches in a data network over which the plurality of data elements are transmitted to the data processing system, and wherein the one or more switches use the history information to select a path through the data network over which to transmit data to the data processing system.

18. The computer program product of claim 11, wherein the plurality of data elements comprises portions of data elements, and wherein each portion of data elements is received from a processor in a plurality of processors.

19. A computer program product comprising a computer recordable device having a computer readable program recorded thereon, wherein the computer readable program, when executed on a computing device comprising at least one processor, causes the computing device to:

receive, by the computing device, a plurality of data elements over a plurality of non-sequential communication channels from a plurality of processors;

write, by the computing device, the plurality of data elements directly to a file system of the data processing system in an unassembled manner;

perform, by the computing device, a data scrubbing operation to determine if there are any missing data elements that are not present in the plurality of data elements written to the file system;

assemble, by the computing device, the plurality of data elements into a plurality of data streams associated with the plurality of non-sequential communication channels in response to results of the data scrubbing indicating that there are no missing data elements; and

release, by the computing device, the assembled plurality of data streams for access via the file system, wherein the plurality of processors are part of a distributed computing cluster operating on message passing interface (MPI) jobs, and wherein the plurality of data elements are transmitted by the plurality of processors in response to the processors encountering an MPI barrier operation.

20. An apparatus, comprising:

a processor; and

a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to;

receive a plurality of data elements over a plurality of non-sequential communication channels;

write the plurality of data elements directly to a file system of the data processing system in an unassembled manner such that data elements received over at least one of the non-sequential communication channels are written in an intermixed manner with data elements received over at least one other non-sequential communication channel, in the file system, wherein the data elements are written directly to the file system without validity or completeness checks being performed prior to writing the data elements directly to the file system;

perform a data scrubbing operation to determine if there are any missing data elements that are not present in the plurality of data elements written to the file system;

assemble the plurality of data elements into a plurality of data streams associated with the plurality of non-sequential communication channels in response to results of the data scrubbing indicating that there are no missing data elements; and

release the assembled plurality of data streams for access via the file system.