In-band synchronization of data streams with audio/video streams

• US 6,359,656 B1
• Filed: 12/20/1996
• Issued: 03/19/2002
• Est. Priority Date: 12/20/1996
• Status: Expired due to Term

• Current Assignee: Mineral Lassen LLC (Intellectual Ventures LLC)
• Original Assignee: Intel Corporation
• Inventors: Huckins, Jeffrey L.
• Primary Examiner(s): Eisenzopf, Reinhard J.
• Assistant Examiner(s): NATNAEL, PAULOS M
• Application Number:
  US08/772,161
• Time in Patent Office: 1,915 Days
• Field of Search: 348/423, 348/845.2, 348/845.3, 348/515, 348/512, 348/460, 348/465, 348/474, 375/364, 375/365, 375/366, 375/216, 370/503, 370/509, 370/468, 370/473, 370/389, 345/723, 345/719, 345/726, 345/302
• US Class Current: 348/512
• CPC Class Codes: H04N 21/235 Processing of additional data, e.g. scrambling of additional data or processing content descriptors
• CPC Class Codes: H04N 21/23614 Multiplexing of additional data and video streams arrangements for simultaneous broadcast of plural pieces of information H04H20/28
• CPC Class Codes: H04N 21/242 Synchronization processes, e.g. processing of PCR [Program Clock References] arrangements for synchronising broadcast or distribution via plural systems in broadcast distribution systems H04H20/18
• CPC Class Codes: H04N 21/426 Internal components of the client ; Characteristics thereof H04N5/44 takes precedence
• CPC Class Codes: H04N 21/43074 of additional data with content streams on the same device, e.g. of EPG data or interactive icon with a TV program
• CPC Class Codes: H04N 21/4341 Demultiplexing of audio and video streams
• CPC Class Codes: H04N 21/4348 Demultiplexing of additional data and video streams
• CPC Class Codes: H04N 21/435 Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73
• CPC Class Codes: H04N 21/8547 involving timestamps for synchronizing content
• CPC Class Codes: H04N 5/04 Synchronising for television systems using pulse code modulation H04N7/56

Abstract

Control information is processed in synchronism with audio and video data according to a protocol such as RTP (Real-time Transfer Protocol). In one embodiment, a payload handler receives incoming data packets and forwards them to either a data control filter or an audio packet handler. The data control filter determines whether the data payload contains video data or control information and forwards video data to a video data packet handler and data control information to a data handler. The data control information can include an action identifier field (e.g., containing a “display” command) and a data object field (e.g., identifying a file location in a memory) so that the data control filter can display the identified file with the presentation of the other video and audio data. Because the data control information is passed “in-stream” with the audio and video data packets, the payload handler is able to synchronize the presentation of audio and video data with the data event referenced by the data control information.

Claims

1. An apparatus for synchronizing data control information with video data, comprising:

a payload handler adapted to receive a plurality of data packets, each of said data packets including;

a data payload including one of video data and data control information

a first header including a timestamp field; and

a data control header identifying a type of data contained in said data payload;

a data control filter coupled to said payload handler and adapted to receive said data control header and said data payload for each of said data packets;

a data handler coupled to said data control filter and adapted to receive data payloads including data control information, said data control information including a command in an action identifier field and said data handler executing said command;

a video data packet handler coupled to said data control filter and adapted to receive data payloads including video data;

said payload handler synchronizes a transfer of said video data and data control information to said data control filter based on information in said timestamp field;

a memory coupled to said data handler;

a data rendering device coupled to said data handler; and

a display coupled to said data rendering device wherein said data control information further includes a data object field referencing a file location in said memory and said action identifier field includes a display command such that said data handler displays a file located at said file location at said display via said data rendering device.

2. The apparatus of claim 1 further comprising:

a video rendering device coupled to said video data packet handler and said display, said video rendering device adapted to output said video data to said display.

3. The apparatus of claim 2 wherein said data packet further includes a payload type field capable of identifying said data payload as audio data, the apparatus further comprising:

an audio data packet handler coupled to said payload handler and adapted to receive data payloads including audio data; and

an audio rendering device adapted to receive audio data from said audio data packet handler and output said audio data via a speaker, such that said payload handler synchronizes said a transfer of said audio data to said audio data packet handler and the transfer of video data and data control information to said data control filter based on information in said timestamp field.

4. An apparatus for synchronizing data control information with video data, comprising:

a payload handler adapted to receive a plurality of data packets, each of said data packets including;

a data payload including one of video data and data control information

a first header including a timestamp field;

a data control header identifying a type of data contained in said data payload; and

a payload type field capable of identifying said data payload as audio data,

a data control filter coupled to said payload handler and adaopted to receive said data control header and said data payload for each of said data packets;

a data cache coupled to said data control filter and adapted to receive data payloads including data control information, said data control information including a command in an action identifier field and said data handler executing said command;

a video data packet handler coupled to said data control filter and adapted to receive data payloads including video data;

said payload handler synchronizes a transfer of said video data and data control information to said data control filter based on information in said timestamp field;

an audio data packet handler coupled to said payload handler and adapted to receive data payloads including audio data; and

an audio rendering device adapted to receive audio data from said audio data packet handler and output said audio data via a speaker, such that said payload handler synchronizes a transfer of said audio data to said audio data packet handler and the transfer of video data and data control information to said data control filter based on information in said timestamp field.

5. An apparatus for synchronizing data control information with audio data, comprising:

a payload handler adapted to receive a plurality of data packets, each of said data packets including;

a data payload including one of audio data and data control information

a first header including a timestamp field; and

a data control header identifying a type of data contained in said data payload;

a data control filter coupled to said payload handler and adapted to receive said data control header and said data payload for each of said data packets;

a data handler coupled to said data control filter and adapted to receive data payloads including data control information, said data control information including a command in an action identifier field and said data handler executing said command;

an audio data packet handler coupled to said data control filter and adapted to receive data payloads including audio data;

said payload handler synchronizes a transfer of said audio data and data control information to said data control filter based on information in said timestamp field;

a memory coupled to said data handler;

a data rendering device coupled to said data handler; and

a display coupled to said data rendering device wherein said data control information further includes a data object field referencing a file location in said memory and said action identifier field includes a display command such that said data handler displays a file at said file location at said display via said data rendering device.

6. The apparatus of claim 5 further comprising:

an audio rendering device coupled to said payload handler and adapted to receive audio data from said audio data packet handler and output said audio data via a speaker.

7. The apparatus of claim 6 wherein said data packet further includes a payload type field capable of identifying said data payload as video data, the apparatus further comprising:

a video data packet handler coupled to said payload handler and adapted to receive data payloads including video data; and

a video rendering device adapted to receive video data from said video data packet handler and output said audio data via a speaker, such that said payload handler synchronizes a transfer of said audio data to said audio data packet handler and the transfer of video data and data control information to said data control filter based on information in said timestamp field.

8. The apparatus of claim 5 wherein said data packet further includes a payload type field capable of identifying said data payload as video data, the apparatus further comprising:

a video data packet handler coupled to said payload handler and adapted to receive data payloads including video data; and

a video rendering device coupled to said video data packet handler and said display, said video rendering device adapted to output said video data to said display, such that said payload handler synchronizes a transfer of said video data to said video data packet handler and the transfer of audio data and data control information to said data control filter based on information in said timestamp field.

9. A method of synchronizing data control information with video data, comprising:

receiving a plurality of data packets at a payload handler, each of said data packets including a data payload including one of video data and data control information, a first header including a timestamp field, and a data control header identifying a type of data contained in said data payload;

transferring said data control header and said data payload for each of said data packets to a data control filter, said payload handler synchronizes the transferring of said video data and data control information to said data control filter based on information in said timestamp field;

determining whether said data payload includes data control information via said data control header;

transferring said data payload to a data handler if said data payload includes data control information, said data control information including a command in an action identifier field and a data object field referencing a file location in a memory and said action identifier field includes a display command;

executing the command in said data control information by said data handler; and

transferring said data payload to a video data packet handler if said data payload includes video data; and

displaying a file stored at said file location in said memory at a display.

10. The method of claim 9 further comprising:

outputting said video data to said display via a video rendering device coupled to said video data packet handler.

11. The method of claim 10 wherein said data packet further includes a payload type field capable of identifying said data payload as audio data, the method further comprising:

transferring data payloads including audio data to an audio data packet handler coupled to said payload handler; and

outputting said audio data to a speaker by an audio rendering device, such that said payload handler synchronizes said transferring of said audio data to said audio data packet handler and the transferring of video data and data control information to said data control filter based on information in said timestamp field.

12. A method of synchronizing data control information with video data, comprising:

receiving a plurality of data packets at a payload handler, each of said data packets including a data payload including one of video data and data control information, a first header including a timestamp field, a data control header identifying a type of data contained in said data payload and a payload type field capable of identifying said data payload as audio data;

transferring said data control header and said data payload for each of said data packets to a data control filter, said payload handler synchronizes the transferring of said video data and data control information to said data control filter based on information in said timestamp field;

determining whether said data payload includes data control information via said data control header;

transferring said data payload to a data handler if said data payload includes data control information, said data control information including a command in an action identifier field;

executing the command in said data control information by said data handler;

transferring said data payload to a video data packet handler if said data payload includes video data;

transferring data payloads including audio data to an audio data packet handler coupled to said payload handler; and

outputting said audio data to a speaker by an audio rendering device, such that said payload handler synchronizes said transferring of said audio data to said audio data packet handler and the transferring of video data and data control information to said data control filter based on information in said timestamp field.

13. A method of synchronizing data control information with audio data, comprising:

receiving a plurality of data packets at a payload handler, each of said data packets including a data payload including one of video data and data control information, a first header including a timestamp field, and a data control header identifying a type of data contained in said data payload;

transferring said data control header and said data payload for each of said data packets to a data control filter, said payload handler synchronizes the transferring of said audio data and data control information to said data control filter based on information in said timestamp field;

determining whether said data payload includes data control information via said data control header;

transferring said data payload to a data handler if said data payload includes data control information, said data control information including a command in an action identifier field, a data object field referencing a file location in a memory and said action identifier field includes a display command;

executing the command in said data control information by said data handler;

transferring said data payload to an audio data packet handler if said data payload includes audio data; and

displaying a file stored at said file location in said memory at a display.

14. The method of claim 13 wherein said data packet firther includes a payload type field capable of identifying said data payload as video data, the method further comprising:

transferring data payloads including video data to an video data packet handler coupled to said payload handler; and

outputting said video data to said display by a video rendering device, such that said payload handler synchronizes said transferring of said video data to said video data packet handler and the transferring of audio data and data control information to said data control filter based on information in said timestamp field.

15. A method of synchronizing data control information with audio data, comprising:

receiving a plurality of data packets at a payload handler, each of said data packets including a data payload including one of video data and data control information, a first header including a timestamp field, and a data control header identifying a type of data contained in said data payload;

transferring said data control header and said data payload for each of said data packets to a data control filter, said payload handler synchronizes the transferring of said audio data and data control information to said data control filter based on information in said timestamp field;

determining whether said data payload includes data control information via said data control header;

transferring said data payload to a data handler if said data payload includes data control information, said data control information including a command in an action identifier field;

executing the command in said data control information by said data handler;

transferring said data payload to an audio data packet handler if said data payload includes audio data; and

outputting said audio data to a speaker via an audio rendering device coupled to said audio data packet handler.

16. The method of claim 15 wherein said data packet further includes a payload type field capable of identifying said data payload as video data, the method further comprising:

transferring data payloads including video data to an video data packet handler coupled to said payload handler; and

outputting said video data to said display by a video rendering device, such that said payload handler synchronizes said transferring of said video data to said video data packet handler and the transferring of audio data and data control information to said data control filter based on information in said timestamp field.