Interleaved multiple multimedia stream for synchronized transmission over a computer network

• US 6,449,653 B2
• Filed: 03/25/1997
• Issued: 09/10/2002
• Est. Priority Date: 03/25/1997
• Status: Expired due to Term

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Klemets, Anders Edgar, Chou, Philip A.
• Primary Examiner(s): Rinehart, Mark H.
• Assistant Examiner(s): Vaughn, Jr., William C.
• Application Number:
  US08/826,345
• Publication Number:
  US 20010013068A1
• Time in Patent Office: 1,995 Days
• Field of Search: 370/79, 370/601, 370/431, 709/231, 709/203, 709/247, 709/248, 709/236, 709/232, 709/217, 709/234, 709/219, 348/423, 348/462, 348/512, 348/513, 348/515, 348/516, 725/2, 725/4, 725/87, 725/94, 725/101
• US Class Current: 709/231
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/752 adapting media to network capabilities
• CPC Class Codes: H04L 65/762 at the source reformatting of additional data in video distribution servers H04N21/2355
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04L 9/40 Network security protocols

Abstract

The production of an interleaved multimedia stream for servers and client computers coupled to each other by a diverse computer network which includes local area networks (LANs) and/or wide area networks (WANs) such as the internet. Interleaved multimedia streams can include compressed video frames for display in a video window, accompanying compressed audio frames and annotation frames. In one embodiment, a producer captures separate video/audio frames and generates an interleaved multimedia file. In another embodiment, the interleaved file include annotation frames which provide either pointer(s) to the event(s) of interest or include displayable data embedded within the annotation stream. The interleaved file is then stored in the web server for subsequent retrieval by client computer(s) in a coordinated manner, so that the client computer(s) is able to synchronously display the video frames and displayable event(s) in a video window and event window(s), respectively. In some embodiments, the interleaved file includes packets with variable length fields, each of which are at least one numerical unit in length.

Claims

1. In a computer having a processor and memory, said computer useful in association with a web server coupled to a client computer via a network, a method for producing an interleaved multimedia file from a video file and an audio file, the method comprising:

retrieving a first video frame from the video file, said first video frame including a video timestamp, in response to determining a video frame buffer is empty;

retrieving a first audio frame from the audio file, said first audio frame including a audio timestamp, in response to determining an audio frame buffer is empty;

writing the first video frame to a first packet of the interleaved file in response to determining that the video timestamp is less than or equal to the audio timestamp,

retrieving a second video frame from the video file in response to determining that the video timestamp is less than or equal to the audio timestamp,

writing the first audio frame to a second packet of the interleaved file in response to determining that the audio timestamp is less than or equal to the video timestamp, and

retrieving a second audio frame from the audio file in response to determining that the audio timestamp is less than or equal to the video timestamp;

writing a size of a variable packet length field of the first packet of the interleaved multimedia file to a length field of the variable packet length field of the first packet of the interleaved multimedia file; and

writing at least one null number into a null field of the variable packet length field of the first packet of the interleaved multimedia file, a number of null numbers written into the null field being selected as a function of a size of the first video frame.

2. The method of claim 1, wherein said first and second packet each have a variable packet length field having a size.

4. The method of claim 1 further comprising writing an annotation frame from an annotation file into said interleaved file.

3. In a computer having a processor and memory, said computer useful in association with a web server coupled to a client computer via a network, a method for producing an interleaved multimedia file from a video file and an audio file, the method comprising:

retrieving a first video frame from the video file, said first video frame including a video timestamp, upon determining that a video frame buffer is empty;

retrieving a first audio frame from the audio file, said first audio frame including an audio timestamp, upon determining that an audio frame buffer is empty; and

writing the first video frame to a first packet of the interleaved file upon determining that the video timestamp is less than or equal to the audio timestamp,

retrieving a second video frame from the video file upon determining that the video timestamp is less than or equal to the audio timestamp,

writing the first audio frame to a second packet of the interleaved file upon determining that the audio timestamp is less than or equal to the video timestamp, and

retrieving a second audio frame from the audio file upon determining that the audio timestamp is less than or equal to the video timestamp, wherein said first and second packet each have a variable packet length field having a size,

wherein the size of the variable packet length field is at least one numerical unit, and writing the first video frame to the first packet includes

writing the size into a length of the variable packet length field of the first packet,

writing a null number into a null field of the variable packet length field of the first packet upon determining the size of the first video frame is between one numerical unit and two numerical units, and

writing three null numbers into the null field of the variable packet length field of the first packet upon determining the size of the first video frame is greater than two numerical units.

5. A producer useful for generating an interleaved file configured to provide a synchronized playback of a video file and an audio file on a client computer, the producer comprising:

a capture module configured to generate an interleaved file from video frames from the video file and audio frames from the audio file, based on timestamps of the video frames and the audio frames, the video frames and audio frames both having packets having variable packet length fields, the capture module configured to write a size of a variable packet length field to a length field of a variable packet length field of a packet of the interleaved multimedia file and to write at least one null number into a null field of the variable packet length field of the first packet of the interleaved multimedia file, a number of null numbers written into the null field being selected as a function of a size of a video frame; and

an author module configured to combine said interleaved file with a plurality of annotation frames, based on the timestamps of the video frames, the audio frames and the annotation frames.

6. The producer of claim 5 wherein said video and audio frames are stored in packets with variable packet length fields.

7. A computer-readable medium useful in association with a computer system having a processor and memory, the computer-readable medium comprising computer-readable code instructions configured to cause said computer system to execute a method comprising:

retrieving a first video frame from a video file, said first video frame including a video timestamp in response to determining that a video frame buffer is empty;

retrieving a first audio frame from an audio file, said first audio frame including an audio timestamp in response to determining that an audio frame buffer is empty;

writing the first video frame to a first packet of the interleaved file in response to determining that the video timestamp is less than or equal to the audio timestamp;

retrieving a second video frame from the video file upon determining that the video timestamp is less than or equal to the audio timestamp;

writing the first audio frame to a second packet of the interleaved file in response to determining that the audio timestamp is less than or equal to the video timestamp;

retrieving a second audio frame from the audio file upon determining that the audio timestamp is less than or equal to the video timestamp;

writing a size of a variable packet length field of the first packet of the interleaved multimedia file to a length field of the variable packet length field of the first packet of the interleaved multimedia file; and

writing at least one null number into a null field of the variable packet length field of the first packet of the interleaved multimedia file, a number of null numbers written into the null field being selected as a function of a size of the first video frame.

8. The computer-readable medium of claim 7, wherein said first and second packet each have a variable packet length field having a size.

10. The computer readable medium of claim 7 further comprising computer-readable code instructions configured to cause said computer system to execute writing an annotation frame from an annotation file into said interleaved file.

9. A computer-readable medium useful in association with a computer system having a processor and memory, the computer-readable medium comprising computer-readable code instructions configured to cause said computer system to execute a method comprising:

retrieving a first video frame from a video file, said first frame including a video timestamp upon determining that a video frame buffer is empty;

retrieving a first audio frame from an audio file, said first audio frame including an audio timestamp upon determining that an audio frame buffer is empty;

writing the first video frame to a first packet of an interleaved file upon determining that the video timestamp is less than or equal to the audio timestamp;

retrieving a second video frame from the video file upon determining that the video timestamp is less than or equal to the audio timestamp;

writing the first audio frame to a second packet of the interleaved file upon determining that the audio timestamp is less than or equal to the video timestamp; and

retrieving a second audio frame from the audio file upon determining that the audio timestamp is less than or equal to the video timestamp,

wherein said first and second packet each have a variable packet length field having a size, and

wherein the size of the variable packet length field is at least one numerical unit, and writing the first video frame to the first packet includes

writing the size into a length field of the variable packet length field of the first packet,

writing a null number into a null field of the variable packet length field of the first packet upon determining the size of the first video frame is between one numerical unit and two numerical units,

writing three null number into a null field of the variable packet length field of the first packet upon determining the size of the first video frame is greater than two numerical units.

11. In a computer having a processor and memory, said computer useful in association with a web server coupled to a client computer via a network, a method for producing an interleaved multimedia file from a video file and an audio file, the method comprising:

retrieving a first video frame from the video file, said first video frame including a video timestamp, upon determining that a video frame buffer is empty;

retrieving a first audio frame from the audio file, said first audio frame including an audio timestamp, upon determining that an audio frame buffer is empty; and

writing the first video frame to a first packet of the interleaved file upon determining that the video timestamp is less than or equal to the audio timestamp;

retrieving a second video frame from the video file upon determining that the video timestamp is less than or equal to the audio timestamp;

writing the first audio frame to a second packet of the interleaved file upon determining that the audio timestamp is less than or equal to the video timestamp; and

retrieving a second audio frame from the audio file upon determining that the audio timestamp is less than or equal to the video timestamp, wherein said first and second packet each have a variable packet length field having a size, wherein the size of the variable packet length field is at least one numerical unit, and writing the first video frame to the first packet includes

writing the size into a length field of the variable packet length field of the first packet, and

writing a number into a field of the variable packet length field, wherein the number corresponds to the size of the video frame.

12. A computer-readable medium useful in association with a computer system having a processor and memory, the computer-readable medium comprising computer-readable code instructions configured to cause said computer system to execute a method comprising:

retrieving a first video frame from a video file, said first video frame including a video timestamp upon determining that a video frame buffer is empty;

retrieving a first audio frame from an audio file, said first audio frame including an audio timestamp upon determining that an audio frame buffer is empty;

writing the first video frame to a first packet of the interleaved file upon determining that the video timestamp is less than or equal to the audio timestamp;

retrieving a second video frame from the video file upon determining that the video timestamp is less than or equal to the audio timestamp;

writing the first audio frame to a second packet of the interleaved file upon determining that the audio timestamp is less than or equal to the video timestamp; and

retrieving a second audio frame from the audio file upon determining that the audio timestamp is less than or equal to the video timestamp;

wherein said first and second packet each have a variable packet length field having a size,

wherein the size of the variable packet length field is at least one numerical unit, and writing the first video frame to the first packet includes

writing the size into a length field of the variable packet length field of the first packet,

writing a null number into a null field of the variable packet length field of the first packet upon determining the size of the first video frame is between one numerical unit and two numerical units,

writing three null numbers into the null field of the variable packet length field of the first packet upon determining the size of the first video frame is greater than two numerical units.

13. In a computer having a processor and a memory, the computer useful in association with a web server coupled to a client computer via a network, a method for producing an interleaved multimedia file from a video file and an audio file, the method comprising:

retrieving a first video frame from the video file in response to determining that a video frame buffer is empty, the first video frame having a video timestamp;

retrieving a first audio frame from the audio file in response to determining that an audio frame buffer is empty, the first audio frame having an audio timestamp;

writing the first video frame to a first packet of the interleaved multimedia file in response to determining that the video timestamp is not greater than the audio timestamp;

retrieving a second video frame from the video file in response to determining that the video timestamp is not greater than the audio timestamp;

writing the first audio frame to a second packet of the interleaved multimedia file in response to determining that the audio timestamp is not greater than the video timestamp;

retrieving a second audio frame from the audio file in response to determining that the audio timestamp is not greater than the video timestamp; and

representing a length of a variable packet length field of a packet of the interleaved multimedia rule such that the length can be determined from one or more null numbers included within the variable packet length field.

14. In a computer having a processor and a memory, the computer useful in association with a web server coupled to a client computer via a network, a method for producing an interleaved multimedia file from a video file and an audio file, the method comprising:

retrieving a first video frame from a video file in response to determining that a video frame buffer is empty, the first video frame having a video timestamp;

retrieving a first audio frame from an audio file in response to determining that an audio frame buffer is empty, the first audio frame having an audio timestamp;

writing the first video frame to a first packet of the interleaved multimedia file in response to determining that the video timestamp is not greater than the audio timestamp;

retrieving a second video frame from the video file in response to determining that the video timestamp is not greater than the audio timestamp;

writing the first audio frame to a second packet of the interleaved multimedia file in response to determining that the audio timestamp is not greater than the video timestamp;

retrieving a second audio frame from the audio file in response to determining that the audio timestamp is not greater than the video timestamp;

writing a size of a variable packet length field of the first packet of the interleaved multimedia file to a length field of the variable packet length field of the first packet of the interleaved multimedia file; and

writing at least one reserved value to the variable packet length field of the first packet of the interleaved multimedia file, a number of reserved values written being selected as a function of a size of the first video frame.

15. A producer useful for generating an interleaved multimedia file configured to provide a synchronized playback of a video file and an audio file on a client computer, the producer comprising:

a capture module configured to generate an interleaved multimedia file from video frames from the video file and audio frames from the audio file, based on timestamps of the video frames and the audio frames, the video frames and audio frames both having packets having variable packet length fields, the capture module configured to represent a length of a variable packet length field of a packet of the interleaved multimedia file such that the length can be determined from one or more null numbers included within the variable packet length field; and

an author module configured to combine the interleaved multimedia file with a plurality of annotation frames based on the timestamps of the video frames, the audio frames, and the annotation frames.

16. A computer-readable medium useful in association with a computer system having a processor and a memory, the computer-readable medium having stored thereon computer-readable instructions configured to cause the computer system to, upon execution:

retrieve a first video frame from the video file in response to determining that a video frame buffer is empty, the first video frame having a video timestamp;

retrieve a first audio frame from the audio file in response to determining that an audio frame buffer is empty, the first frame audio having an audio timestamp;

write the first video frame to a first packet of an interleaved multimedia file in response to determining that the video timestamp is not greater than the audio timestamp;

retrieve a second video frame from the video file in response to determining that the video timestamp is not greater than the audio timestamp;

write the first audio frame to a second packet of the interleaved multimedia file in response to determining that the audio timestamp is not greater than the video timestamp;

retrieve a second audio frame from the audio file on response to determining that the audio timestamp is not greater than the video timestamp; and

represent a length of variable packet length field of a packet of the interleaved multimedia file such that the length can be determined from one or more null numbers included within the variable packet length field.