Audio/video syncronization with no clean points

• US 20040019911A1
• Filed: 07/25/2002
• Published: 01/29/2004
• Est. Priority Date: 07/25/2002
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Gates, Matthijs A., Srinivasan, Jai
• Granted Patent: US 7,313,313 B2
• Time in Patent Office: Days
• US Class Current: 725/90
• CPC Class Codes: H04N 21/2368 Multiplexing of audio and video streams
• CPC Class Codes: H04N 21/42646 for reading from or writing on a non-volatile solid state storage medium, e.g. DVD, CD-ROM
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/4334 Recording operations recording of a television signal H04N5/76; arrangements for recording or accumulating broadcast information or broadcast-related information H04H60/27
• CPC Class Codes: H04N 21/4341 Demultiplexing of audio and video streams
• CPC Class Codes: H04N 21/440281 by altering the temporal resolution, e.g. by frame skipping television signal recording using magnetic recording on tape for reproducing at a rate different from the recording rate H04N5/783
• CPC Class Codes: H04N 5/9267 using time division multiplex of the PCM audio and PCM video signals

Abstract

A computing system that has access to a streamed multimedia sample that is stored on a storage medium applies a zero-based presentation timestamp numbering scheme to the data packets of the multimedia sample, on-the-fly, and after each seek operation. After seeking to a desired starting point, the computing system reads ahead a maximum number of data packets within each media stream of the multimedia sample. A smallest timestamp (normalizing presentation timestamp) is identified and then subtracted from the timestamps of the data packets that are transmitted to the appropriate rendering devices, thereby creating appropriate presentation timestamps for the data packets for synchronizing the playback of the multimedia sample.

Claims

1. In a computing system that has access to streamed multimedia data, the streamed multimedia data including at least two media streams having corresponding data packets, and wherein at least some of the data packets include original timestamps that corresponds with an intended playback synchronization of the multimedia data, a method for synchronizing the playback of the at least two media streams commencing at a starting point that is seeked for within the multimedia data, the method comprising:

an act of seeking to a desired starting point within the streamed multimedia data, the desired starting point corresponding with a desired playback portion of the multimedia data;

an act of identifying a total number of media streams that are included within the desired playback portion of the multimedia data;

an act of reading ahead a maximum number of data packets in each of the total number of media streams from the desired starting point;

an act of identifying a smallest original timestamp of a data packet within the maximum number of data packets, the smallest original timestamp comprising a normalizing presentation timestamp that corresponds with a presentation starting point for commencing the playback of the desired playback portion of the multimedia data;

an act of assigning a presentation timestamp to each one of the maximum number of data packets that has an original timestamp by subtracting the normalizing presentation timestamp from the original timestamp of each data packet; and

an act of transmitting the maximum number of data packets to at least one rendering device for enabling the total number of media streams to be played back according to presentation timestamp on the at least one rendering device with the intended playback synchronization of the multimedia data.

2. A method as recited in claim 1, wherein the desired starting point does not include a cleanpoint.

3. A method as recited in claim 1, wherein the total number of media streams is at least two.

4. A method as recited in claim 3, wherein the total number of media streams includes at least one audio stream and at least one video stream.

5. A method as recited in claim 4, wherein the at least one rendering device includes at least one audio rendering device and at least one video rendering device.

6. A method as recited in claim 1, wherein the total number of media streams is greater than two.

7. A method as recited in claim 1, wherein the maximum number of data packets is a number in a range of between2 and 20.

8. A method as recited in claim 7, wherein the maximum number of data packets is a number in a range of between 5 and 10.

9. A method as recited in claim 1, wherein the streamed multimedia data is accessed from a storage medium that is local to the computing system.

10. A method as recited in claim 1, wherein the streamed multimedia data is accessed from a storage medium that is remote from the computing system.

11. A method as recited in claim 1, wherein each data packet in the maximum number of data packets includes a timestamp.

12. A method as recited in claim 1, wherein only some of the maximum number of data packets include timestamps.

13. A method as recited in claim 1, wherein the act of assigning a presentation timestamp further includes the act of assigning a presentation timestamp to each data packet that has an original timestamp and that is read from the media streams until seeking to a new desired starting point, wherein assigning a presentation timestamp to each data packet is performed by subtracting the normalizing presentation timestamp from the original timestamp of each data packet.

14. A method as recited in claim 1, wherein the act of transmitting the maximum number of data packets further includes the act of transmitting each data packet that is read from the media streams to the at least one rendering device.

15. A method as recited in claim 1, wherein the act of assigning presentation timestamps occurs at the computing system.

16. A method as recited in claim 1, wherein the act of assigning presentation timestamps occurs at the at least one rendering device.

17. A method as recited in claim 1, wherein the maximum number of data packets are queued by the computing system upon performing the act of reading ahead, such that the maximum number of data packets do not need to be read more than once upon seeking to a desired starting point and prior to transmitting the maximum number of data packets to at least one rendering device.

18. In a computing system that has access to streamed multimedia data, the streamed multimedia data including at least two media streams having corresponding data packets, and wherein at least some of the data packets include original timestamps that corresponds with an intended playback synchronization of the multimedia data, a method for synchronizing the playback of the at least two media streams commencing at a starting point that is seeked for within the multimedia data, the method comprising:

an act of seeking to a desired starting point within the streamed multimedia data, the desired starting point corresponding with a desired playback portion of the multimedia data;

a step for identifying a normalizing presentation timestamp upon which to base the playback of the desired playback portion of the multimedia data;

an act of assigning a presentation timestamp to each one of the maximum number of data packets that has an original timestamp by subtracting the normalizing presentation timestamp from the original timestamp of each data packet; and

an act of transmitting the maximum number of data packets to at least one rendering device for enabling the total number of media streams to be played back according to presentation timestamp on the at least one rendering device with the intended playback synchronization of the multimedia data.

19. A method as recited in claim 18, wherein the step for identifying a normalizing presentation timestamp includes:

an act of identifying a total number of media streams that correspond with the desired playback portion of the multimedia data;

an act of reading ahead a maximum number of data packets in each of the total number of media streams from the desired starting point; and

an act of identifying a smallest original timestamp of a data packet within the maximum number of data packets, the smallest original timestamp comprising a normalizing presentation timestamp that corresponds with a presentation starting point for playing back the desired playback portion of the multimedia data.

20. A method as recited in claim 19, wherein the desired starting point does not include a cleanpoint.

21. A method as recited in claim 18, wherein the total number of media streams is at least two.

22. A method as recited in claim 21, wherein the total number of media streams include at least one audio stream and at least one video stream.

23. A method as recited in claim 18, wherein the maximum number of data packets is a number in a range of between 2 and 20.

24. A method as recited in claim 18, wherein each data packet in the maximum number of data packets includes a timestamp.

25. A method as recited in claim 18, wherein the act of assigning presentation timestamps occurs at the computing system.

26. A method as recited in claim 18, wherein the act of assigning presentation timestamps occurs at the at least one rendering device.

27. A method as recited in claim 18, wherein the maximum number of data packets are queued by the computing system upon performing the act of reading ahead, such that the maximum number of data packets do not need to be read more than once upon seeking to a desired starting point and prior to transmitting the maximum number of data packets to at least one rendering device.

28. A method as recited in claim 18, wherein the act of assigning a presentation timestamp further includes the act of assigning a presentation timestamp to each data packet that has an original timestamp and that is read from the media streams until seeking to a new desired starting point, wherein assigning a presentation timestamp to each data packet is performed by subtracting the normalizing presentation timestamp from the original timestamp of each data packet.

29. A method as recited in claim 28, wherein the act of transmitting the maximum number of data packets further includes the act of transmitting each data packet that is read from the media streams to the at least one rendering device.

30. A computer program product for implementing a method in a computing system, wherein the computing system has access to streamed multimedia data, the streamed multimedia data including data packets of at least two media streams, and wherein at least some of the data packets include original timestamps that correspond with an intended playback synchronization of the multimedia, the computer program product comprising:

one or more computer-readable media carrying computer-executable instructions, that when executed at the computing system, cause the computing system to perform a method for synchronizing the playback of the at least two media streams commencing at a starting point that is seeked for within the multimedia data, the method including the acts of;

the act of seeking to a desired starting point within the streamed multimedia data, the desired starting point corresponding with a desired playback portion of the multimedia data;

the act of identifying a total number of media streams that are included within the desired playback portion of the multimedia data;

the act of reading ahead a maximum number of data packets in each of the total number of media streams from the desired starting point;

the act of identifying a smallest original timestamp of a data packet within the maximum number of data packets, the smallest original timestamp comprising a normalizing presentation timestamp that corresponds with a presentation starting point for commencing the playback of the desired playback portion of the multimedia data;

the act of assigning a presentation timestamp to each one of the maximum number of data packets that has an original timestamp by subtracting the normalizing presentation timestamp from the original timestamp of each data packet; and

the act of transmitting the maximum number of data packets to at least one rendering device for enabling the total number of media streams to be played back according to presentation timestamp on the at least one rendering device with the intended playback synchronization of the multimedia data.

31. A computer program product as recited in claim 30, wherein the one or more computer-readable media include system memory.

32. A computer program product as recited in claim 30, wherein the desired starting point does not include a cleanpoint.

33. A computer program product as recited in claim 30, wherein the total number of media streams is at least two.

34. A computer program product as recited in claim 33, wherein the total number of media streams includes at least one audio stream and at least one video stream.

35. A computer program product as recited in claim 30, wherein the maximum number of data packets is a number in a range of between 2 and 20.

36. A computer program product as recited in claim 30, wherein the streamed multimedia data is accessed from a storage medium that is local to the computing system.

37. A computer program product as recited in claim 30, wherein only some of the maximum number of data packets include timestamps.

38. A computer program product as recited in claim 37, wherein the method further includes an act of assigning a presentation timestamp to data packets that do not include timestamps.

39. A computer program product as recited in claim 38, wherein presentation timestamps are assigned to data packets that do not include timestamps based at least in part on a relative position of the data packets within corresponding media streams of the multimedia data.

40. A computer program product as recited in claim 30, wherein the act of assigning presentation timestamps occurs prior to the act of transmitting the maximum number of data packets.

41. A computer program product as recited in claim 30, wherein the maximum number of data packets are queued by the computing system upon performing the act of reading ahead, such that the maximum number of data packets do not need to be read more than once upon seeking to a desired starting point and prior to transmitting the maximum number of data packets to at least one rendering device.

42. A computer program product as recited in claim 41, wherein the maximum number of data packets are queued in system memory of the computing system.

43. A method as recited in claim 30, wherein the act of assigning a presentation timestamp further includes the act of assigning a presentation timestamp to each data packet that has an original timestamp and that is read from the media streams until seeking to a new desired starting point, wherein assigning a presentation timestamp to each data packet is performed by subtracting the normalizing presentation timestamp from the original timestamp of each data packet.

44. A method as recited in claim 43, wherein the act of transmitting the maximum number of data packets further includes the act of transmitting each data packet that is read from the media streams to the at least one rendering device.

45. In a computing system that has access to streamed multimedia data, the streamed multimedia data including at least two media streams having corresponding data packets, and wherein at least some of the data packets include original timestamps that corresponds with an intended playback synchronization of the multimedia data, a method for synchronizing the playback of the at least two media streams in reverse and commencing at a starting point that is seeked for within the multimedia data, the method comprising:

an act of seeking to a desired starting point within the streamed multimedia data, the desired starting point corresponding with a desired playback portion of the multimedia data;

an act of identifying a total number of media streams that are included within the desired playback portion of the multimedia data;

an act of reading backwards a maximum number of data packets in each of the total number of media streams from the desired starting point;

an act of identifying a largest original timestamp of a data packet within the maximum number of data packets, the largest original timestamp comprising a normalizing presentation timestamp that corresponds with a presentation starting point for commencing the playback of the desired playback portion of the multimedia data;

an act of assigning a presentation timestamp to each one of the maximum number of data packets that has an original timestamp by subtracting the original timestamp of each data packet from the normalizing presentation timestamp; and

an act of transmitting the maximum number of data packets to at least one rendering device for enabling the total number of media streams to be played back according to presentation timestamp on the at least one rendering device with the intended playback synchronization of the multimedia data.

46. A method as recited in claim 45, wherein the act of assigning a presentation timestamp further includes the act of assigning a presentation timestamp to each data packet with an original timestamp preceding the desired starting point and that is read from the media streams, and until seeking to a new desired starting point, wherein assigning a presentation timestamp to each data packet is performed by subtracting the original timestamp of each data packet from the normalizing presentation timestamp.

47. A method as recited in claim 46, wherein the act of transmitting the maximum number of data packets further includes the act of transmitting each data packet that is read from the media streams to the at least one rendering device.

48. A method as recited in claim 45, wherein the maximum number of data packets are queued by the computing system upon performing the act of reading backwards, such that the maximum number of data packets do not need to be read more than once upon seeking to a desired starting point and prior to transmitting the maximum number of data packets to at least one rendering device.