Multimedia timeline modification in networked client/server systems

• US 6,622,171 B2
• Filed: 09/15/1998
• Issued: 09/16/2003
• Est. Priority Date: 09/15/1998
• Status: Expired due to Term

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Gupta, Anoop, Omoigui, Nosakhare D.
• Primary Examiner(s): Powell, Mark R.
• Assistant Examiner(s): Vaughn, Jr., William C.
• Application Number:
  US09/153,664
• Publication Number:
  US 20020038374A1
• Time in Patent Office: 1,827 Days
• Field of Search: 709/219, 709/217, 709/231, 709/233, 709/234, 709/232, 709/247, 709/225, 709/226, 709/248, 709/246, 709/245, 709/263, 725/41, 725/43, 725/90, 725/94, 725/96, 345/700, 345/704, 345/725, 345/748
• US Class Current: 709/231
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/613 for the control of the source by the destination control signals issued by the client directed to the server or network components specially adapted for selective content distribution H04N21/637
• CPC Class Codes: H04L 65/752 adapting media to network capabilities
• CPC Class Codes: H04L 65/764 at the destination reformatting of additional data in video clients H04N21/4355

Abstract

Multimedia content is streamed over a network system from a server computer to a client computer. The client allows a user to enter a variable playback speed and varies the speed at which the multimedia content is rendered at the client. Time-scale modification technology is used to maintain the original pitch of any audio content, thereby maintaining its intelligibility.

Claims

1. A method of obtaining and presenting multimedia content, comprising the following steps:

accepting a speed designation at a network client from a human user;

a network server selecting a composite media stream of a plurality of composite media streams to be rendered at the network client, each of the plurality of composite media streams corresponding to the same multimedia content and having timelines modified by different degrees, the selected composite media stream having a modified timeline most closely matching the speed designation, the selected composite media stream having a plurality of individual media streams representing the multimedia content, at least one of the individual media streams comprising an audio stream representing audio content having an original pitch, the individual media streams having timelines that are synchronized with each other;

streaming the composite media stream from the network server to the network client;

modifying, at the network client, the timelines of the individual media streams in accordance with the speed designation while keeping the timelines synchronized regardless of the speed designation and while maintaining the original pitch of the audio content;

rendering the composite media stream at the network client in accordance with the modified timelines of the individual media streams; and

wherein the plurality of composite media streams include multiple composite media streams corresponding to the same multimedia content and having the same degree of timeline modification but different degrees of quality, and wherein the selecting the composite media stream further comprises determining an available bandwidth between the network server and the network client and selecting the composite media stream that provides the best quality while requiring no more than the available bandwidth.

2. A method as recited in claim 1, wherein the step of accepting the speed designation is performing during the streaming step.

3. A computer-readable storage medium containing a program for obtaining and presenting multimedia content, the program having instructions that are executable by a network client to perform steps comprising:

making a plurality of forward playback speed designations available for selection;

accepting one of the plurality of speed designations from a human user;

selecting, independently of the accepting step and based on a negotiation protocol between a network server and the network client, a composite media stream of a plurality of composite media streams from the network server to be rendered at the network client, each of the plurality of composite media streams corresponding to the same multimedia content and having timelines modified by different degrees, the selected composite media stream having a modified timeline most closely according with the accepted speed designation, the composite media stream having a plurality of individual media streams representing the multimedia content, the individual media streams having timelines that are synchronized with each other;

streaming the composite media stream from the network server to the network client;

modifying, at the network client, the timelines of the individual media streams in accordance with the speed designation so that the timelines of the plurality of individual media streams match the accepted speed designation while keeping the timelines synchronized regardless of the speed designation;

rendering the composite media stream at the network client in accordance with the modified timelines of the individual media streams; and

wherein the plurality of composite media streams include multiple composite media streams corresponding to the same multimedia content and having the same degree of timeline modification but different degrees of quality, and wherein the selecting the composite media stream further comprises determining an available bandwidth between the network server and the network client and selecting the composite media stream that provides the best quality while requiring no more than the available bandwidth.

4. A computer-readable storage medium as recited in claim 3, wherein the step of accepting the speed designation is performing during the streaming step.

5. A computer-readable storage medium as recited in claim 3, wherein the streaming step comprises streaming the composite media stream at different rates depending on the speed designation from the human user.

6. A networked client/server system, comprising:

a network server;

a plurality of network clients that communicate with the network server over a data communications network;

a plurality of composite media streams available from the network server, each of the plurality of composite media streams corresponding to the same multimedia content and having timelines modified by different degrees, each composite media stream comprising a plurality of individual media streams that can be rendered by the network clients to produce different types of user-perceivable media, the individual media streams including audio streams that can be rendered by the network clients to produce audio, such audio having an original pitch, the individual media streams having timelines that are synchronized with each other;

the network server being configured to select a composite media stream from the plurality of composite media streams having a modified timeline most closely according with a user selected speed designation identified by a network client, the network server further streaming the selected composite media stream to the network client at an appropriate rate determined by the network server based at least in part on the speed designation;

the network clients being responsive to user input to modify the timelines of the individual media streams of the selected composite media stream while keeping the timelines synchronized with each other regardless of the speed designation and while maintaining the original pitch of any audio produced from audio streams; and

wherein the plurality of composite media streams include multiple composite media streams corresponding to the same multimedia content and having the same degree of timeline modification but different degrees of quality, and wherein the network sever is further configured to select the composite media stream by determining an available bandwidth between the network server and the network client and selecting the composite media stream that provides the best quality while requiring no more than the available bandwidth.

7. A networked client/server system as recited in claim 6, wherein each composite media stream includes at least an audio stream and a video stream.

8. A computer-readable storage medium containing a program for streaming multimedia content to a receiving computer, the program having instructions that are executable by a sending computer to perform steps comprising:

receiving a user-selected speed designation for playback of the multimedia content at a receiving computer;

selecting, from a plurality of composite media streams at the sending computer that correspond to the multimedia content and that have modified timelines modified by different degrees, a composite media stream that most closely matches the user-selected speed designation;

then streaming the selected composite media stream from the sending computer to the receiving computer for subsequent timeline modification at the receiving computer so that the timelines of individual streams of the selected composite media stream match the user-selected speed designation; and

wherein the plurality of composite media streams include multiple composite media streams corresponding to the multimedia content and having the same degree of timeline modification but different degrees of quality, and wherein the selecting the composite media stream further comprises determining an available bandwidth between the sending computer and the receiving computer and selecting the composite media stream that provides the best quality while requiring no more than the available bandwidth.

9. A computer-readable storage medium as recited in claim 8, wherein the composite media stream includes at least an audio stream and a video stream.

10. A computer-readable storage medium as recited in claim 8, wherein the composite media stream includes at least an audio stream representing audio content and wherein the modifying step comprises modifying the timeline of the composite media stream while maintaining intelligibility of the audio content.

11. A method of streaming particular multimedia content from a sending computer to a receiving computer, the method comprising the following steps:

storing multiple media streams at the sending computer corresponding to the multimedia content, said multiple media streams having timelines that are modified by different degrees for forward playback and including streams corresponding to at least first and second media types, wherein a pair of the first and second types of media streams can be rendered to produce the multimedia content;

accepting a user-selected speed designation for playback of the multimedia content at a receiving computer;

the sending computer selecting those stored media streams of the first and second types that have modified timelines most closely according with the speed designation;

streaming the selected media streams from the sending computer to the receiving computer;

further modifying the timelines of the selected media streams at the receiving computer in accordance with the speed designation before rendering the selected media streams; and

wherein the multiple media streams include multiple media streams corresponding to the multimedia content and having timelines modified by the same degree but different degrees of quality, and wherein the selecting the stored media streams further comprises determining an available bandwidth between the sending computer and the receiving computer and selecting the media streams that provide the best quality while requiring no more than the available bandwidth.

12. A method as recited in claim 11, further comprising:

checking whether the user-selected speed designation exactly matches a speed designation of the selected media streams; and

further modifying the timelines of the selected media streams at the receiving computer only if the user-selected speed designation does not exactly match the speed designation of the selected media streams.

13. A method of obtaining and presenting multimedia content, comprising the following steps:

selecting multimedia content to be rendered at a network client;

accepting a speed designation at the network client from a human user independently of the selecting step;

selecting, from a plurality of composite media streams at a network server that correspond to the multimedia content, a composite media stream that has a modified timeline most closely according with the speed designation, wherein the selecting occurs via a negotiation protocol between the network client and the network server based on timeline modification indicated by the speed designation, wherein the plurality of composite media streams include multiple composite media streams corresponding to the multimedia content and having the same degree of timeline modification but different degrees of quality, and wherein the selecting the composite media stream further comprises determining an available bandwidth between the network server and the network client and selecting the composite media stream that provides the best quality while requiring no more than the available bandwidth;

the network server streaming the composite media stream to the network client at an appropriate rate determined by the network server and based at least in part on the speed designation, the composite media stream representing the selected multimedia content and including a plurality of individual streams;

the network client further modifying timelines of the plurality of individual streams so that the timelines of the plurality of individual streams match the speed designation;

rendering the composite media stream at a speed based on the further modified timelines as it is streamed to produce the multimedia content at the network client;

varying the speed of the multimedia content depending on the speed designation from the human user while keeping timelines of the plurality of individual streams synchronized regardless of the speed designation.

14. A method as recited in claim 13, wherein the step of varying the speed of the presented multimedia content is performed by receiving different composite media streams from the network server, wherein the different composite media streams have different timelines.

15. A method as recited in claim 13, comprising a further step of allowing the user to change the speed designation during rendering of the composite media stream.

16. A method as recited in claim 13, wherein the varying step uses time-scale-modification to change the speed of the multimedia content while maintaining its intelligibility.

17. A method as recited in claim 13, wherein:

the multimedia content includes audio content having an original pitch; and

the varying step uses time-scale-modification to change the speed of the multimedia content while maintaining the original pitch of the audio content.

18. A method as recited in claim 13 wherein more speed designations are available to the human user than there are composite media streams stored at the network server.

19. A method as recited in claim 13, wherein the speed designation is one of a plurality of speed designations available to the human user, and wherein each of the plurality of speed designations corresponds to a forward playback speed faster than a normal playback speed.

20. A method as recited in claim 13, wherein the speed designation is one of a plurality of speed designations available to the human user, wherein at least one of the plurality of speed designations corresponds to a forward playback speed faster than a normal playback speed, and wherein at least one of the plurality of speed designations corresponds to a forward playback speed slower than the normal playback speed.

21. A method as recited in claim 13, wherein at least one of the composite media streams is modified for playback faster than a normal playback speed, and wherein at least one of the composite media streams is modified for playback slower than the normal playback speed.

22. A method as recited in claim 13, wherein the speed designation is a speed factor relative to an original playback speed of the multimedia content.

23. A method as recited in claim 13, further comprising the network server selecting the composite media stream based at least in part on a user preference for a particular quality of one or more of the plurality of individual streams.

24. A method of obtaining and presenting multimedia content, comprising:

selecting multimedia content to be rendered at a network client;

accepting a speed designation at the network client from a human user independently of the selecting of the multimedia content;

selecting, by a network server and based at least in part on the speed designation, a composite media stream to be streamed to the network client, wherein the composite media stream represents the selected multimedia content, and wherein the selected composite media stream has a timeline modified by a factor different than the speed designation;

streaming the composite media stream from the network server;

rendering the composite media stream as it is streamed to produce the multimedia content at the network client;

varying the speed of the multimedia content depending on the speed designation from the human user; and

wherein multiple composite media streams represent the selected multimedia content and have a same degree of timeline modification but different degrees of quality, and wherein the selecting the composite media stream further comprises determining an available bandwidth between the network server and the network client and selecting the composite media stream that provides the best quality while requiring no more than the available bandwidth.

25. A method as recited in claim 24, wherein:

the multimedia content includes audio content having an original pitch; and

the varying uses time-scale-modification to change the speed of the multimedia content while maintaining the original pitch of the audio content.

26. A method of obtaining and presenting multimedia content from an intelligent network server to an intelligent network client, the method comprising:

selecting a composite media stream from the network server to be rendered at the network client, the composite media stream having a plurality of individual media streams representing the multimedia content, at least one of the individual media streams comprising an audio stream representing audio content having an original pitch, the individual media streams having timelines that are synchronized with each other;

accepting a speed designation at the network client from a human user independently of the selecting;

selecting, by the network server, a composite media stream based on the speed designation;

streaming the composite media stream from the network server to the network client;

modifying the timelines of the individual media streams in accordance with the speed designation while keeping the timelines synchronized and while maintaining the original pitch of the audio content;

rendering the composite media stream at the network client in accordance with the modified timelines of the individual media streams; and

wherein the network server maintains a plurality of composite media streams representing the multimedia content and having the same degree of timeline modification but different degrees of quality, and wherein the selecting the composite media stream further comprises determining an available bandwidth between the network server and the network client and selecting the composite media stream that provides the best quality while requiring no more than the available bandwidth.

27. A computer-readable storage medium containing a program for streaming multimedia content to a network client, the program having instructions that are executable by a network server to:

receive a speed designation for playback of the multimedia content at a network client;

compose a composite media stream that represents the multimedia content, the composite media stream having a timeline that is modified in accordance with the speed designation;

determine, based at least in part on the speed designation, a rate at which the timeline-modified composite media stream is to be streamed to the network client;

stream the timeline-modified composite media stream from the network server to the network client at the determined rate; and

wherein multiple composite media streams represent the multimedia content and have the same degree of timeline modification but different degrees of quality, and wherein the instructions that are executable by the network server to compose the composite media stream further cause the network server to determine an available bandwidth between the network server and the network client and select, as streams of the composite media stream, streams that provide the best quality while requiring no more than the available bandwidth.

28. A computer-readable storage medium as recited in claim 27, wherein the instructions that are executable by the network server to compose the composite media stream comprise instructions that are executable by the network server to modify the timeline of the composite media stream in accordance with the speed designation prior to streaming the timeline-modified composite media stream from the network server to the network client at the determined rate.

29. A method of streaming particular multimedia content from a network server to a network client, the method comprising:

storing multiple media streams at the network server corresponding to the multimedia content, said multiple media streams having timelines that are modified by different degrees and including streams corresponding to at least first and second media types, wherein a pair of the first and second types of media streams can be rendered to produce the multimedia content;

accepting a speed designation for playback of the multimedia content at a network client;

selecting those stored media streams of the first and second types that have modified timelines most closely according with the speed designation;

streaming the selected media streams from the network server to the network client at a rate that is based at least in part on the speed designation; and

wherein the multiple media streams include multiple media streams corresponding to the multimedia content and having timelines modified by the same degree but different degrees of quality, and wherein the selecting the media streams further comprises determining an available bandwidth between the network server and the network client and selecting the media streams that provide the best quality while requiring no more than the available bandwidth.

30. A method of obtaining and presenting multimedia content, the method comprising:

selecting multimedia content to be rendered at a network client;

accepting a speed designation at the network client from a human user independently of the selecting;

a network server selecting, from a plurality of composite media streams at a network server that correspond to the multimedia content, a composite media stream that has a modified timeline most closely according with the speed designation;

the network server streaming the composite media stream to the network client at an appropriate rate determined by the network server, the composite media stream representing the selected multimedia content and including a plurality of individual streams;

the network client further modifying timelines of the plurality of individual streams so that the timelines of the plurality of individual streams match the speed designation;

rendering the composite media stream at a speed based on the further modified timelines as it is streamed to produce the multimedia content at the network client;

varying the speed of the multimedia content depending on the speed designation from the human user while keeping timelines of the plurality of individual streams synchronized; and

wherein the plurality of composite media streams include multiple composite media streams corresponding to the multimedia content and having timelines modified by the same degree but different degrees of quality, and wherein the selecting the composite media stream further comprises determining an available bandwidth between the network server and the network client and selecting the composite media stream that provides the best quality while requiring no more than the available bandwidth.

31. A method as recited in claim 30, further comprising the network server selecting the composite media stream based at least in part on a user preference for a particular quality of one or more of the plurality of individual streams.

32. A method of obtaining and presenting multimedia content, the method comprising:

selecting multimedia content to be rendered at a network client;

accepting a speed designation at the network client from a human user independently of the selecting;

a network server modifying, based at least in part on the speed designation, timelines of a plurality of individual streams of a composite media stream representing the selected multimedia content;

the network server streaming the composite media stream to the network client at a rate determined by the network server;

the network client further modifying timelines of the plurality of individual streams so that the timelines of the plurality of individual streams match the speed designation;

rendering the composite media stream at a speed based on the further modified timelines as it is streamed to produce the multimedia content at the network client;

varying the speed of the multimedia content depending on the speed designation from the human user while keeping timelines of the plurality of individual streams synchronized; and

wherein multiple composite media streams correspond to the selected multimedia content and have a same degree of timeline modification but different degrees of quality, the network server further selecting one of the multiple composite media streams by determining an available bandwidth between the network server and the network client and selecting the composite media stream that provides the best quality while requiring no more than the available bandwidth.