Methods and apparatus for implementing control functions in a streamed video display system

• US 6,754,715 B1
• Filed: 11/09/1999
• Issued: 06/22/2004
• Est. Priority Date: 01/30/1997
• Status: Expired due to Term

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Klemets, Anders, del Val, David, Cannon, Anthony
• Primary Examiner(s): Follansbee, John
• Assistant Examiner(s): Nguyen, Dustin
• Application Number:
  US09/435,947
• Time in Patent Office: 1,687 Days
• Field of Search: 725/93, 725/88, 709/219, 709/202, 709/231, 345/501, 345/547, 375/240, 710/35, 348/719
• US Class Current: 709/231
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/612 for unicast
• CPC Class Codes: H04L 65/613 for the control of the source by the destination control signals issued by the client directed to the server or network components specially adapted for selective content distribution H04N21/637
• CPC Class Codes: H04L 65/752 adapting media to network capabilities
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04L 67/61 taking into account QoS or priority requirements
• CPC Class Codes: H04L 67/63 Routing a service request depending on the request content or context
• CPC Class Codes: H04L 69/18 Multiprotocol handlers, e.g. single devices capable of handling multiple protocols
• CPC Class Codes: H04L 69/324 in the data link layer [OSI layer 2], e.g. HDLC
• CPC Class Codes: H04L 9/40 Network security protocols
• CPC Class Codes: H04N 21/23406 involving management of server-side video buffer
• CPC Class Codes: H04N 21/234381 by altering the temporal resolution, e.g. decreasing the frame rate by frame skipping
• CPC Class Codes: H04N 21/23439 for generating different versions
• CPC Class Codes: H04N 21/44004 involving video buffer management, e.g. video decoder buffer or video display buffer
• CPC Class Codes: H04N 21/440281 by altering the temporal resolution, e.g. by frame skipping television signal recording using magnetic recording on tape for reproducing at a rate different from the recording rate H04N5/783
• CPC Class Codes: H04N 21/6373 for rate control , e.g. request to the server to modify its transmission rate flow control in packet networks H04L47/10
• CPC Class Codes: H04N 21/6377 directed to server one-way streaming services wherein the source is controlled by the destination H04L65/613
• CPC Class Codes: H04N 21/6379 directed to encoder , e.g. for requesting a lower encoding rate
• CPC Class Codes: H04N 21/658 Transmission by the client directed to the server
• CPC Class Codes: H04N 21/6587 Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 21/8547 : involving timestamps for synchronizing content
• CPC Class Codes: H04N 7/17336 : Handling of requests in head-ends

Abstract

A method for displaying streamed digital video data on a client computer. The client computer is configured to receive the streamed digital video data from a server computer via a computer network. The streamed digital video data is transmitted from the server computer to the client computer as a stream of video frames. The method includes receiving a first plurality of video frames at the client computer. The plurality of video frames represents a subset of the stream of video frames. The stream of video frames comprises independent playable video frames and dependent playable video frames. The method further includes displaying the first plurality of video frames on a video display terminal associated with the client computer. There is further included issuing a rewind command from the client computer to the server. The rewind command causes a second plurality of video frames of the stream of video frames different from the first plurality of video frames to be streamed from the server computer to the client computer. The second plurality of video frames has been streamed at least once to the client computer.

Claims

1. A computer-readable medium comprising computer-program instructions for transmitting streamed digital video data computer-program instructions being executable by a processor for:

inputting a first plurality of data packets into a server play-out buffer, the first plurality of data packets containing video frames representing the streamed digital video data, an output of the server play-out buffer being configured to be coupled to a network data connection for transmitting the first plurality of data packets to a client computer;

communicating at least a subset of the first plurality of the data packets from the server play-out buffer to a remission buffer having an output coupled to the network data connection;

outputting the first plurality of data packets from the server play-out buffer onto the network data connection for transmitting the first plurality of data packets to the client computer via the network data connection; and

responsive to receiving a request from the client computer for retransmission of one or more data packets of the at least a subset of the first plurality of the data packets, outputting at least one data packet from the retransmission buffer to the network data connection for communication to the client computer.

2. The computer-readable medium of claim 1 further comprising:

responsive to receiving a retransmission request from the client computer, outputting a selected data packet from the retransmission buffer to the network data connection to a retransmission request from said the client computer, said, the selected data packet representing a data packet from the first plurality of data packets that was not satisfactorily received by the client computer.

3. The computer-readable medium of claim 2 further comprising responsive to receiving, by the retransmission buffer, the at least a subset of the first plurality of data packets, discarding any previously buffered data packets from the retransmission buffer.

4. The computer-readable medium of claim 3, wherein the method further comprises sizing the retransmission buffer such that a given data packet remains in the retransmission buffer for a predefined period of time before being discarded, the predefined time period being longer than an average acceptable latency period, the average acceptable latency period being a function of an elapsed amount of time between data packet transmission from a server computer to the client computer to rendering of the data packet by the client computer.

5. The computer-readable medium of claim 1 wherein the video frames represent playable video frames of the streamed digital video data, the playable frames representing frames configured to be displayed while a renderer application associated with the client computer is in play mode.

6. The computer-readable medium of claim 5 wherein the streamed digital video data represents live video and the play mode represents live play mode.

7. The computer-readable medium of claim 5 wherein the streamed digital video data represents real-time video and the play mode represents real-time video mode.

8. The computer-readable medium of claim 1 wherein the streamed digital video data represents a fast forward stream of video frames, the fast forward stream of video frames represents a first stream of video frames encoded differently from a second stream of video frames, the first stream of video frames being configured for normal play by a renderer application associated with the client computer, both the first stream and the second stream pertain to a recorded event, the first stream of video frames, when streamed to the client computer, permits the user to view the recorded event at a faster pace than a pace afforded by the second stream.

9. A server for transmitting streamed digital video data to a client computer, the server being coupled over a network to the client computer, the server comprising:

a processor; and

a memory coupled to the processor, the memory comprising computer-program instructions executable by a processor for;

inputting a first plurality of data packets into a server play-out buffer, the first plurality of data packets containing video frames representing the streamed digital video data, an output of the server play-out buffer being configured to be coupled to a network data connection for transmitting the first plurality of data packets to a client computer;

communicating at least a subset of the first plurality of the data packets from the server play-out buffer to a retransmission buffer having an output coupled to the network data connection;

outputting the first plurality of data packets from the server play-out buffer onto the network data connection for transmitting the first plurality of data packets to the client computer via the network data connection; and

responsive to receiving a request from the client computer for retransmission of the at least a subset of the first plurality of the data packets, outputting at least one data packet from the retransmission buffer to the network data connection, the at least one data packet comprising a selected data packet corresponding to a data packet from the first plurality of data packets that was not satisfactorily received by the client computer.

10. The server of claim 9, wherein the computer-program instructions further comprise instructions for:

responsive to receiving, by the retransmission buffer, the at least a subset of the first plurality of data packets, discarding any previously buffered data packets from the retransmission buffer.

11. The server of claim 10, wherein the computer-program instructions further comprise instructions for sizing the retransmission buffer such that a given data packet remains in the retransmission buffer for a predefined period of time before being discarded, the predefined time period being longer than an average acceptable latency period, the average acceptable latency period being a function of an elapsed amount of time between data packet transmission from a server computer to the client computer to rendering of the data packet by the client computer.

12. The server of claim 9, wherein the video frames represent playable video frames of the streamed digital video data, the playable frames representing frames configured to be displayed while a renderer application associated with the client computer is in play mode.

13. The server of claim 12, wherein the streamed digital video data represents live video and the play mode represents live play mode.

14. The computer-readable medium of claim 12, wherein the streamed digital video data represents real-time video and the play mode represents real-time video mode.

15. The server of claim 9, wherein the streamed digital video data represents a fast forward stream of video frames, the fast forward stream of video frames represents a first stream of video frames encoded differently from a second stream of video frames, the first stream of video frames being configured for normal play by a renderer application associated with the client computer, both the first stream and the second stream pertain to a recorded event, the first stream of video frames, when streamed to the client computer, permits the user to view the recorded event at a faster pace than a pace afforded by the second stream.

16. A computer-readable medium for implementing a streaming video display system, the computer-readable medium comprising computer-program instructions executable by a processor for:

communicating a first plurality of video frames to a client computer, the first plurality of video frames representing a subset of a stream of video frames, the stream of video frames comprising independent playable video frames and dependent playable video frames;

receiving a rewind command and a time parameter from the client computer, the time parameter representing a time stamp associated with the video frame rendered by the client computer immediately before issuance of the rewind command; and

responsive to receiving the rewind command, streaming a second plurality of video frames of the stream different from the first plurality of video frames to the client computer, the second plurality of video frames consisting of at least a subset of the independent playable video frames arranged in reverse chronological order.

17. The computer-readable medium of claim 16, wherein the second plurality of video frames represents rewound independent playable video frames that are prior in time to a video frame rendered immediately before issuing the rewind command and that are sent in reverse chronological order to the client computer.

18. The computer-readable medium of claim 17, wherein the computer-program instructions further comprise instructions for responsive to receiving the time parameter, ascertaining which video frame of multiple video frames in a digital video data file was rendered before issuance of the rewind command, the digital video file representing a copy of the live streamed digital video data.

19. The computer-readable medium of claim 18, wherein the computer-program instructions further comprise instructions for ascertaining a first independent playable video frame of the rewound independent playable video frames by seeking backward in the digital video data file from the video frame rendered immediately before issuing the rewind command for an immediately prior in time independent playable video frame.

20. The computer-readable medium of claim 18, further comprising ascertaining a first independent playable video frame of the rewound independent playable video frames by seeking forward in the digital video data file from the video frame rendered immediately before issuing the rewind command for an immediately later in time independent playable video frame.

21. The computer-readable medium of claim 16, wherein the computer-program instructions further comprise instructions for, in response to a real-time play command issued from the client computer after the rewind command, streaming a third plurality of video frames from the server computer to the client computer for display on the video display terminal, the third plurality of video frames representing video frames that are later in time to a last displayed video frame of the second plurality of the video frames prior to issuing the real-time play command.

22. The computer-readable medium of claim 21, wherein the computer-program instructions further comprise instructions for:

issuing a time parameter associated with the last displayed video frame of the second plurality of the video frames; and

using the time parameter to ascertain the last displayed video frame of the second plurality of the video frames in a digital video data file representing a copy of the live streamed digital video data.

23. The computer-readable medium of claim 22, wherein the computer-program instructions further comprise instructions for ascertaining a first video frame of the third plurality of the video frames by seeking backward in the digital video data file from the last displayed video frame for an immediately prior in time independent playable video frame.

24. The computer-readable medium of claim 22, wherein the computer-program instructions further comprise instructions for ascertaining a first video frame of the third plurality of the video frames by seeking forward in the digital video data file from the last displayed video frame for an immediately later in time independent playable video frame.

25. A server for streaming video, the server comprising:

a processor; and

a memory coupled to the processor, the memory comprising computer-program instructions executable by a processor for;

communicating a first plurality of video frames to client computer, the first plurality of video frames representing a subset of a stream of video frames, the stream of video frames comprising independent playable video frames and dependent playable video frames;

receiving a rewind command and a time parameter from the client computer, the time parameter representing a time stamp associated with the video frame rendered by the client computer immediately before issuance of the rewind command; and

responsive to receiving the rewind command, streaming a second plurality of video frames of the stream different from the first plurality of video frames to the client computer, the second plurality of video frames consisting of at least a subset of the independent playable video frames arranged in reverse chronological order, the subset being prior in time to the video frame rendered by the client computer immediately before issuance of the rewind command.

26. The server of claim 25, wherein the computer-program instructions further comprise instructions for:

responsive to receiving the time parameter, identifying the video frame in a digital video data file as a function of the time stamp, the digital video file representing a copy of the live streamed digital video data.

27. The server of claim 26, wherein the computer-program instructions further comprise instructions for ascertaining a first independent playable video frame of the rewound independent playable video frames by seeking backward in the digital video data file from the video frame rendered immediately before issuing the rewind command for an immediately prior in time independent playable video frame.

28. The server of claim 26, further comprising ascertaining a first independent playable video frame of the rewound independent playable video frames by seeking forward in the digital video data file from the video frame for an immediately later in time independent playable video frame.

29. The server of claim 25, wherein the computer-program instructions further comprise instructions for, in response to a real-time play command issued from the client computer after the rewind command, streaming a third plurality of video frames to the client computer, the third plurality of video frames representing video frames that are later in time to a last client computer displayed video frame of the second plurality of the video frames prior to issuance of the real-time play command.

30. The server of claim 29, wherein the computer-program instructions further comprise instructions for:

issuing a time parameter associated with the last displayed video frame of the second plurality of the video frames; and

using the time parameter to ascertain the last displayed video frame of the second plurality of the video frames in a digital video data file representing a copy of the live streamed digital video data.

31. The server of claim 30, wherein the computer-program instructions further comprise instructions for ascertaining a first video frame of the third plurality of the video frames by seeking backward in the digital video data file from the last displayed video frame for an immediately prior in time independent playable video frame.

32. The server of claim 30, wherein the computer-program instructions further comprise instructions for ascertaining a first video frame of the third plurality of the video frames by seeking forward in the digital video data file from the last displayed video frame for an immediately later in time independent playable video frame.

33. A method for implementing control features in a streamed video display system, the streamed video system comprising a server coupled over a network to a client computer, the method comprising:

receiving a control command and a time parameter from the client computer, the control command indicating that the client computer is transitioning from a first control mode to a second control mode;

responsive to receiving the control command;

ascertaining a first independent video frame of a stream of video frames to transmit to the client computer, the first independent video frame being ascertained as a function of the time parameter;

transmitting a packet sequence number associated with the first independent video frame to the client computer;

streaming a first plurality of video frames of the stream of video frames starting from the first independent video frame to the client computer; and

communicating at least a subset of the first plurality of video frames to a retransmission buffer for retransmission to the client computer in response to receipt of a request for retransmission from the client computer.

34. The method of claim 33 wherein the first control mode is a stop mode and the second control mode is a real-time play mode in which the client computer displays video frames starting from a time specified by the time parameter.

35. The method of claim 33 wherein the first control mode is a stop mode and the second control mode is a live play mode in which the client computer displays live video frames as they are received from an encoder coupled to the server.

36. The method of claim 33 wherein the first control mode is a live play mode and the second control mode is a rewind mode, the live play mode representing a mode wherein the client computer displays live video frames as they are received from an encoder coupled to the server computer, the rewind mode representing a mode wherein the client computer displays independent video frames in reverse chronological order starting from an independent video frame whose time stamp most closely corresponds to the time parameter.

37. The method of claim 33 wherein the first control mode is a real-time play mode and the second control mode is a fast forward mode, the real-time play mode representing a mode wherein the client computer displays pre-stored video frames, the fast forward mode representing a mode wherein the client computer displays fast forwardable video frames starting from an independent video frame whose time stamp most closely corresponds to the time parameter, the fast forwardable video frames being configured to permit a recorded event to be displayed at a faster pace than a pace of the real-time play mode.

38. The method of claim 37 wherein the fast forwardable video frames represent video frames of a fast forward stream of video frames, the fast forward stream of video frames representing a stream of video frames different from a play stream of video frames for the real-time play mode, the fast forward stream and the play stream being encoded from a single recorded event.

39. The method of claim 33 wherein the first control mode is a live play mode and the second mode is a pause mode, the live play mode representing a mode wherein the client computer displays live video frames as they are received from the server computer, the pause mode representing a mode wherein the stream of video frames is transmitted to the client computer at a bandwidth of zero.

40. The method of claim 33 wherein the first control mode is a fast forward mode and the second mode is a live play mode, the live play mode representing a mode wherein the client computer displays live video frames as they are received from an encoder coupled to the server computer, the fast forward mode representing a mode wherein the client computer displays fast forwardable video frames of a stored video stream starting from an independent video frame whose time stamp most closely corresponds to the time parameter, the fast forwardable video frames being configured to permit a recorded event to be displayed at a faster pace than a pace of the live play mode, the first control mode being transitioned to the second control mode automatically without intervention from an user of the client computer upon reaching an end of the stored video stream.

41. A computer-readable medium for implementing control features in a streamed video display system, the computer-readable medium comprising computer-program instructions executable by a processor for:

receiving a control command and a time parameter from a client computer, the control command indicating that the client computer is transitioning from a first control mode to a second control mode;

responsive to receiving the control command;

ascertaining a first independent video frame of a stream of video frames to transmit to the client computer, the first independent video frame being ascertained as a function of the time parameter;

streaming a first plurality of video frames of the stream of video frames starting from the first independent video frame to the client computer;

communicating at least a subset of the first plurality of video frames to a retransmission buffer for retransmission to the client computer in response to receipt of a request for retransmission from the client computer; and

wherein the first control mode and second control modes are respectively selected from a stop mode and a real-time play mode, a stop mode and a live play mode, a live play mode and a rewind mode, a real-time play mode and a fast forward mode, a live play mode and a pause mode, or a fast forward mode and a live play mode.

42. A server for implementing control features in a streamed video display system, the server comprising a processor;

and a memory coupled to the processor, the memory comprising computer-program instructions as recited in claim 41.

43. A method for implementing control features in a streamed video display system, the method comprising:

communicating a control command and a time parameter to a server, the control command indicating that a client computer is transitioning from a first control mode to a second control mode, the time parameter indicating a time stamp of a video frame previously rendered by the client;

responsive to communicating the control command, receiving a first plurality of video frames of a stream of video frames and a packet sequence number from the server, the first plurality of video frames comprising a first independent video frame of the stream of video frames, the first independent video frame being based on the time parameter, the packet sequence number corresponding to the first independent video frame; and

wherein the first control mode and second control modes are respectively selected from a stop mode and a real-time play mode, a stop mode and a live play mode, a live play mode and a rewind mode, a real-time play mode and a fast forward mode, a live play mode and a pause mode, or a fast forward mode and a live play mode.

44. The method of claim 43, wherein the method further comprises:

determining that a data packet of the first plurality of data has not been properly received;

communicating a request for retransmission of the data packet to the server; and

responsive to communicating the request, receiving a retransmission of the data packet from the server, the retransmission originating from a server retransmission buffer.

45. A computer-readable medium for implementing control features in a streamed video display system, the computer-readable medium comprising computer-program instructions executable by a processor for:

communicating a control command and a time parameter to a server, the control command indicating that the client computer is transitioning from a first control mode to a second control mode, the time parameter indicating a time stamp of a video frame previously rendered by the client;

responsive to communicating the control command, receiving a first plurality of video frames of a stream of video frames and a packet sequence number from the server, the first plurality of video frames comprising a first independent video frame of the stream of video frames, the first independent video frame being based on the time parameter, the packet sequence number corresponding to the first independent video frame; and

wherein the first control mode and second control modes are respectively selected from a stop mode and a real-time play mode, a stop mode and a live play mode, a live play mode and a rewind mode, a real-time play mode and a fast forward mode, a live play mode and a pause mode, or a fast forward mode and a live play mode.

46. The computer-readable medium of claim 45, wherein the computer-program instructions further comprise instructions for:

determining that a data packet of the first plurality of data has not been properly received;

communicating a request for retransmission of the data packet to the server; and

responsive to communicating the request, receiving a retransmission of the data packet from the server, the retransmission originating from a server retransmission buffer.

47. A client computer for implementing control features in a streamed video display system, the client computer comprising:

a processor; and

a memory coupled to the processor, the memory comprising computer-program program instructions executable by the processor for;

communicating a control command and a time parameter to a server, the control command indicating that the client computer is transitioning from a first control mode to a second control mode, the time parameter indicating a time stamp of a video frame previously rendered by the client;

responsive to communicating the control command, receiving a first plurality of video frames of a stream of video frames and a packet sequence number from the server, the first plurality of video frames comprising a first independent video frame of the stream of video frames, the first independent video frame being based on the time parameter, the packet sequence number corresponding to the first independent video frame; and

wherein the first control mode and second control modes are respectively selected from a stop mode and a real-time play mode, a stop mode and a live play mode, a live play mode and a rewind mode, a real-time play mode and a fast forward mode, a live play mode and a pause mode, or a fast forward mode and a live play mode.

48. The client computer of claim 47, wherein the computer-program instructions further comprise instructions for:

determining that a data packet of the first plurality of data has not been properly received;

communicating a request for retransmission of the data packet to the server; and

responsive to communicating the request, receiving a retransmission of the data packet from the server, the retransmission originating from a server retransmission buffer.