Recovering from dropped frames in real-time transmission of video over IP networks

• US 8,929,443 B2
• Filed: 01/09/2009
• Issued: 01/06/2015
• Est. Priority Date: 01/09/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Crinon, Regis J., Ali, Walid, Pinto, Nelson Philip
• Primary Examiner(s): Phillips, Hassan
• Assistant Examiner(s): Mais, Mark A
• Application Number:
  US12/350,975
• Publication Number:
  US 20100177776A1
• Time in Patent Office: 2,188 Days
• Field of Search: None
• US Class Current: 375/240.12
• CPC Class Codes: H04L 12/6418 Hybrid transport
• CPC Class Codes: H04N 19/103 Selection of coding mode or of prediction mode
• CPC Class Codes: H04N 19/105 Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
• CPC Class Codes: H04N 19/166 concerning the amount of transmission errors, e.g. bit error rate [BER]
• CPC Class Codes: H04N 7/52 Systems for transmission of a pulse code modulated video signal with one or more other pulse code modulated signals, e.g. an audio signal or a synchronizing signal assembling of a multiplex stream by combining a video stream with other content or additional data, remultiplexing of multiplex streams, insertion of stuffing bits into the multiplex stream, assembling of a packetised elementary stream at server side H04N21/236; disassembling of a multiplex stream, remultiplexing of multiplex streams, extraction or processing of Service Information, disassembling of packetised elementary stream at client side H04N21/434

Abstract

Technologies for recovering from dropped frames in the real-time transmission of video over an IP network are provided. A video streaming module receives a notification from a receiving module that a data packet has been lost. The video streaming module determines, based on the type of video frame conveyed in the lost packet and the timing of the lost packet in relation to the sequence of video frames transmitted to the receiving module, whether or not a replacement video frame should be sent to the receiving module. If the video streaming module determines a replacement video frame is warranted, then the video streaming module instructs a video encoding module to generate a replacement video frame and then transmits the replacement video frame to the receiving module.

Claims

1. A computer-implemented method for recovering from a loss of a data packet while transmitting a sequence of video frames to a receiver over a network, the method comprising running computer-executable instructions on a computer to perform the operations of:

receiving at the computer a notification from the receiver that a data packet has been lost, the notification specifying a sequence number of the lost data packet;

upon receiving the notification, determining from the sequence number of the lost data packet a type of a video frame conveyed by the lost data packet;

determining the sequence number of a last data packet of a most recent key frame of the sequence of video frames transmitted to the receiver;

determining if the video frame conveyed by the lost data packet comprises a B-frame type;

upon determining that the video frame conveyed by the lost data packet comprises the B-frame type, ignoring the notification;

upon determining that the video frame conveyed by the lost data packet does not comprise the B-frame type, determining if the sequence number of the lost data packet is greater than the sequence number of the last data packet of the most recent key frame of the sequence of video frames transmitted to the receiver;

upon determining that the sequence number of the lost data packet is greater than the sequence number of the last data packet of the most recent key frame, generating a new video frame of an SP-frame type and transmitting the new video frame of the SP-frame type from the computer to the receiver;

upon determining that the sequence number of the lost data packet is not greater than the sequence number of the last data packet of the most recent key frame, determining if the video frame conveyed by the lost data packet comprises the P-frame type;

upon determining that the video frame conveyed by the lost data packet comprises the P-frame type, ignoring the notification; and

upon determining that the video frame conveyed by the lost data packet does not comprise the P-frame type, generating a new video frame comprising a frame of an I-frame type or the SP-frame type, and transmitting the new video frame from the computer to the receiver.

2. The method of claim 1, wherein the sequence of video frames comprises a plurality of video frames of one of the I-frame type, the SP-frame type, the P-frame type, and the B-frame type.

3. The method of claim 1, wherein the most recent key frame comprises the most recent video frame of the SP-frame type transmitted to the receiver.

4. The method of claim 2, wherein the replacement video frame comprises a video frame of the I-frame type or the SP-frame type.

5. The method of claim 1, wherein determining a type of the new video frame to generate and transmit to the receiver upon determining that the video frame conveyed by the lost data packet does not comprise the P-frame type depends upon a number of available SP-frame buffers in a video encoder module.

6. The method of claim 1, wherein the network comprises an Internet Protocol (“

IP”

) network and wherein the data packet comprises a Real-time Transmission Protocol (“

RTP”

) packet.

7. The method claim 1, wherein the sequence of video frames comprises a sequence of video frames encoded according to one of an RTVideo standard, an H.264 standard, an MPEG-2 standard, and a VC-1 standard.

8. The method of claim 1, wherein transmitting the new video frame of the SP-frame type from the computer to the receiver is controlled by a throttling mechanism.

9. A system for recovering from a loss of a data packet in a transmission of a sequence of video frames to a destination computer over a network, the system comprising:

a source computer operatively connected to the network; and

a video streaming module executing on the source computer and operative to

receive a notification from the destination computer that the data packet has been lost, the notification specifying a sequence number of the lost data packet,

determine from the sequence number of the lost data packet a type of a video frame conveyed by the lost data packet,

determine the sequence number of a last data packet of a most recent key frame of the sequence of video frames transmitted to the destination computer,

determine if the video frame conveyed by the lost data packet comprises a B-frame type,

upon determining that the video frame conveyed by the lost data packet comprises the B-frame, type ignore the notification,

upon determining that the video frame conveyed by the lost data packet does not coin rise the B-frame type, determine if the sequence number of the lost data packet is greater than the sequence number of the last data packet of the most recent key frame of the sequence of video frames transmitted to the destination computer,

upon determining that the sequence number of the lost data packet is greater than the sequence number of the last data packet of the most recent key frame, generate a new video frame of an SP-frame type and transmit the new video frame of the SP-frame type to the destination computer,

upon determining that the sequence number of the lost data packet is not greater than the sequence number of the last data packet of the most recent key frame, determine if the video frame conveyed by the lost data packet comprises a P-frame type,

upon determining that the video frame conveyed by the lost data packet comprises the P-frame type, ignore the notification, and

upon determining that the video frame conveyed by the lost data packet does not comprise the P-frame type, generate a new video frame comprising one of a frame of an I-frame type or a frame of the SP-frame type, and transmit the new video frame to the destination computer.

10. The system of claim 9, wherein the sequence of video frames comprises a plurality of video frames of one of the I-frame type, the SP-frame type, the P-frame type, and the B-frame type.

11. The system of claim 9, wherein the most recent key frame comprises the most recent video frame of the SP-frame type transmitted to the destination computer.

12. The system of claim 10, wherein the replacement video frame comprises a video frame of the I-frame type or the SP-frame type.

13. The system of claim 9, wherein determining a type of the new video frame to generate and transmit to the destination computer upon determining that the video frame conveyed by the lost data packet does not comprise the P-frame type depends upon a number of available SP-frame buffers in the video streaming module.

14. The system of claim 9, wherein the network comprises an Internet Protocol (“

IP”

) network and wherein the data packet comprises a Real-time Transmission Protocol (“

RTP”

) packet.

15. The system of claim 9, wherein the sequence of video frames comprises a sequence of video frames encoded according to one of an RTVideo standard, an H.264 standard, an MPEG-2 standard, and a VC-1 standard.

16. The system of claim 9, wherein transmitting the new video frame of the SP-frame type from the computer to the destination computer is controlled by a throttling mechanism.

17. A computer-readable storage medium comprising one of an optical disk, a solid state storage device, or a magnetic storage device, having computer executable instructions stored thereon that, when executed by a computer operatively connected to a network, cause the computer to:

receive a notification from a destination computer operatively connected to the network that a data packet of a plurality of data packets conveying a sequence of video frames has been lost, the notification specifying a sequence number of the lost data packet;

determine if a video frame conveyed by the lost data packet comprises a B-frame type;

upon determining that the video frame conveyed by the lost data packet comprises the B-frame type, ignore the notification;

upon determining that the video frame conveyed by the lost data packet is not the B-frame type, determine if the sequence number of the lost data packet is greater than a sequence number of a last data packet of a most recent video frame of an I-frame type or an SP-frame type transmitted to the destination computer;

upon determining that the sequence number of the lost data packet is greater than the sequence number of the last data packet of the most recent video frame of the I-frame type or the SP-frame type transmitted to the destination computer, generate a new video frame of the SP-frame type to replace the video frame of the lost data packet and transmit the new video frame of the SP-frame type to the destination computer;

upon determining that the sequence number of the lost data packet is not greater than the sequence number of the last data packet of the most recent video frame of the I-frame type or the SP-frame type transmitted to the destination computer, determine if the video frame conveyed by the lost data packet comprises a P-frame type;

upon determining that the video frame conveyed by the lost data packet comprises the P-frame type, ignore the notification; and

upon determining that the video frame conveyed by the lost data packet does not comprise the P-frame type, generate a new video frame of the I-frame type to replace the video frame of the lost data packet and transmit the new video frame of the I-frame type to the destination computer.

18. The computer-readable storage medium comprising one of an optical disk, a solid state storage device, or a magnetic storage device, of claim 17, wherein the sequence of video frames comprises a sequence of video frames encoded according to one of an RTVideo standard, an H.264 standard, an MPEG-2 standard, and a VC-1 standard.

19. The computer-readable storage medium comprising one of an optical disk, a solid state storage device, or a magnetic storage device, of claim 17, wherein determining a type of the new video frame to generate and transmit to the destination computer upon determining that the video frame conveyed by the lost data packet does not comprise the P-frame type depends upon a number of available SP-frame buffers in a video encoder module.

20. The computer-readable storage medium comprising one of an optical disk, a solid state storage device, or a magnetic storage device, of claim 17, wherein transmitting the new video frame of the SP-frame type from the computer to the destination computer is controlled by a throttling mechanism.