Systems and methods for error resilience and random access in video communication systems

• US 8,804,848 B2
• Filed: 09/21/2011
• Issued: 08/12/2014
• Est. Priority Date: 12/08/2005
• Status: Active Grant

• Current Assignee: Vidyo Incorporated (Enghouse Systems Limited)
• Original Assignee: Vidyo Incorporated (Enghouse Systems Limited)
• Inventors: Cipolli, Stephen, Civanlar, Reha, Eleftheriadis, Alexandros, Lennox, Jonathan, Sasson, Roi, Saxena, Manoj, Shapiro, Ofer
• Primary Examiner(s): Dang, Hung
• Assistant Examiner(s): Park, Sunghyoun
• Application Number:
  US13/238,624
• Publication Number:
  US 20120069135A1
• Time in Patent Office: 1,056 Days
• Field of Search: None
• US Class Current: 375/240.27
• CPC Class Codes: H04L 1/1607 Details of the supervisory signal
• CPC Class Codes: H04L 1/1812 Hybrid protocols; Hybrid automatic repeat request [HARQ]
• CPC Class Codes: H04L 1/1848 Time-out mechanisms
• CPC Class Codes: H04L 65/752 adapting media to network capabilities
• CPC Class Codes: H04L 65/756 adapting media to device capabilities
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/166 concerning the amount of transmission errors, e.g. bit error rate [BER]
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/31 in the temporal domain
• CPC Class Codes: H04N 19/37 with arrangements for assigning different transmission priorities to video input data or to video coded data
• CPC Class Codes: H04N 19/463 by compressing encoding parameters before transmission
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 19/573 Motion compensation with multiple frame prediction using two or more reference frames in a given prediction direction
• CPC Class Codes: H04N 19/577 Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
• CPC Class Codes: H04N 19/58 Motion compensation with long-term prediction, i.e. the reference frame for a current frame not being the temporally closest one H04N19/23 takes precedence
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/65 using error resilience
• CPC Class Codes: H04N 19/66 : involving data partitioning, i.e. separation of data into packets or partitions according to importance
• CPC Class Codes: H04N 19/67 : involving unequal error protection [UEP], i.e. providing protection according to the importance of the data
• CPC Class Codes: H04N 19/70 : characterised by syntax aspects related to video coding, e.g. related to compression standards
• CPC Class Codes: H04N 19/89 : involving methods or arrangements for detection of transmission errors at the decoder
• CPC Class Codes: H04N 21/2662 : Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 21/4621 : Controlling the complexity of the content stream or additional data, e.g. lowering the resolution or bit-rate of the video stream for a mobile client with a small screen arrangements for using the results of monitoring on user's side in broadcast systems H04H60/65; flow control in packet networks H04L47/10
• CPC Class Codes: H04N 21/631 : Multimode Transmission, e.g. transmitting basic layers and enhancement layers of the content over different transmission paths or transmitting with different error corrections, different keys or with different transmission protocols
• CPC Class Codes: H04N 21/6375 : for requesting retransmission , e.g. of data packets lost or corrupted during transmission from serverARQ protocols H04L1/18; implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP] H04L69/16
• CPC Class Codes: H04N 21/6583 : Acknowledgement

Abstract

Systems and methods for error resilient transmission and for random access in video communication systems are provided. The video communication systems are based on single-layer, scalable video, or simulcast video coding with temporal scalability, which may be used in video communication systems. A set of video frames or pictures in a video signal transmission is designated for reliable or guaranteed delivery to receivers using secure or high reliability links, or by retransmission techniques. The reliably-delivered video frames are used as reference pictures for resynchronization of receivers with the transmitted video signal after error incidence and for random access.

Claims

1. A system for media communications between a transmitting endpoint and one or more receiving endpoint(s) over a communications network, the network having one or more layered media transport link(s) including a more reliable media transport link to or from each endpoint, the system comprising:

an encoder which encodes transmitted media as frames in a threaded coding structure having a number of different frame types, the frames including a sequence of reference frames;

a decoder which decodes received media based on reference frames therein; and

a particular select subset of the frames in the threaded coding structure that is designated as a type (“

R”

) for reliable transport to the one or more receiving endpoint(s) over the more reliable transport link,

wherein the subset of the frames of the type R is particularly selected to include at least the frames of the lowest temporal layer in the threaded coding structure and such that the decoder can decode at least a portion of the received media based on a reliably received frame of the type R after packet loss or error and thereafter is synchronized with the encoder,

wherein reliable transport of the reference frames of the type R comprises retransmissions over the more reliable transport link based on a protocol of Acknowledgment (ACK) and/or No Acknowledgment (NACK) messages, and

wherein the particular select subset of the frames that is designated as a type R for reliable transport to the decoder over the more reliable transport link comprises intra and inter macroblocks, and wherein the encoder further comprises a reliability and random access (RRC) module that recalculates the allocation of intra macroblocks in coding future frames in response to packet loss in or a new endpoint joining a current communication session.

2. The system of claim 1 further comprising processing modules for processing type R frames, wherein the processing modules are disposed at the encoder, and/or the decoder, and/or an intermediate network server at ends of the more reliable media transport link.

3. The system of claim 1 comprising at least one of a single-layer codec, a scalable codec, and a simulcast codec.

4. The system of claim 1 comprising a codec conforming to H.264 AVC, wherein the particular select subset of the frames of the type R for reliable transport to the decoder over the more reliable transport link includes a frame that is marked as a long term reference picture, and wherein the decoding of at least a portion of the received media based on a reliably received reference frame of the type R by the decoder is controlled by memory management control operation (MMCO) commands.

5. The system of claim 1 wherein the encoder and any intermediate server in the network comprise a timer with a preset time period, and wherein reliable transport of frames of the type R comprises retransmissions over the more reliable transport link based on non-receipt of an ACK message within the set time period.

6. The system of claim 1 wherein the reliable transport of a reference frames of the type R comprises retransmissions over the more reliable transport link based on receipt of a NACK message.

7. The system of claim 1 wherein the encoder, which encodes transmitted media as frames in a threaded coding structure, encodes frames of the type R for transmission according to a preset schedule.

8. The system of claim 1 wherein the encoder, which encodes transmitted media as frames in a threaded coding structure, designates a specific transmitted frame of the type R as a reference picture for future use after confirming receipt of the specific frame by all intended recipients.

9. The system of claim 8 wherein the encoder, which encodes transmitted media as frames in a threaded coding structure, abandons a specific frame of the type R as a candidate reference picture for future use upon unsuccessful reliable transmission of the specific frame to any intended recipient.

10. The system of claim 1, further comprising at least one bridge that is coupled to the transmitting endpoint and to the one or more receiving endpoint(s) so that media communication between the transmitting endpoint and the one or more receiving endpoint(s) occurs through the at least one bridge, the at least one bridge being configured to reliably receive the subset of the frames of the type R from the transmitting endpoint.

11. The system of claim 10 wherein the at least one bridge is further configured to transmit a NACK message to the transmitting endpoint when it detects a lost packet in order to instruct the transmitting endpoint to retransmit it, and wherein the receiving endpoint(s) are further configured to transmit a NACK message to the transmitting endpoint when they detect a lost packet in order to instruct the transmitting endpoint to retransmit it.

12. A system for media communications between a transmitting endpoint and one or more receiving endpoint(s) over a communications network, the network having one or more layered media transport link(s) including a more reliable media transport link to or from each endpoint, the system comprising:

an encoder which encodes transmitted media as frames in a threaded coding structure having a number of different frame types, the frames including a sequence of reference frames;

a decoder which decodes received media based on reference frames therein;

a particular select subset of the frames in the threaded coding structure that is designated as a type (“

R”

) for reliable transport to the one or more receiving endpoint(s) over the more reliable transport link,

wherein the subset of the frames of the type R is particularly selected to include at least the frames of the lowest temporal layer in the threaded coding structure and such that the decoder can decode at least a portion of the received media based on a reliably received frame of the type R after packet loss or error and thereafter is synchronized with the encoder; and

at least one bridge that is coupled to the transmitting endpoint and to the one or more receiving endpoint(s) so that media communication between the transmitting endpoint and the one or more receiving endpoint(s) occurs through the at least one bridge, the at least one bridge being configured to reliably receive the subset of the frames of the type R from the transmitting endpoint, and reliably deliver the subset of the frames of the type R to the one or more receiving endpoint(s),

wherein the at least one bridge retransmits a reference frame of the type R over a connection to a receiving endpoint or another bridge which is congested and is consistently dropping media packets, but reports positive receipt of the reference frame of the type R to the transmitting endpoint, whereby communications to other receiving endpoints and bridge are not affected by the connection which is congested.

13. The system of claim 12 wherein NACK messages are used to indicate non-receipt or error in receipt of packets, and wherein an intermediate bridge or receiving endpoint immediately sends a NACK message to an upstream bridge or to the transmitting endpoint upon detection of a packet loss.

14. The system of claim 12 wherein positive ACK messages are used to indicate receipt of packets, and wherein an intermediate bridge sends a single aggregated ACK message to an upstream bridge or to the transmitting endpoint after receiving positive ACK messages from all downstream receivers and bridges.

15. The system of claim 12 wherein positive ACK messages are used to indicate error-free receipt of packets, and wherein an intermediate bridge generates a positive ACK message upon error-free receipt of a packet from a sender without waiting for any further ACK messages from downstream receivers and/or bridges.

16. The system of claim 12 comprising at least one of a single-layer codec, a scalable codec, and a simulcast codec.

17. A system for media communications between a transmitting endpoint and one or more receiving endpoint(s) over a communications network, the network having one or more layered media transport link(s) including a more reliable media transport link to or from each endpoint, the system comprising:

an encoder which encodes transmitted media as frames in a threaded coding structure having a number of different frame types, the frames including a sequence of reference frames;

a decoder which decodes received media based on reference frames therein;

a particular select subset of the frames in the threaded coding structure that is designated as a type (“

R”

) for reliable transport to the one or more receiving endpoint(s) over the more reliable transport link,

wherein the subset of the frames of the type R is particularly selected to include at least the frames of the lowest temporal layer in the threaded coding structure and such that the decoder can decode at least a portion of the received media based on a reliably received frame of the type R after packet loss or error and thereafter is synchronized with the encoder; and

at least one bridge that is coupled to the transmitting endpoint and to the one or more receiving endpoint(s) so that media communication between the transmitting endpoint and the one or more receiving endpoint(s) occurs through the at least one bridge, the at least one bridge being configured to reliably receive the subset of the frames of the type R from the transmitting endpoint, and reliably deliver the subset of the frames of the type R to the one or more receiving endpoint(s),

wherein the at least one bridge and/or transmitting endpoint caches an intra frame of the lowest temporal layer frames and transmits the intra frame to a receiving endpoint or another bridge in response to packet loss or in response to a new receiving endpoint joining a communication session, and

wherein the transmitting encoder uses periodic intra macroblocks to facilitate drift elimination after the intra frame sent by the at least one bridge is used by the receiving endpoint.

18. The system of claim 17 wherein the encoder generates a switching slice picture (SI) that is cached by the transmitting endpoint and/or the at least one bridge, and wherein the cached SI picture is transmitted to a receiving endpoint and/or other bridges upon request and/or in response to the receiving endpoint newly joining a communication session.

19. The system of claim 17 wherein the at least one bridge decodes all lowest temporal layer frames and caches the result, and wherein the at least one bridge transmits an intra version of the most recent cached frame to receiving endpoints and/or other bridges that request it either due to packet loss or because they are newly joining a communication session.

20. The system of claim 17 wherein the encoder generates and transmits an intra frame that is not referenced by other frames, wherein the intra frame is cached on the at least one bridge, and wherein the cached intra frame is transmitted to a receiving endpoint and/or other bridge only upon explicit request in response to packet losses and/or in response to the receiving endpoint newly joining a communication session.

21. The system of claim 17 wherein the encoder generates and transmits a redundant intra frame or slice, wherein the redundant intra frame or slice is cached on the at least one bridge, and wherein the redundant intra frame or slice is transmitted to a receiving endpoint and/or other bridge only upon explicit request in response to packet losses and/or in response to the receiving endpoint newly joining a communication session.

22. A system for media communications between a transmittin endpoint and one or more receiving endpoint(s) over a communications network, the network having one or more layered media transport link(s) including a more reliable media transport link to or from each endpoint, the system comprising:

an encoder which encodes transmitted media as frames in a threaded coding structure having a number of different frame types, the frames including a sequence of reference frames;

a decoder which decodes received media based on reference frames therein; and

a particular select subset of the frames in the threaded coding structure that is designated as a type (“

R”

) for reliable transport to the one or more receiving endpoint(s) over the more reliable transport link,

wherein the subset of the frames of the type R is particularly selected to include at least the frames of the lowest temporal layer in the threaded coding structure and such that the decoder can decode at least a portion of the received media based on a reliably received frame of the type R after packet loss or error and thereafter is synchronized with the encoder, and

wherein reliable transport of the reference frames of the type R comprises retransmissions over the more reliable transport link, and wherein the decoder resynchronizes with the encoder after an error, or synchronizes with the encoder upon entering a communication session as a new participant, by promptly decoding all retransmitted lowest temporal layer frames even if they are received too late to be currently displayed.

23. The system of claim 22 further comprising at least one bridge that is coupled to the transmitting endpoint and to the one or more receiving endpoint(s) so that media communication between the transmitting endpoint and the one or more receiving endpoint(s) occurs through the at least one bridge wherein picture encoding utilizes structured intra macroblock coding, and where the at least one bridge caches a cycle of R frames and transmits it to endpoints that perform layer switching or are entering the session as new participants, so that they can be decoded faster than real-time to enable fast switching.

24. The system of claim 23 wherein the at least one bridge removes inter-coded macroblocks or slices from packets prior to transmitting them to the endpoint that is performing layer switching or is entering the session as a new participant.

25. A method for media communications between a transmitting endpoint and one or more receiving endpoint(s) over a communications network, the network having one or more layered media transport link(s) including a more reliable media transport link to or from each endpoint, the endpoints including:

an encoder which encodes transmitted media as frames in a threaded coding structure having a number of different frame types, the frames including a sequence of reference frames; and

a decoder which decodes received media based on reference frames therein;

the method comprising;

designating a particular select subset of the frames in the threaded coding structure as a type (“

R”

) for reliable transport to the decoder over the more reliable transport link,

wherein the subset of the frames of the type R is particularly selected to include at least the frames of the lowest temporal layer in the threaded coding structure and such that the decoder can decode at least a portion of the received media based on a reliably received frame of the type R after packet loss or error and thereafter is synchronized with the encoder;

sending the frames of the type R to the decoder, wherein the particular select subset of the frames that is designated as a type R for reliable transport to the decoder over the more reliable transport link comprises intra and inter macroblocks; and

recalculating the allocation of intra macroblocks in coding future pictures in response to packet loss or a new endpoint joining a current communication session.

26. The method of claim 25, further comprising processing the type R frames at ends of the more reliable media transport link to or from the encoder, the decoder, and/or an intermediate network server.

27. The method of claim 25, further comprising using at least one of a single-layer, a scalable codec, and a simulcast codec.

28. The method of claim 25, further comprising:

using a codec conforming to H.264 AVC;

including a frame that is marked as a long term reference picture in the particular select subset of the frames of the type R for reliable transport to the decoder over the more reliable transport link; and

decoding at least a portion of the received media based on a reliably received reference frame of the type R by using memory management control operation (MMCO) commands to control the decoder.

29. The method of claim 25 wherein reliable transport of the reference frames of the type R comprises retransmitting the frames over the more reliable transport link based on a protocol of Acknowledgment (ACK) and/or No Acknowledgment (NACK) messages.

30. The method of claim 29 wherein the frames are retransmitted over the more reliable transport link if an ACK message is not received within a set time period.

31. The method of claim 29 wherein the frames are retransmitted over the more reliable transport link upon receipt of a NACK message.

32. The method of claim 25, further comprising encoding frames of the type R for transmission according to a preset schedule.

33. The method of claim 25, further comprising designating a specific transmitted frame of the type R as a reference picture for future use after confirming receipt of the specific frame by all intended recipients.

34. The method of claim 33 further comprising abandoning a specific frame of the type R as a candidate reference picture for future use upon unsuccessful reliable transmission of the specific frame to any intended recipient.

35. The method of claim 25, wherein at least one bridge is coupled to the transmitting endpoint and to the one receiving endpoint(s) so that media communication occurs through the at least one bridge, the method further comprising reliably receiving the subset of the frames of the type R from the transmitting endpoint.

36. The method of claim 35 wherein at least one bridge is further configured to transmit a NACK message to the transmitting endpoint when it detects a lost packet in order to instruct the transmitting endpoint to retransmit it, and wherein the receiving endpoint(s) are further configured to transmit a NACK message to the transmitting endpoint when they detect a lost packet in order to instruct the transmitting endpoint to retransmit it.

49. A non-transitory computer-readable storage medium comprising an executable set of instructions to direct a processor to perform the methods in one of claim 26-31, 37, 41-46, 47, 48, 35, or 36.

37. A method for media communications between a transmitting endpoint and one or more receiving endpoint(s) over a communications network, the network having one or more layered media transport link(s) including a more reliable media transport link to or from each endpoint, the endpoints including:

an encoder which encodes transmitted media as frames in a threaded coding structure having a number of different frame types, the frames including a sequence of reference frames; and

a decoder which decodes received media based on reference frames therein;

the method comprising;

designating a particular select subset of the frames in the threaded coding structure as a type (“

R”

) for reliable transport to the decoder over the more reliable transport link,

wherein the subset of the frames of the type R is particularly selected to include at least the frames of the lowest temporal layer in the threaded coding structure and such that the decoder can decode at least a portion of the received media based on a reliably received frame of the type R after packet loss or error and thereafter is synchronized with the encoder;

sending the frames of the type R to the decoder, wherein at least one bridge is coupled to the transmitting endpoint and to the one receiving endpoint(s) so that media communication occurs through the at least one bridge;

reliably receiving the subset of the frames of the type R from the transmitting endpoint and reliably delivering the subset of the frames of the type R to the one or more receiving endpoint(s); and

retransmitting a reference frame of the type R over a connection to a receiving endpoint or another bridge which is congested and is consistently dropping media packets, but reports positive receipt of the reference frame of the type R to the transmitting endpoint, whereby communications to other receiving endpoints and bridge are not affected by the connection which is congested.

38. The method of claim 37 further comprising:

using NACK messages to indicate non-receipt or error in receipt of packets; and

upon detection of a packet loss, immediately sending a NACK message from an intermediate bridge or receiving endpoint to an upstream bridge or to the transmitting endpoint.

39. The method of claim 37 further comprising:

using ACK messages to indicate receipt of packets; and

after receiving positive ACK messages from all downstream receivers and bridges at an intermediate bridge, sending a single aggregated ACK message from the intermediate bride to an upstream bridge or to the transmitting endpoint.

40. The method of claim 37 further comprising:

using ACK messages to indicate receipt of packets; and

generating a positive ACK message from an intermediate bridge upon error-free receipt of a packet from a sender at the intermediate bridge without waiting for any further ACK messages from downstream receivers and/or bridges.

41. A method for media communications between a transmitting endpoint and one or more receiving endpoint(s) over a communications network, the network having one or more layered media transport link(s) including a more reliable media transport link to or from each endpoint, the endpoints including:

an encoder which encodes transmitted media as frames in a threaded coding structure having a number of different frame types, the frames including a sequence of reference frames; and

a decoder which decodes received media based on reference frames therein;

the method comprising;

designating a particular select subset of the frames in the threaded coding structure as a type (“

R”

) for reliable transport to the decoder over the more reliable transport link,

wherein the subset of the frames of the type R is particularly selected to include at least the frames of the lowest temporal layer in the threaded coding structure and such that the decoder can decode at least a portion of the received media based on a reliably received frame of the type R after packet loss or error and thereafter is synchronized with the encoder;

sending the frames of the type R to the decoder, wherein at least one bridge is coupled to the transmitting endpoint and to the one receiving endpoint(s) so that media communication occurs through the at least one bridge;

reliably receiving the subset of the frames of the type R from the transmitting endpoint and reliably delivering the subset of the frames of the type R to the one or more receiving endpoint(s);

caching an intra frame of the lowest temporal layer frames at the at least one bridge and/or transmitting endpoint;

in response to packet loss or in response to new receiving endpoint joining a communication session, transmitting the intra frame to a receiving endpoint or another bridge; and

using at the transmitting encoder periodic intra macroblocks to facilitate drift elimination after intra picture sent by the at least one bridge is used by the receiving endpoint.

42. The method of claim 41 further comprising:

generating a switching slice picture (SI) that is cached by the transmitting endpoint and/or the at least one bridge; and

upon request and/or in response to the receiving endpoint newly joining a communication session, transmitting the cached SI picture to a receiving endpoint and/or other bridges.

43. The method of claim 41 further comprising:

decoding all lowest temporal layer pictures at the at least one bridge and caching the result; and

transmitting an intra version of the most recent cached picture to receiving endpoints and/or other bridges that request it either due to packet loss or because they are newly joining a communication session.

44. The method of claim 41 further comprising:

generating at the encoder an intra picture that is not referenced by other pictures;

caching the intra picture on the at least one bridge, and

transmitting the cached intra picture to a receiving endpoint and/or other bridge only upon explicit request in response to packet losses and/or in response to the receiving endpoint newly joining a communication session.

45. The method of claim 41 further comprising:

generating at the encoder a redundant intra picture or slice;

caching the redundant intra picture or slice on the at least one bridge; and

transmitting the redundant intra picture or slice to a receiving endpoint and/or other bridge only upon explicit request in response to packet losses and/or in response to the receiving endpoint newly joining a communication session.

46. A method for media communications between a transmitting endpoint and one or more receiving endpoint(s) over a communications network, the network having one or more layered media transport link(s) including a more reliable media transport link to or from each endpoint, the endpoints including:

an encoder which encodes transmitted media as frames in a threaded coding structure having a number of different frame types, the frames including a sequence of reference frames; and

a decoder which decodes received media based on reference frames therein;

the method comprising;

designating a particular select subset of the frames in the threaded coding structure as a type (“

R”

) for reliable transport to the decoder over the more reliable transport link,

wherein the subset of the frames of the type R is particularly selected to include at least the frames of the lowest temporal layer in the threaded coding structure and such that the decoder can decode at least a portion of the received media based on a reliably received frame of the type R after packet loss or error and thereafter is synchronized with the encoder;

sending the frames of the type R to the decoder, wherein reliable transport of the reference frames of the type R comprises retransmissions over the more reliable transport link; and

after an error, or upon entering a communication session as a new participant, promptly decoding all retransmitted lowest temporal layer pictures at the decoder even if they are received too late to be currently displayed, so that the decoder is synchronized with the encoder.

47. The method of claim 46, wherein at least one bridge mediates communications between the transmitting endpoint and the one or more receiving endpoint(s), and wherein picture encoding utilizes structured intra macroblock coding, the method further comprising:

caching a cycle of LR pictures at the at least one bridge; and

transmitting the cycle of LR pictures to endpoints that are layer switching or are entering the session as new participants, so that they can be decoded faster than real-time to enable fast switching.

48. The method of claim 47 further comprising removing inter-coded macroblocks or slices from packets at the at least one bridge prior to transmitting them to the endpoint that is joining the session or performing a layer transition.