SYSTEMS AND METHODS FOR ERROR RESILIENCE AND RANDOM ACCESS IN VIDEO COMMUNICATION SYSTEMS

• US 20070206673A1
• Filed: 12/08/2006
• Published: 09/06/2007
• Est. Priority Date: 12/08/2005
• Status: Active Grant

• Current Assignee: Vidyo Incorporated (Enghouse Systems Limited)
• Original Assignee: Vidyo Incorporated (Enghouse Systems Limited)
• Inventors: Eleftheriadis, Alexandros, Sasson, Roi, Shapiro, Ofer, Lennox, Jonathan, Civanlar, Reha, Saxena, Manoj, CIPOLLI, STEPHEN
• Granted Patent: US 9,077,964 B2
• Time in Patent Office: Days
• US Class Current: 375/240.100
• CPC Class Codes: H04L 1/1607 Details of the supervisory signal
• CPC Class Codes: H04L 1/1812 Hybrid protocols; Hybrid automatic repeat request [HARQ]
• CPC Class Codes: H04L 1/1848 Time-out mechanisms
• CPC Class Codes: H04L 65/752 adapting media to network capabilities
• CPC Class Codes: H04L 65/756 adapting media to device capabilities
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/166 concerning the amount of transmission errors, e.g. bit error rate [BER]
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/31 in the temporal domain
• CPC Class Codes: H04N 19/37 with arrangements for assigning different transmission priorities to video input data or to video coded data
• CPC Class Codes: H04N 19/463 by compressing encoding parameters before transmission
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 19/573 Motion compensation with multiple frame prediction using two or more reference frames in a given prediction direction
• CPC Class Codes: H04N 19/577 Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
• CPC Class Codes: H04N 19/58 Motion compensation with long-term prediction, i.e. the reference frame for a current frame not being the temporally closest one H04N19/23 takes precedence
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/65 using error resilience
• CPC Class Codes: H04N 19/66 : involving data partitioning, i.e. separation of data into packets or partitions according to importance
• CPC Class Codes: H04N 19/67 : involving unequal error protection [UEP], i.e. providing protection according to the importance of the data
• CPC Class Codes: H04N 19/70 : characterised by syntax aspects related to video coding, e.g. related to compression standards
• CPC Class Codes: H04N 19/89 : involving methods or arrangements for detection of transmission errors at the decoder
• CPC Class Codes: H04N 21/2662 : Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 21/4621 : Controlling the complexity of the content stream or additional data, e.g. lowering the resolution or bit-rate of the video stream for a mobile client with a small screen arrangements for using the results of monitoring on user's side in broadcast systems H04H60/65; flow control in packet networks H04L47/10
• CPC Class Codes: H04N 21/631 : Multimode Transmission, e.g. transmitting basic layers and enhancement layers of the content over different transmission paths or transmitting with different error corrections, different keys or with different transmission protocols
• CPC Class Codes: H04N 21/6375 : for requesting retransmission , e.g. of data packets lost or corrupted during transmission from serverARQ protocols H04L1/18; implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP] H04L69/16
• CPC Class Codes: H04N 21/6583 : Acknowledgement

Abstract

Systems and methods for error resilient transmission and for random access in video communication systems are provided. The video communication systems are based on single-layer, scalable video, or simulcast video coding with temporal scalability, which may be used in video communication systems. A set of video frames or pictures in a video signal transmission is designated for reliable or guaranteed delivery to receivers using secure or high reliability links, or by retransmission techniques. The reliably-delivered video frames are used as reference pictures for resynchronization of receivers with the transmitted video signal after error incidence and for random access.

Claims

1. A system for media communications between a transmitting endpoint and one or more receiving endpoint(s) over a communications network, the network having one or more layered media transport link(s) including a more reliable media transport link to or from each endpoint, the system comprising:

an encoder which encodes transmitted media as frames in a threaded coding structure having a number of different frame types, the frames including a sequence of reference frames;

a decoder which decodes received media based on reference frames therein; and

a particular select subset of the frames in the threaded coding structure that is designated as a type (“

R”

) for reliable transport to the decoder over the more reliable transport link,

wherein the subset of the frames of the type R is particularly selected to include at least the frames of the lowest temporal layer in the threaded coding structure and such that the decoder can decode at least a portion of the received media based on a reliably received frame of the type R after packet loss or error and thereafter is synchronized with the encoder.

2. The system of claim 1 further comprising processing modules for processing type R frames, wherein the processing modules are disposed at the encoder, and/or the decoder, and/or an intermediate network server at ends of the more reliable media transport link.

3. The system of claim 1 comprising at least one of a single-layer, a scalable codec, and a simulcast codec.

4. The system of claim 1 comprising a codec conforming to H.264 AVC, wherein the particular select subset of the frames of the type R for reliable transport to the decoder over the more reliable transport link includes a frame that is marked as a long term reference picture, and wherein the decoding of at least a portion of the received media based on a reliably received reference frame of the type R by the decoder is controlled by MMCO commands.

5. The system of claim 1 wherein reliable transport of the reference frames of the type R comprises retransmissions over the more reliable transport link based on a protocol of Acknowledgment (ACK) and/or No Acknowledgment (NACK) messages.

6. The system of claim 5 wherein the particular select subset of the frames that is designated as a type R for reliable transport to the decoder over the more reliable transport link comprises intra and inter macroblocks, and wherein the encoder further comprises a reliability and random access (RRC) module that recalculates the allocation of intra macroblocks in coding future pictures in response to packet loss in or a new endpoint joining a current communication session.

7. The system of claim 5 wherein the encoder and any intermediate server in the network comprise a timer with a preset time period, and wherein reliable transport of frames of the type R comprises retransmissions over the more reliable transport link based on non-receipt of an ACK message within the set time period.

8. The system of claim 5 wherein the wherein reliable transport of a reference frames of the type R comprises retransmissions over the more reliable transport link based on receipt of a NACK message.

17. The system of claim 1 further comprising at least one SVCS that mediates communications between the transmitting endpoint and the one or more receiving endpoint(s), the at least one SVCS having separately managed connections to the transmitting endpoint, the one or more receiving endpoint(s), and other SVCS in the communications network.

18. The system of claim 17 wherein the at least one SVCS retransmits a reference frame of the type R over a separately managed connection to a receiving endpoint or another SVCS which is congested and is consistently dropping media packets, but reports positive receipt of the reference frame of the type R to the transmitting endpoint, whereby communications to other receiving endpoints and SVCS are not affected by the separately managed connection which is congested.

19. The system of claim 17 wherein the at least one SVCS and/or transmitting endpoint caches an intra frame of the lowest temporal layer frames and transmits the intra frame to a receiving endpoint or another SVCS in response to packet loss or in response to new receiving endpoint joining a communication session.

20. The system of claim 19 wherein the transmitting encoder uses periodic intra macroblocks to facilitate drift elimination after intra picture sent by the at least one SVCS is used by the receiving endpoint.

21. The system of claim 19 wherein the encoder generates a switching slice picture (SI) that is cached by the transmitting endpoint and/or the at least one SVCS, and wherein the cached SI picture is transmitted to a receiving endpoint and/or other SVCSs upon request and/or in response to the receiving endpoint newly joining a communication session.

22. The system of claim 19 wherein the at least one SVCS decodes all lowest temporal layer pictures and caches the result, and wherein the at least one SVCS transmits an intra version of the most recent cached picture to receiving endpoints and/or other SVCSs that request it either due to packet loss or because they are newly joining a communication session.

23. The system of claim 19 wherein the encoder generates and transmits an intra picture that is not referenced by other pictures, wherein the intra picture is cached on the at least one SVCS, and wherein the cached intra picture is transmitted to a receiving endpoint and/or other SVCS only upon explicit request in response to packet losses and/or in response to the receiving endpoint newly joining a communication session.

24. The system of claim 19 wherein the encoder generates and transmits an redundant intra picture or slice, wherein the redundant intra picture or slice is cached on the at least one SVCS, and wherein the redundant intra picture or slice is transmitted to a receiving endpoint and/or other SVCS only upon explicit request in response to packet losses and/or in response to the receiving endpoint newly joining a communication session.

25. The system of claim 17 wherein NACK messages are used to indicate non-receipt or error in receipt of packets, and wherein an intermediate SVCS or receiving endpoint immediately sends a NACK message to an upstream SVCS or to the transmitting endpoint upon detection of a packet loss.

26. The system of claim 17 wherein positive ACK messages are used to indicate receipt of packets, and wherein an intermediate SVCS sends a single aggregated ACK message to an upstream SVCS or to the transmitting endpoint after receiving positive ACK messages from all downstream receivers and SVCSs.

27. The system of claim 17 wherein positive ACK messages are used to indicate error-free receipt of packets, and wherein an intermediate SVCS generates a positive ACK message upon error-free receipt of a packet from a sender without waiting for any further ACK messages from downstream receivers and/or SVCSs.

28. The system of claim 1 wherein the encoder, which encodes transmitted media as frames in a threaded coding structure, encodes frames of the type R for transmission according to a preset schedule.

29. The system of claim 1 wherein the encoder, which encodes transmitted media as frames in a threaded coding structure, designates a specific transmitted frame of the type R as a reference picture for future use after confirming receipt of the specific frame by all intended recipients.

30. The system of claim 29 wherein the encoder, which encodes transmitted media as frames in a threaded coding structure, abandons a specific frame of the type R as a candidate reference picture for future use upon unsuccessful reliable transmission of the specific frame to any intended recipient.

31. The system of claim 1 wherein reliable transport of the reference frames of the type R comprises retransmissions over the more reliable transport link, and wherein decoder resynchronizes with the encoder after an error, or synchronizes with the encoder upon entering a communication session as a new participant, by promptly decoding all retransmitted lowest temporal layer pictures even if they are received too late to be currently displayed.

32. The system of claim 1 wherein the threaded coding structure comprises at least a high resolution layer and a low resolution layer, wherein the encoder uses single-loop spatial scalability coding with periodic intra macroblocks at both layers, wherein:

the decoder performs layer switching from low to high resolution by decoding low resolution layer data and displaying it upsampled, while at the same time receiving and decoding high resolution layer data but not displaying it until drift is eliminated, after which time the decoder can switch to displaying decoded high resolution pictures, and

the decoder performs layer switching from high to low resolution by using a downsampled version of a most recent high resolution picture as a reference picture for its low resolution decoding loop, and proceeding with decoding and display of the decoded low resolution pictures in which the intra macroblocks will gradually eliminate the drift.

33. The system of claim 32 wherein the encoder uses H.264 AVC or SVC for encoding frames, and wherein a Recovery Point SEI message is used to indicate the frame number when recovery is complete after error incidence and if the match with the encoder is exact or not exact.

34. The system of claim 1 further comprising at least one SVCS that mediates communications between the transmitting endpoint and the one or more receiving endpoint(s), wherein picture encoding utilizes structured intra macroblock coding, and where the at least one SVCS caches a cycle of LR pictures and transmits it to endpoints that perform layer switching or are entering the session as new participants, so that they can be decoded faster than real-time to enable fast switching.

35. The system of claim 34 wherein the at least one SVCS removes inter-coded macroblocks or slices from packets prior to transmitting them to the endpoint that is joining the session or performing a layer transition.

36. The system of claim 1 wherein the encoder uses spatial scalability with at least two spatial layers, and wherein a receiving endpoint decodes an approximation to an enhancement layer picture after loss of the enhancement layer picture or upon switching to the enhanced layer resolution by using an upsampled lower layer picture as reference, but applying the available motion vectors, modes, and prediction error coded in the enhancement layer.

37. The system of claim 36 wherein the receiving endpoint operates a single decoding loop and the upsampled lower layer picture consists of only the intra macroblocks available in the lower layers.

38. The system of claim 36 wherein the receiving endpoint operates additional decoding loops, and wherein:

the single decoding loop decodes the target resolution pictures; and

the additional decoding loops decode only the lowest temporal layer pictures of the other resolutions,

so that transition from one resolution to another can be performed at the time instances of the lowest temporal level pictures without error, and at other time instances with a small error.

39. The system of claim 38 wherein the SVCS or transmitting endpoint transmits the pictures of the lowest temporal layer of the scalability layers other than the one to be displayed such that the bit rate variation of the transmitted data is minimized.

9. A system for media communications between a transmitting endpoint and one or more receiving endpoint(s) or server(s) over a communication network, the system comprising:

an encoder which encodes transmitted media as frames in a threaded coding structure having a number of different layers including a lowest temporal layer, wherein transmitted pictures comprise data elements that indicate;

for the lowest temporal level pictures, a sequence number identifying said pictures, and

for other temporal level pictures, a reference to the sequence number of the most recent, in decoding order, lowest temporal level picture, such that a receiving endpoint or server can detect if a lowest temporal level picture has been lost by examining if the picture corresponding to the referenced picture frame index has been received at the receiving endpoint or server.

10. The system of claim 9 wherein the data elements additionally indicate a series number associated with each spatial or quality layer, wherein the receiving endpoint or server detects if a lowest temporal level frame of a particular spatial or quality layer is lost by determining if the frame corresponding to the referenced series number and sequence number has been received at the receiving endpoint or server.

11. The system of claim 9 wherein the communication network uses the Internet Protocol, media transport is performed using the real-time protocol (RTP), the data elements are contained as fields in RTP header extensions, and include a flag to indicate presence of a lowest temporal layer frame or fragment thereof in the packet.

12. The system of claim 11 wherein a receiving endpoint or server in the network sends a negative acknowledgment message formatted as an RTCP feedback message in response to the receiving endpoint'"'"'s or server'"'"'s detection of a lost R frame or portion of a frame, wherein the feedback message includes the sequence number of the lost frame, the series number indicating the lost frame layer and a bit mask indicating which among the pictures that follow the one indicated by the said sequence number are also lost.

13. The system of claim 12 wherein the transmitting endpoint or server in the network upon receiving the negative acknowledgment message check whether the lost frame has been superseded by a recent frame, and wherein the transmitting endpoint or server accordingly retransmit the lost frame if not superseded, or transmit the recent frame if the lost frame is superseded with an indication of a range of frames including the lost frame that have been superseded.

14. The system of claim 9, wherein the encoder conforms to H.264 SVC and the data elements are carried in NAL unit header extension for SVC elements.

15. The system of claim 14 wherein the data elements comprise an additional byte in the NAL header extension for SVC and wherein a flag in the NAL header extension for SVC signals the presence of the additional byte.

16. The system of claim 14 wherein the data elements comprise bits related to FGS coding in the NAL header extension for SVC that are not used by pictures of the lowest quality layer.

40. A system for decoding compressed digital video that is coded using a technique that provides two or more temporal layers, wherein compressed video pictures are structured into one or more packets, the system comprising:

a packet header containing data elements that indicate;

for the lowest temporal level pictures, a sequence number identifying the pictures,

for other temporal level pictures, a reference to the sequence number of the most recent, in decoding order, lowest temporal level picture,

so that a receiver can detect loss of a lowest temporal level picture by examining if the picture corresponding to the referenced picture frame index is available at the receiver.

41. The system of claim 40, wherein the data elements comprises a set of extension bits and a flag, which when set, indicates the presence of the set of extension bits.

42. The system of claim 40, wherein the data elements comprise bits that are not used in the lowest quality layer pictures.

43. A system for decoding compressed digital video that is coded using a technique that provides two or more temporal layers, wherein compressed video pictures are structured into one or more packets, and received over an IP-based network using RTP, the system comprising:

a RTP header extension that includes;

a series number associated with each layer,

a sequence number that is associated with each lowest temporal layer picture, and

a flag that is used to indicate if a packet contains a picture or picture fragment of the lowest layer temporal picture,

wherein the sequence number is referenced by all other pictures that use said lowest temporal layer picture as reference, such that loss of a lowest temporal level picture can be immediately detected by examining if the picture corresponding to the referenced series number and sequence number is available at the system.

44. The system of claim 43 which sends a negative acknowledgment message formatted as a RTCP feedback message upon detecting a lost lowest temporal layer picture, with the feed back message indicating:

the sequence number of the lost picture, the series number that the lost picture belongs to, and a bitmask indicating which among the pictures that follow the one indicated by the said sequence number is also lost, whereby a transmitting system can take corrective action.

45. A system for decoding compressed digital video that is coded using a technique that provides two or more temporal layers, wherein compressed video pictures are structured into one or more packets, and packets of at least the lowest temporal level pictures may be received in a compressed digital video stream from a sender faster than real-time and after their intended presentation time, in case of errors or upon initiating decoding, the system comprising:

a decoder, which in case of errors or upon initiating decoding, decodes the received pictures faster than real-time and, if decoded after their intended presentation time, without displaying them,

whereby the decoder can synchronize with the received compressed digital video stream and then after such synchronization is achieved perform normal decoding and presentation.

46. A method for media communications between a transmitting endpoint and one or more receiving endpoint(s) over a communications network, the network having one or more layered media transport link(s) including a more reliable media transport link to or from each endpoint, the endpoints including:

an encoder which encodes transmitted media as frames in a threaded coding structure having a number of different frame types, the frames including a sequence of reference frames; and

a decoder which decodes received media based on reference frames therein;

the method comprising;

designating a particular select subset of the frames in the threaded coding structure as a type (“

R”

) for reliable transport to the decoder over the more reliable transport link,

wherein the subset of the frames of the type R is particularly selected to include at least the frames of the lowest temporal layer in the threaded coding structure and such that the decoder can decode at least a portion of the received media based on a reliably received frame of the type R after packet loss or error and thereafter is synchronized with the encoder; and

,

sending the frames of the type R to the decoder.

47. The method of claim 46, further comprising processing the type R frames at ends of the more reliable media transport link to or from the encoder, the decoder, and/or an intermediate network server.

48. The method of claim 46 comprising using at least one of a single-layer, a scalable codec, and a simulcast codec.

49. The method of claim 46 comprising:

using a codec conforming to H.264 AVC;

including a frame that is marked as a long term reference picture in the particular select subset of the frames of the type R for reliable transport to the decoder over the more reliable transport link; and

decoding at least a portion of the received media based on a reliably received reference frame of the type R by using MMCO commands to control the decoder.

50. The method of claim 46 wherein reliable transport of the reference frames of the type R comprises retransmitting the frames over the more reliable transport link based on a protocol of Acknowledgment (ACK) and/or No Acknowledgment (NACK) messages.

51. The method of claim 46, wherein the particular select subset of the frames that is designated as a type R for reliable transport to the decoder over the more reliable transport link comprises intra and inter macroblocks, the method further comprising:

recalculating the allocation of intra macroblocks in coding future pictures in response to packet loss in or a new endpoint joining a current communication session.

52. The method of claim 50 wherein the frames are retransmitted over the more reliable transport link if an ACK message is not received within a set time period.

53. The method of claim 50 wherein the frames are retransmitted over the more reliable transport link upon receipt of a NACK message.

62. The method of claim 46, wherein at least one SVCS mediates communications between the transmitting endpoint and the one or more receiving endpoint(s), the method further comprising providing the at least one SVCS with separately managed connections to the transmitting endpoint, the one or more receiving endpoint(s), and other SVCS in the communications network.

63. The method of claim 62, further comprising retransmitting a reference frame of the type R over a separately managed connection to a receiving endpoint or another SVCS which is congested and is consistently dropping media packets, but reports positive receipt of the reference frame of the type R to the transmitting endpoint, whereby communications to other receiving endpoints and SVCS are not affected by the separately managed connection which is congested.

64. The method of claim 62 further comprising:

caching an intra frame of the lowest temporal layer frames at the at least one SVCS and/or transmitting endpoint; and

in response to packet loss or in response to new receiving endpoint joining a communication session, transmitting the intra frame to a receiving endpoint or another SVCS.

65. The method of claim 64 further comprising:

using at the transmitting encoder periodic intra macroblocks to facilitate drift elimination after intra picture sent by the at least one SVCS is used by the receiving endpoint.

66. The method of claim 64 further comprising:

generating a switching slice picture (SI) that is cached by the transmitting endpoint and/or the at least one SVCS; and

upon request and/or in response to the receiving endpoint newly joining a communication session, transmitting the cached SI picture to a receiving endpoint and/or other SVCSs.

67. The method of claim 64 further comprising:

decoding all lowest temporal layer pictures at the at least one SVCS and caching the result; and

transmitting an intra version of the most recent cached picture to receiving endpoints and/or other SVCSs that request it either due to packet loss or because they are newly joining a communication session.

68. The method of claim 64 further comprising:

generating at the encoder an intra picture that is not referenced by other pictures;

caching the intra picture on the at least one SVCS, and

transmitting the cached intra picture to a receiving endpoint and/or other SVCS only upon explicit request in response to packet losses and/or in response to the receiving endpoint newly joining a communication session.

69. The method of claim 64 further comprising:

generating at the encoder a redundant intra picture or slice;

caching the redundant intra picture or slice on the at least one SVCS; and

transmitting the redundant intra picture or slice to a receiving endpoint and/or other SVCS only upon explicit request in response to packet losses and/or in response to the receiving endpoint newly joining a communication session.

70. The method of claim 62 further comprising:

using NACK messages to indicate non-receipt or error in receipt of packets; and

upon detection of a packet loss, immediately sending a NACK message from an intermediate SVCS or receiving endpoint to an upstream SVCS or to the transmitting endpoint.

71. The method of claim 62 further comprising:

using ACK messages to indicate receipt of packets; and

after receiving positive ACK messages from all downstream receivers and SVCSs at an intermediate SVCS, sending a single aggregated ACK message from the intermediate SVCS to an upstream SVCS or to the transmitting endpoint.

72. The method of claim 62 further comprising:

using ACK messages to indicate receipt of packets; and

generating a positive ACK message from an intermediate SVCS upon error-free receipt of a packet from a sender at the intermediate SVCS without waiting for any further ACK messages from downstream receivers and/or SVCSs.

73. The method of claim 46, further comprising encoding frames of the type R for transmission according to a preset schedule.

74. The method of claim 46, further comprising designating a specific transmitted frame of the type R as a reference picture for future use after confirming receipt of the specific frame by all intended recipients.

75. The method of claim 74 further comprising abandoning a specific frame of the type R as a candidate reference picture for future use upon unsuccessful reliable transmission of the specific frame to any intended recipient.

76. The method of claim 46, wherein reliable transport of the reference frames of the type R comprises retransmissions over the more reliable transport link, the method further comprising:

after an error, or upon entering a communication session as a new participant, promptly decoding all retransmitted lowest temporal layer pictures at the decoder even if they are received too late to be currently displayed,

so that the decoder is synchronized with the encoder.

77. The method of claim 46, wherein the threaded coding structure comprises at least a high resolution layer and a low resolution layer, wherein the encoder uses single-loop spatial scalability coding with periodic intra macroblocks at both layers, the method further comprising:

performing layer switching from low to high resolution by decoding low resolution layer data and displaying it upsampled, while at the same time receiving and decoding high resolution layer data but not displaying it until drift is eliminated, after which time the decoder can switch to displaying decoded high resolution pictures, and

performing layer switching from high to low resolution by using a downsampled version of a most recent high resolution picture as a reference picture for its low resolution decoding loop, and proceeding with decoding and display of the decoded low resolution pictures in which the intra macroblocks will gradually eliminate the drift.

78. The method of claim 77 wherein the encoding complies to H.264 AVC or SVC, the method further comprising using a Recovery Point SEI message to indicate the frame number when recovery is complete after error incidence and if the match with the encoder is exact or not exact.

79. The method of claim 46, wherein at least one SVCS mediates communications between the transmitting endpoint and the one or more receiving endpoint(s), and wherein picture encoding utilizes structured intra macroblock coding, the method further comprising:

caching a cycle of LR pictures at the at least one SVCS; and

transmitting the cycle of LR pictures to endpoints that are layer switching or are entering the session as new participants, so that they can be decoded faster than real-time to enable fast switching.

80. The method of claim 79 further comprising removing inter-coded macroblocks or slices from packets at the at least one SVCS prior to transmitting them to the endpoint that is joining the session or performing a layer transition.

81. The method of claim 46, wherein encoding uses spatial scalability with at least two spatial layers, the method further comprising:

after loss of an enhancement layer picture or upon switching to the enhanced layer resolution at a receiving endpoint, approximating the enhancement layer picture by using an upsampled lower layer picture as reference, but applying the available motion vectors, modes, and prediction error coded in the enhancement layer.

82. The method of claim 81 wherein the receiving endpoint operates a single decoding loop, the method further comprising using only the intra macroblocks available in the lower layers for the upsampled lower layer picture.

83. The method of claim 81 wherein the receiving endpoint operates additional decoding loops, the method further comprising:

using the single decoding loop to decode the target resolution pictures; and

using the additional decoding loops to decode only the lowest temporal layer pictures of the other resolutions,

so that transition from one resolution to another can be performed at the time instances of the lowest temporal level pictures without error, and at other time instances with a small error.

84. The method of claim 83 further comprising:

from the SVCS or transmitting endpoint, transmitting the pictures of the lowest temporal layer of the scalability layers other than the one to be displayed, so that the bit rate variation of the transmitted data is minimized.

91. Computer readable media comprising a set of instructions to perform the steps recited in claim 46.

54. A method for media communications between a transmitting endpoint and one or more receiving endpoint(s) or server(s) over a communication network, wherein transmitted media is encoded as frames in a threaded coding structure having a number of different layers including a lowest temporal layer, the method comprising providing data elements that indicate:

for the lowest temporal level pictures, a sequence number identifying said pictures, and

for other temporal level pictures a reference to the sequence number of the most recent, in decoding order, lowest temporal level picture, so that a receiving endpoint or server can detect if a lowest temporal level picture has been lost by examining if the picture corresponding to the referenced picture frame index is available at the receiving endpoint.

55. The method of claim 54 wherein the data elements additionally indicate a series number associated with each spatial or quality layer, wherein the receiving endpoint or server detects if a lowest temporal level frame of a particular spatial or quality layer is lost by determining if the frame corresponding to the referenced series number and sequence number has been received at the receiving endpoint or server.

56. The method of claim 54 wherein the communication network uses the Internet

Protocol, media transport is performed using the real-time protocol (RTP), the data elements are contained as fields in RTP header extensions, and include a flag to indicate presence of a lowest temporal layer frame or fragment thereof in the packet.

57. The method of claim 56 wherein a receiving endpoint or server in the network sends a negative acknowledgment message formatted as an RTCP feedback message in response to the receiving endpoint or server'"'"'s detection of a lost R frame or portion of a frame, the method further comprising:

including in the feedback message the sequence number of the lost frame, the series number indicating the lost frame layer and a bit mask indicating which among the pictures that follow the one indicated by the said sequence number are also lost.

58. The method of claim 57, further comprising:

upon receiving the negative acknowledgment message, checking at the transmitting endpoint or server in the network whether the lost frame has been superseded by a recent frame; and

accordingly, retransmitting the lost frame if not superseded, or retransmitting the recent frame if the lost frame is superseded with an indication of a range of frames including the lost frame that have been superseded.

59. The method of claim 54 wherein the encoding conforms to H.264 SVC, and the data elements are carried in NAL unit header extension for SVC elements.

60. The method of claim 59 wherein the data elements comprise an additional byte in the NAL header extension for SVC and wherein a flag in the NAL header extension for SVC which signals the presence of the additional byte.

61. The method of claim 59 the data elements comprise bits related to FGS coding in the NAL header extension for SVC that are not used by pictures of the lowest quality layer.

85. A method for decoding compressed digital video that is coded using a technique that provides two or more temporal layers, wherein compressed video pictures are structured into one or more packets, the method comprising:

providing data elements in a transmitted packet header to indicate;

for the lowest temporal level pictures, a sequence number identifying the pictures,

for other temporal level pictures, a reference to the sequence number of the most recent, in decoding order, lowest temporal level picture,

so that a receiver can detect loss of a lowest temporal level picture by examining if the picture corresponding to the referenced picture frame index is available at the receiver.

86. The method of claim 85, wherein the data elements comprise a set of extension bits and a flag, which when set, indicates the presence of the set of extension bits.

87. The method of claim 85, wherein the data elements comprise bits that are not used in the lowest quality layer pictures.

88. A method for decoding compressed digital video that is coded using a technique that provides two or more temporal layers, wherein compressed video pictures are structured into one or more packets, and received over an IP-based network using RTP, the method comprising:

providing an RTP header extension that includes;

a series number associated with each layer,

a sequence number that is associated with each lowest temporal layer picture, and

a flag that is used to indicate if a packet contains a picture or picture fragment of the lowest layer temporal picture,

wherein the sequence number is referenced by all other pictures that use said lowest temporal layer picture as reference, and;

examining the RTP header extension in a received picture to verify availability of the picture corresponding to the referenced series number and sequence number so that loss of a lowest temporal level picture can be detected.

89. The method of claim 88 further comprising:

sending a negative acknowledgment message formatted as a RTCP feedback message upon detection of loss of a lowest temporal layer picture, with the feed back message indicating;

the sequence number of the lost picture, the series number that the lost picture belongs to, and a bitmask indicating which among the pictures that follow the one indicated by the said sequence number is also lost, whereby a transmitting system can take corrective action.

90. A method for decoding compressed digital video that is coded using a technique that provides two or more temporal layers, wherein compressed video pictures are structured into one or more packets, and packets of at least the lowest temporal level pictures may be received in a compressed digital video stream from a sender faster than real-time and after their intended presentation time, in case of errors or upon initiating decoding, the method comprising:

decoding pictures received at an endpoint faster than real-time and, if decoded after their intended presentation time, without displaying them, whereby the decoder can synchronize with the received compressed digital video stream; and

,

after such synchronization is achieved, performing normal decoding and presentation.