Systems and methods for error resilience and random access in video communication systems

• US 9,077,964 B2
• Filed: 12/08/2006
• Issued: 07/07/2015
• Est. Priority Date: 12/08/2005
• Status: Active Grant

• Current Assignee: Vidyo Incorporated (Enghouse Systems Limited)
• Original Assignee: Layered Meddia Incorporated
• Inventors: Cipolli, Stephen, Civanlar, Reha, Eleftheriadis, Alexandros, Lennox, Jonathan, Sasson, Roi, Saxena, Manoj, Shapiro, Ofer
• Primary Examiner(s): Dang, Hung
• Assistant Examiner(s): Park, Sunghyoun
• Application Number:
  US11/608,776
• Publication Number:
  US 20070206673A1
• Time in Patent Office: 3,133 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: H04L 1/1607 Details of the supervisory signal
• CPC Class Codes: H04L 1/1812 Hybrid protocols; Hybrid automatic repeat request [HARQ]
• CPC Class Codes: H04L 1/1848 Time-out mechanisms
• CPC Class Codes: H04L 65/752 adapting media to network capabilities
• CPC Class Codes: H04L 65/756 adapting media to device capabilities
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/166 concerning the amount of transmission errors, e.g. bit error rate [BER]
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/31 in the temporal domain
• CPC Class Codes: H04N 19/37 with arrangements for assigning different transmission priorities to video input data or to video coded data
• CPC Class Codes: H04N 19/463 by compressing encoding parameters before transmission
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 19/573 Motion compensation with multiple frame prediction using two or more reference frames in a given prediction direction
• CPC Class Codes: H04N 19/577 Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
• CPC Class Codes: H04N 19/58 Motion compensation with long-term prediction, i.e. the reference frame for a current frame not being the temporally closest one H04N19/23 takes precedence
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/65 using error resilience
• CPC Class Codes: H04N 19/66 : involving data partitioning, i.e. separation of data into packets or partitions according to importance
• CPC Class Codes: H04N 19/67 : involving unequal error protection [UEP], i.e. providing protection according to the importance of the data
• CPC Class Codes: H04N 19/70 : characterised by syntax aspects related to video coding, e.g. related to compression standards
• CPC Class Codes: H04N 19/89 : involving methods or arrangements for detection of transmission errors at the decoder
• CPC Class Codes: H04N 21/2662 : Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 21/4621 : Controlling the complexity of the content stream or additional data, e.g. lowering the resolution or bit-rate of the video stream for a mobile client with a small screen arrangements for using the results of monitoring on user's side in broadcast systems H04H60/65; flow control in packet networks H04L47/10
• CPC Class Codes: H04N 21/631 : Multimode Transmission, e.g. transmitting basic layers and enhancement layers of the content over different transmission paths or transmitting with different error corrections, different keys or with different transmission protocols
• CPC Class Codes: H04N 21/6375 : for requesting retransmission , e.g. of data packets lost or corrupted during transmission from serverARQ protocols H04L1/18; implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP] H04L69/16
• CPC Class Codes: H04N 21/6583 : Acknowledgement

Abstract

Systems and methods for error resilient transmission and for random access in video communication systems are provided. The video communication systems are based on single-layer, scalable video, or simulcast video coding with temporal scalability, which may be used in video communication systems. A set of video frames or pictures in a video signal transmission is designated for reliable or guaranteed delivery to receivers using secure or high reliability links, or by retransmission techniques. The reliably-delivered video frames are used as reference pictures for resynchronization of receivers with the transmitted video signal after error incidence and for random access.

Claims

1. A system for media communications between a transmitting endpoint and one or more receiving endpoint(s) or server(s) over a communication network, the system comprising:

an encoder configured to use a threaded coding structure having a number of different layers including a lowest temporal layer,

wherein transmitted frames comprise data elements that indicate;

for at least one but not all of the lowest temporal layer frames, a sequence number identifying said at least one frame, where the sequence number increases for each of the at least one but not all of the lowest temporal layer frames, and

for other temporal layer frames, a reference to the sequence number of the most recent, in decoding order, of the at least one but not all of the lowest temporal layer frames.

2. The system of claim 1 wherein the data elements additionally indicate a series number associated with each spatial or quality layer, wherein the receiving endpoint or server detects if a lowest temporal layer frame of a particular spatial or quality layer is lost by determining if the frame corresponding to the referenced series number and sequence number has been received at the receiving endpoint or server.

3. The system of claim 1 wherein the data elements include a flag to indicate presence of a lowest temporal layer frame or fragment thereof in the packet.

4. The system of claim 3 wherein a receiving endpoint or server in the network sends a negative acknowledgment message in response to the receiving endpoint'"'"'s or server'"'"'s detection of a lost R (Reliable) frame or portion of a frame,

wherein the negative acknowledgment message includes the sequence number of the lost frame, a series number indicating the lost frame layer, and information indicating which among the frames that follow the one indicated by the said sequence number are also lost.

5. The system of claim 4 wherein the transmitting endpoint or server in the network upon receiving the negative acknowledgment message check whether the lost frame has been superseded by a recent frame, and wherein the transmitting endpoint or server accordingly retransmit the lost frame if not superseded, or transmit the recent frame if the lost frame is superseded with an indication of a range of frames including the lost frame that have been superseded.

6. The system of claim 1, wherein the encoder conforms to H.264 Scalable Video Coding (SVC) and the data elements are carried in Network Adaptation Layer (NAL) unit header extension for SVC elements.

7. The system of claim 6 wherein the data elements comprise an additional byte in the NAL header extension for SVC and wherein a flag in the NAL, header extension for SVC signals the presence of the additional byte.

8. A system for media communications comprising:

a decoder for decoding compressed digital video that is coded using a technique that provides two or more temporal layers, wherein compressed video frames are structured into one or more packets, wherein the decoder is configured to receive;

a packet header containing at least one data element that indicates;

for at least one but not all of the lowest temporal layer frames, a sequence number identifying said at least one frame, where the sequence number increases for each of the at least one but not all of the lowest temporal layer frames,

for other temporal layer frames, a reference to the sequence number of the most recent, in decoding order, of the at least one but not all of the lowest temporal layer frames.

9. The system of claim 8, wherein the data elements comprises a set of extension bits and a flag, which when set, indicates the presence of the set of extension bits.

10. The system of claim 8 further comprising:

a receiver which, upon detecting the loss of a lowest temporal layer frame, generates a negative acknowledgment message that indicates the sequence number of the lost lowest temporal layer frame.

11. A method for media communications between a transmitting endpoint and one or more receiving endpoint(s) or bridge(s) over a communication network, wherein transmitted media is encoded as frames in a threaded coding structure having a number of different layers including a lowest temporal layer, the method comprising providing data elements that indicate:

for at least one but not all of the lowest temporal layer frames, a sequence number identifying said at least one frame, where the sequence number increases for each of the at least one but not all of the lowest temporal layer frames, and

for other temporal layer frames a reference to the sequence number of the most recent, in decoding order, of the at least one but not all of the lowest temporal layer frames.

12. The method of claim 11 wherein the data elements additionally indicate a series number associated with each spatial or quality layer, wherein the receiving endpoint or bridge detects if a lowest temporal layer frame of a particular spatial or quality layer is lost by determining if the frame corresponding to the referenced series number and sequence number has been received at the receiving endpoint or bridge.

13. The method of claim 11 wherein a receiving endpoint or bridge in the network sends a negative acknowledgment message in response to the receiving endpoint or bridge'"'"'s detection of a lost R frame or portion of a frame, the method further comprising:

including in the feedback message the sequence number of the lost frame, and information indicating which among the pictures that follow the one indicated by the said sequence number are also lost.

14. The method of claim 13, further comprising:

upon receiving the negative acknowledgment message, checking at the transmitting endpoint or bridge in the network whether the lost frame has been superseded by a recent frame; and

accordingly, retransmitting the lost frame if not superseded, or retransmitting the recent frame if the lost frame is superseded with an indication of a range of frames including the lost frame that have been superseded.

15. The method of claim 11 wherein the encoding conforms to H.264 Scalable Video Coding (SVC), and the data elements are carried in Network Adaptation Layer (NAL) unit header extension for SVC elements.

16. The method of claim 15 wherein the data elements comprise an additional byte in the NAL header extension for SVC and wherein a flag in the NAL header extension for SVC which signals the presence of the additional byte.

17. The method of claim 15 the data elements comprise bits related to Fine-Granularity Scalability (FGS) coding in the NAL header extension fbr SVC that are not used by pictures of the lowest quality layer.

18. A method for decoding compressed digital video that is coded using a technique that provides two or more temporal layers, wherein compressed video pictures are structured into one or more packets, the method comprising:

receiving data elements in a packet header to indicate;

for at least one but not all of the lowest temporal layer pictures, a sequence number identifying the at least one picture, where the sequence number increases for each of the at least one but not all of the lowest temporal layer pictures,

for other temporal layer pictures, a reference to the sequence number of the most recent, in decoding order, of the at least one but not all of the lowest temporal layer pictures.

19. The method of claim 18, wherein the data elements comprise a set of extension bits and a flag, which when set, indicates the presence of the set of extension bits.

20. A non-transitory computer-readable medium for media communications between a transmitting endpoint and one or more receiving endpoint(s) or bridge(s) over a communication network, wherein transmitted media is encoded as frames in a threaded coding structure having a number of different layers including a lowest temporal layer, the computer-readable medium having a set of instructions operable to direct a processing system to provide data elements that indicate:

for at least one but not all of the lowest temporal layer frames, a sequence number identifying said at least one frame, where the sequence number increases for each of the at least one but not all of the lowest temporal layer frames, and

for other temporal layer frames a reference to the sequence number of the most recent, in decoding order, of the at least one but not all of the lowest temporal layer frames.

21. The non-transitory computer-readable medium of claim 20 wherein a receiving endpoint or bridge in the network sends a negative acknowledgment message in response to the receiving endpoint or bridge'"'"'s detection of a lost R frame or portion of a frame, wherein the set of instructions is further operable to direct the processing system to:

include in the feedback message the sequence number of the lost frame, and information indicating which among the pictures that follow the one indicated by the said sequence number are also lost.

22. The non-transitory computer-readable medium of claim 21, wherein the set of instructions is further operable to direct the processing system to:

upon receiving the negative acknowledgment message, check at the transmitting endpoint or bridge in the network whether the lost frame has been superseded by a recent frame; and

accordingly, retransmit the lost frame if not superseded, or retransmitting the recent frame if the lost frame is superseded with an indication of a range of frames including the lost frame that have been superseded.

23. A non-transitory computer-readable medium for decoding compressed digital video that is coded using a technique that provides two or more temporal layers, wherein compressed video pictures are structured into one or more packets, the computer-readable medium having a set of instructions operable to direct a processing system to:

provide data elements in a transmitted packet header to indicate;

for at least one but not all of the lowest temporal layer pictures, a sequence number identifying the at least one picture, where the sequence number increases for each of the at least one but not all of the lowest temporal layer pictures,

for other temporal layer pictures, a reference to the sequence number of the most recent, in decoding order, of the at least one but not all of the lowest temporal layer pictures.

24. A non-transitory computer-readable medium for decoding compressed digital video that is coded using a technique that provides two or more temporal layers, wherein compressed video pictures are structured into one or more packets, and received over an IP-based network using Real-Time Transport Protocol (RTP), the computer-readable medium having a set of instructions operable to direct a processing system to:

provide an RTP header extension that includes;

a series number associated with each layer,

a sequence number that is associated with each lowest temporal layer picture, where the sequence number increases for each of the at least one but not all of the lowest temporal layer pictures, and

a flag that is used to indicate if a packet contains a picture or picture fragment of the lowest layer temporal picture,

wherein the sequence number is referenced by all other pictures that use said lowest temporal layer picture as reference, and

examine the RTP header extension in a received picture to verily availability of the picture corresponding to the referenced series number and sequence number so that loss of a lowest temporal layer picture can be detected.

25. The non-transitory computer-readable medium of claim 24 wherein the set of instructions is further operable to direct the processing system to:

send a negative acknowledgment message formatted as a Real-Time Transport Control Protocol (RTCP) feedback message upon detection of loss of a lowest temporal layer picture, with the feed back message indicating;

the sequence number of the lost picture, the series number that the lost picture belongs to, and a bitmask indicating which among the pictures that follow the one indicated by the said sequence number is also lost, whereby a transmitting system can take corrective action.

26. A decoder for decoding compressed digital video that is coded using a technique that provides two or more temporal layers including a lowest temporal layer, wherein the decoder is configured to receive:

a frame header containing at least one data element that indicates;

for at least one but not all of the lowest temporal layer frames, a sequence number identifying the lowest temporal layer frame, where the sequence number increases for each of the at least one but not all of the lowest temporal layer frames,

for other temporal layer frames, a reference to the sequence number of the most recent, in decoding order, of the at least one but not all of the lowest temporal layer frames.

27. A non-transitory computer-readable medium for decoding compressed digital video that is coded using a technique that provides two or more temporal layers including a lowest temporal layer, the computer-readable medium having a set of instructions operable to direct a processing system to receive:

a frame header containing at least one data element that indicates;

for at least one but not all of the lowest temporal layer frames, a sequence number identifying the lowest temporal layer frame, where the sequence number increases for each of the at least one but not all of the lowest temporal layer frames,

for other temporal layer frames, a reference to the sequence number of the most recent, in decoding order, of the at least one but not all of the lowest temporal layer frames.