Systems and methods for error resilience in video communication systems

• US 8,462,856 B2
• Filed: 01/09/2008
• Issued: 06/11/2013
• Est. Priority Date: 01/09/2007
• Status: Active Grant

• Current Assignee: Vidyo Incorporated (Enghouse Systems Limited)
• Original Assignee: Vidyo Incorporated (Enghouse Systems Limited)
• Inventors: Eleftheriadis, Alexandros, Cipolli, Stephen, Lennox, Jonathan
• Primary Examiner(s): Tran, Thai
• Assistant Examiner(s): Park, Sunghyoun
• Application Number:
  US11/971,769
• Publication Number:
  US 20080165864A1
• Time in Patent Office: 1,980 Days
• Field of Search: None
• US Class Current: 375/240.29
• CPC Class Codes: H04N 19/166 concerning the amount of transmission errors, e.g. bit error rate [BER]
• CPC Class Codes: H04N 19/31 in the temporal domain
• CPC Class Codes: H04N 19/33 in the spatial domain
• CPC Class Codes: H04N 19/573 Motion compensation with multiple frame prediction using two or more reference frames in a given prediction direction
• CPC Class Codes: H04N 19/58 Motion compensation with long-term prediction, i.e. the reference frame for a current frame not being the temporally closest one H04N19/23 takes precedence
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/65 using error resilience
• CPC Class Codes: H04N 19/70 characterised by syntax aspects related to video coding, e.g. related to compression standards
• CPC Class Codes: H04N 19/89 involving methods or arrangements for detection of transmission errors at the decoder
• CPC Class Codes: H04N 21/234327 by decomposing into layers, e.g. base layer and one or more enhancement layers
• CPC Class Codes: H04N 21/6125 involving transmission via Internet transmission by internet of broadcast information H04H60/82
• CPC Class Codes: H04N 21/6375 for requesting retransmission , e.g. of data packets lost or corrupted during transmission from serverARQ protocols H04L1/18; implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP] H04L69/16
• CPC Class Codes: H04N 21/6377 directed to server one-way streaming services wherein the source is controlled by the destination H04L65/613
• CPC Class Codes: H04N 21/658 Transmission by the client directed to the server

Abstract

Systems and methods for error resilient transmission and for random access in video communication systems are provided. The video communication systems are based on single-layer, scalable video, or simulcast video coding with temporal scalability, which may be used in video communication systems. A set of video frames or pictures in a video signal transmission is designated for reliable or guaranteed delivery to receivers using secure or high reliability links, or by retransmission techniques. The reliably-delivered video frames are used as reference pictures for resynchronization of receivers with the transmitted video signal after error incidence and for random access.

Claims

1. A system for media communications between a transmitting endpoint or server and one or more receiving endpoint(s) or server(s) over a packet-based communication network, the system comprising:

an encoder configured to use a threaded coding structure having a number of different layers including a lowest temporal layer, wherein each picture is associated with a picture index number;

wherein data corresponding to a single picture is portioned and transmitted in one or more individual data packets,

wherein an individual data packet comprises data elements that indicate;

for the lowest temporal layer pictures, a sequence number identifying said pictures,

for other temporal layer pictures, a reference to the sequence number of the most recent, in decoding order, lowest temporal layer picture, and

for all pictures, a ‘

start’

flag and an ‘

end’

flag that respectively indicate if the individual data packet contains the first or last data portions of the picture,

so that a receiving endpoint or server on receipt of the individual data packet can detect a loss of portions of a lowest temporal layer picture by examining if all data portions of the picture corresponding to the referenced picture index have been received at the receiving endpoint or server such that their packet sequence numbers contain no gaps, and where the first such data portion has the ‘

start’

bit set and the last such data portion has the ‘

end’

bit set.

2. The system of claim 1 wherein the data elements additionally indicate a series number associated with each spatial or quality layer, wherein the receiving endpoint or server detects if a lowest temporal layer picture of a particular spatial or quality layer is lost by determining if the picture corresponding to the referenced series number and sequence number has been received at the receiving endpoint or server.

3. The system of claim 1 comprising a codec conforming to H.264, wherein the lowest temporal layer pictures include pictures that are marked as long term reference pictures, and wherein decoding of at least a portion of the received media based on a lowest temporal layer picture by a decoder is controlled by Memory Management Control Operation (MMCO) commands.

4. The system of claim 1 wherein the communication network uses the Internet Protocol, media transport is performed using real-time protocol (RTP), and the data elements include data to indicate presence of a lowest temporal layer frame or fragment thereof in the packet.

5. The system of claim 4 wherein a receiving endpoint or server sends a negative acknowledgment message to the transmitting endpoint or server in response to the receiving endpoint'"'"'s or server'"'"'s detection of a lost lowest temporal layer picture or portion of such picture.

6. The system of claim 5 wherein the transmitting endpoint or server upon receiving the negative acknowledgment message retransmit the lost lowest temporal layer picture or portion of such picture.

7. The system of claim 1, wherein the encoder conforms to H.264 Scalable Video Coding (SVC), and the data elements are carried in Network Adaptation Layer (NAL) unit header extension for SVC elements.

8. The system of claim 1, wherein the encoder conforms to H.264 SVC, and the data elements are carried in Supplemental Enhancement Information (SEI) message.

9. A method for media communications between a transmitting endpoint or server and one or more receiving endpoint(s) or server(s) over a packet-based communication network, wherein an encoder is configured to use a threaded coding structure having a number of different layers including a lowest temporal layer, and wherein data corresponding to a single picture is portioned and transmitted in one or more individual data packets, the method comprising:

placing in each individual data packet data elements that indicate;

for the lowest temporal layer pictures, a sequence or index number identifying said pictures,

for other temporal layer pictures, a reference to the sequence number of the most recent, in decoding order, lowest temporal layer picture, and

for all pictures, a ‘

start’

flag and an ‘

end’

flag that indicate if the individual data packet contains the first or last, respectively, data portions of the picture, so that a receiving endpoint or server on receipt of the individual data packet can detect a loss of portions of a lowest temporal layer picture by examining if all data portions of the picture corresponding to the referenced picture index have been received at the receiving endpoint or server such that their packet sequence numbers contain no gaps, and where the first such data portion has the ‘

start’

bit set and the last such data portion has the ‘

end’

bit set.

10. The method of claim 9 wherein the data elements additionally indicate a series number associated with each spatial or quality layer, so that the receiving endpoint or server detects if a lowest temporal layer picture of a particular spatial or quality layer is lost by determining if the picture corresponding to the referenced series number and sequence number has been received at the receiving endpoint or server.

11. The method of claim 9, wherein the transmitted media is coded using a codec conforming to H.264, wherein the lowest temporal layer pictures include pictures that are marked as long term reference pictures, and wherein decoding of at least a portion of the received media based on a lowest temporal layer picture by a decoder is controlled by Memory Management Control Operation (MMCO) commands.

12. The method of claim 9 wherein the communication network uses the Internet Protocol, media transport is performed using real-time transport protocol (RTP), and the data elements include data to indicate presence of a lowest temporal layer picture or fragment thereof in the packet.

13. The method of claim 12 wherein a receiving endpoint or server sends a negative acknowledgment message to the transmitting endpoint or server in response to the receiving endpoint'"'"'s or server'"'"'s detection of a lost lowest temporal layer picture or portion of such picture.

14. The method of claim 13 wherein the transmitting endpoint or server upon receiving the negative acknowledgment message retransmit the lost lowest temporal layer picture or portion of such picture.

15. The method of claim 14, wherein the encoder conforms to H.264 Scalable Video Coding (SVC), and the data elements are carried in Network Adaptation Layer (NAL) unit header extension for SVC elements.

16. The method of claim 9, wherein the encoder conforms to H.264 SVC, and the data elements are carried in Supplemental Enhancement Information (SEI) message.

17. A non-transitory computer readable medium comprising a set of instructions to direct a processor to perform the methods recited in one of claims 9-16.