Methods and systems for forward error correction based loss recovery for interactive video transmission

• US 6,421,387 B1
• Filed: 08/10/2000
• Issued: 07/16/2002
• Est. Priority Date: 05/15/1998
• Status: Expired due to Fees

• Current Assignee: North Carolina State University (University of North Carolina System)
• Original Assignee: North Carolina State University (University of North Carolina System)
• Inventors: Rhee, Injong
• Primary Examiner(s): BRITTON, HOWARD W
• Application Number:
  US09/635,614
• Time in Patent Office: 705 Days
• Field of Search: H04/N.764, H04/N.736, 375/240, 375/240.27, 375/240.12, 714/18, 714/48
• US Class Current: 375/240.27
• CPC Class Codes: H04B 1/66 for reducing bandwidth of signals; for improving efficiency of transmission H04B1/68 takes precedence
• CPC Class Codes: H04L 1/0045 Arrangements at the receiver end
• CPC Class Codes: H04L 1/0057 Block codes H04L1/0061, H04L1/0064 take precedence
• CPC Class Codes: H04L 1/1816 with retransmission of the same, encoded, message
• CPC Class Codes: H04N 19/89 involving methods or arrangements for detection of transmission errors at the decoder
• CPC Class Codes: H04N 21/6375 for requesting retransmission , e.g. of data packets lost or corrupted during transmission from serverARQ protocols H04L1/18; implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP] H04L69/16
• CPC Class Codes: H04N 21/6437 Real-time Transport Protocol [RTP]

Abstract

Real-time interactive video transmission in the current Internet has mediocre quality because of high packet loss rates. Loss of packets belonging to a video frame is evident not only in the reduced quality of that frame but also in the propagation of that distortion to successive frames. This error propagation problem is inherent in any motion-based video codec because of the interdependence of encoded video frames. Since packet losses in the best-effort Internet environment cannot be prevented, minimizing the impact of these packet losses to the final video quality is important. A new forward error correction (FEC) technique effectively alleviates error propagation in the transmission of interactive video. The technique is based on a recently developed error recovery scheme called Recovery from Error Spread using Continuous Updates (RESCU). RESCU allows transport level recovery techniques previously known to be infeasible for interactive video transmission applications to be successfully used in such applications. The FEC technique can be very useful when the feedback channel from the receiver is highly limited, or transmission delay is high. Both simulation and Internet experiments indicate that the FEC technique effectively alleviates the error spread problem and is able to sustain much better video quality than H.261 or other conventional FEC schemes under various packet loss rates.

Claims

1. A method for transmitting video data over a lossy packet-based network, the method comprising:

at a transmitter,

(a) encoding a first video frame into k data packets, k being an integer;

(b) determining n minus k (n−

k) forward error correction (FEC) packets for the first video frame, n being an integer;

(c) transmitting the k data packets over a lossy packet-based network to a receiver during a first frame interval;

(d) transmitting the n−

k FEC packets to the receiver during frame intervals after the first frame interval;

at the receiver,

(e) receiving some of the k data packets, decoding the received data packets to reproduce the first video frame, and displaying the first video frame with errors;

(f) receiving the n−

k FEC packets and using the n−

k FEC packets to restore the first video frame; and

(g) using the first video frame as a reference frame for decoding at least one subsequent video frame.

2. The method of claim 1 wherein transmitting the n−

k FEC packets to the receiver comprises interleaving the n−

k FEC packets with data packets of video frames to be displayed at the receiver after display of the first video frame.

7. The method of claim 1 wherein transmitting the n−

k FEC packets to the receiver includes encapsulating the n−

k FEC packets in real time protocol (RTP) packets.

8. The method of claim 7 wherein encapsulating the n−

k FEC packets in RTP packets includes adding an RTP header and a FEC header to each of the n−

k FEC packets.

9. The method of claim 8 wherein each FEC header includes a field for indicating the number of data packets used to generate each FEC packet.

10. The method of claim 1 wherein the first video frame is a periodic frame and using first video frame as a reference frame for decoding at least one subsequent video frame includes using the first video frame as a reference frame for decoding a subsequent periodic frame.

11. The method of claim 10 wherein using the first video frame as a reference frame for decoding a subsequent periodic frame comprises using the first video frame as a reference frame for decoding a second periodic frame immediately following the first video frame.

12. The method of claim 11 comprising decoding the second periodic frame using the first video frame as a reference frame before display of the second periodic frame.

13. The method of claim 11 comprising decoding the second periodic frame using the first video frame as a reference frame after display of the second periodic frame.

14. The method of claim 13 comprising using the second period frame as a reference frame for decoding a third periodic frame received after the second periodic frame.

3. The method of 1 comprising determining a time interval between the n−

k FEC packets required to protect the n−

k FEC packets from burst losses and wherein transmitting the n−

k FEC packets to the receiver includes spacing transmission of each of the n−

k FEC packets by the time interval.

4. The method of claim 3 wherein determining a time interval required to protect the n−

k FEC packets from burst loss includes determining the time interval based on feedback from the receiver.

5. The method of claim 4 wherein the feedback from the receiver includes acknowledgments of previously transmitted data packets.

6. The method of claim 5 wherein the feedback from the receiver includes negative acknowledgments of indicative of lost data packets.

15. A method for optimized cascaded error recovery when transmitting video over a lossy packet-based network, the method comprising:

(a) receiving, over a lossy packet-based network, a plurality of data packets associated with a first video frame F_i, i being an integer;

(b) decoding the data packets to reproduce the first video frame F_i;

(c) displaying the first video frame F_i with errors;

(d) after display of the first video frame F_i, receiving error correction packets associated with the first video frame F_i;

(e) repairing the first video frame F_i using the error correction packets;

(f) receiving data packets associated with periodic frames F_i+n, n being an integer, transmitted after the first video frame F_i;

(g) using motion estimation to determine the effect of errors in the first video frame F_i on the subsequent periodic frames F_i+n and

(h) restoring the subsequent periodic frames F_i+n using the restored first video frame F_i and the motion estimation.

16. The method of claim 15 wherein receiving error correction packets includes receiving forward error correction (FEC) packets.

17. The method of claim 15 wherein receiving error correction packets includes receiving retransmitted data packets.

18. The method of claim 15 wherein restoring the periodic frames F_i+n includes restoring a periodic frame F_i+2 using the first video frame as a reference and motion estimation without restoring a periodic frame F_i+1.

19. A computer program product comprising computer-executable instructions embodied in a computer readable medium for performing steps comprising:

at a transmitter,

(a) encoding a first video frame into k data packets, k being an integer;

(b) determining n minus k (n−

k) forward error correction (FEC) packets for the first video frame, n being an integer;

(c) transmitting the k data packets over a lossy packet-based network to a receiver during a first frame interval;

(d) transmitting the n−

k FEC packets to the receiver during frame intervals after the first frame interval;

at the receiver,

(e) receiving some of the k data packets, decoding the received data packets to reproduce the first video frame and displaying the first video frame with errors;

(f) receiving the n−

k FEC packets and using the n−

k FEC packets to restore the first video frame; and

(g) using the first video frame as a reference frame for decoding at least one subsequent video frame.

20. The computer program product of claim 19 wherein transmitting the n−

k FEC packets to the receiver comprises interleaving the n−

k FEC packets with data packets of video frames to be displayed at the receiver after display of the first video frame.

25. The computer program product of claim 19 wherein transmitting the n−

k FEC packets to the receiver includes encapsulating the n−

k FEC packets in real time protocol (RTP) packets.

26. The computer program product of claim 25 wherein encapsulating the FEC packets in RTP packets includes adding an RTP header and a FEC header to each of the n−

k FEC packets.

27. The computer program product of claim 26 wherein each FEC header includes a field for indicating the number of data packets used to generate each FEC packet.

28. The computer program product of claim 19 wherein the first video frame is a periodic frame and using first video frame as a reference frame for decoding at least one subsequent video frame includes using the first video frame as a reference frame for decoding a subsequent periodic frame.

29. The computer program product of claim 28 wherein using the first video frame as a reference frame for decoding a subsequent periodic frame comprises using the first video frame as a reference frame for decoding a second periodic frame immediately following the first video frame.

30. The computer program product of claim 29 comprising decoding the second periodic frame using the first video frame as a reference frame before display of the second periodic frame.

31. The computer program product of claim 29 comprising decoding the second periodic frame using the first video frame as a reference frame after display of the second periodic frame.

32. The computer program product of claim 31 comprising using the second period frame as a reference frame for decoding a third periodic frame received after the second periodic frame.

21. The computer program product of 19 comprising determining a time interval between FEC packets required to protect the FEC packets from burst losses and wherein transmitting the n−

k FEC packets to the receiver includes spacing transmission of each of the n−

k FEC packets by the time interval.

22. The computer program product of claim 21 wherein determining a time interval required to protect the FEC packets from burst loss includes determining the time interval based on feedback from the receiver.

23. The computer program product of claim 22 wherein the feedback from the receiver includes acknowledgments of previously transmitted data packets.

24. The computer program product of claim 23 wherein the feedback from the receiver includes negative acknowledgments of indicative of lost data packets.

33. A computer program product comprising computer-executable instructions embodied in a computer readable medium for performing steps comprising:

(a) receiving, over a lossy packet-based network, a plurality of data packets associated with a first video frame F_i, i being an integer;

(b) decoding the data packets to reproduce the first video frame F_i;

(c) displaying the first video frame F_i with errors;

(d) after display of the first video frame F_i, receiving error correction packets associated with the first video frame F_i;

(e) repairing the first video frame F_i using the error correction packets;

(f) receiving data packets associated with periodic frames F_i+n, n being an integer, transmitted after the first video frame F_i;

(g) using motion estimation to determine the effect of errors in the first video frame F_i on the subsequent periodic frames F_i+n and

(h) restoring the subsequent periodic frames F_i+n using the restored first video frame F_i and the motion estimation.

34. The computer program product of claim 33 wherein receiving error correction packets includes receiving forward error correction (FEC) packets.

35. The computer program product of claim 33 wherein receiving error correction packets includes receiving retransmitted data packets.

36. The computer program product of claim 33 wherein restoring the periodic frames F_i+n includes restoring a periodic frame F_i+2 using the first video frame as a reference and motion estimation without restoring a periodic frame F_i+1.