SYSTEMS AND METHODS FOR TRANSMITTING AND RECEIVING DATA STREAMS WITH FEEDBACK INFORMATION OVER A LOSSY NETWORK

• US 20110228845A1
• Filed: 05/31/2011
• Published: 09/22/2011
• Est. Priority Date: 07/29/2009
• Status: Active Grant

• Current Assignee: Debarag Banerjee
• Original Assignee: Debarag Banerjee
• Inventors: Banerjee, Debarag
• Granted Patent: US 8,472,520 B2
• Time in Patent Office: Days
• US Class Current: 375/240.07
• CPC Class Codes: H04L 1/0002 by adapting the transmission rate
• CPC Class Codes: H04L 1/0018 based on latency requirement
• CPC Class Codes: H04L 1/1877 for semi-reliable protocols, e.g. for less sensitive applications like streaming video buffer level management for video bitstream control arrangements H04N21/44004
• CPC Class Codes: H04L 1/1887 Scheduling and prioritising arrangements
• CPC Class Codes: H04L 2001/0093 Point-to-multipoint
• CPC Class Codes: H04L 43/0811 by checking connectivity
• CPC Class Codes: H04L 65/80 Responding to QoS

Abstract

Systems, apparatus, and methods are provided to enhance transmitting and receiving video data streams with feedback information over a lossy and/or congested network. The systems, apparatus, and methods may optimize retransmission for lost packets or lost data frames. Optimizations are also provided to select reference frames in a data stream so that block errors are contained when RPS is used to encode data frames. Optimization in decoding may include reconstructing high-resolution data frames from low-resolution data frames.

Claims

1. A system, comprising:

a decoder configured to receive an encoded frame from an encoder via a forward data channel, and to return feedback information regarding the reception of some or all of the encoded frame to the encoder via a feedback channel, wherein the encoder is configured to;

receive data frames at a first resolution from a source,

for each data frame at the first resolution, generate a corresponding encoded data frame at a second resolution,

responsive to the feedback information received from the decoder, analyze the feedback information to determine which of

(i) retransmission of a portion of the encoded frame at the first resolution, or

(ii) transmission of the corresponding data frame at the second resolution is required, and,

transmit either the portion of the encoded frame at the first resolution or the corresponding encoded data frame at the second resolution, as appropriate, to the decoder via the forward data channel,

otherwise, if retransmission of a portion of the encoded frame at the first resolution or transmission of the encoded data frame at the second resolution is not required,

transmit a subsequent encoded frame at the first resolution to the decoder via the forward data channel.

2. The system in claim 1, wherein the data frames comprise video data.

3. The system in claim 2, wherein the encoder is further configured to generate ancillary information to aid the decoder in properly reconstructing a sequence of the data frames.

4. The system in claim 2, wherein the encoder is further configured to change a context in which one or more of the data frames are encoded as part of a sequence.

5. The system in claim 4, wherein the encoder is configured change context between independently coded ones of the data frames.

6. The system in claim 4, wherein the encoder is configured change context between progressively coded ones of the data frames.

7. The system in claim 4, wherein the encoder is configured change context between bi-directionally coded ones of the data frames.

8. The system in claim 4, wherein the encoder is further configured to change reference pictures used in different encoding contexts.

9. The system in claim 2, wherein the feedback information includes acknowledgements appropriate for a communication protocol used by the encoder and the decoder.

10. The system in claim 9, wherein the acknowledgements include positive acknowledgements of a successful reception by the decoder.

11. The system in claim 9, wherein the acknowledgements include indications that reception by the decoder was not successful.

12. The system in claim 9 wherein the acknowledgements include both positive acknowledgements of a successful reception by the decoder and indications that reception by the decoder was not successful.

13. The system in claim 2, wherein the feedback information is provided at one of a media access control (MAC) layer or at a network layer.

14. The system in claim 2, wherein the feedback information is provided for every packet received by decoder.

15. The system in claim 2, wherein the feedback information is provided for groups of packets received by the decoder.

16. The system in claim 2, wherein if an entire current frame is lost or corrupted, transmission of the corresponding data frame at the second resolution to the decoder is required.

17. The system in claim 2, wherein the encoder is configured to prepare the corresponding encoded data frame at the second resolution concurrently with transmission of the data frame at the first resolution.

18. The system in claim 1, wherein the encoder is configured to retransmit the portion of the encoded frame at the first resolution if a number of packets required to retransmit the portion of the encoded frame at the first resolution is fewer than a number of packets required to transmit the corresponding encoded data frame at the second resolution, and to transmit the corresponding encoded data frame at the second resolution if the number of packets required to transmit the corresponding encoded data frame at the second resolution is fewer than the number of packets required to retransmit the portion of the encoded frame at the first resolution.

19. The system in claim 18, wherein the encoder is further configured to retransmit the corresponding encoded data frame at the second resolution or transmit a portion of the transmit the corresponding encoded data frame at the second resolution according to feedback from the decoder concerning loss of a corresponding encoded data frame at the second resolution.

20. The system in claim 1, wherein the encoder is configured to make multiple attempts to transmit the corresponding encoded data frame at the second resolution where necessary.

21. The system in claim 1, the encoder is configured to produce the corresponding encoded data frame at the second resolution by decoding and filtering the encoded frame at the first resolution, sub-sampling a result to produce a frame having fewer pixels than the encoded frame at the first resolution, and then re-encoding the frame having fewer pixels than the encoded frame at the first resolution to produce the corresponding encoded data frame at the second resolution.

22. The system in claim 1, wherein the encoder is further configured to determine whether a subject data frame is a critical frame, and, if so to afford the critical frame a higher priority and a higher maximum retransmission count than if the subject data frame is determined not to be a critical frame.

23. The system in claim 22, wherein packets associated with I-frames are deemed high priority packets.

24. The system in claim 22, wherein the encoder is further configured to determine whether or not to reshape at least some of the data frames following prioritization and encoding, but prior to transmission.

25. The system in claim 24, wherein reshaping of at least some of the data frames comprises modifying I-frames to become P-frames.

26. The system in claim 1, wherein the encoder is configured to keep a list of reference frames, each last-positively-acknowledged frame from the decoder being selected as a reference frame.

27. The system in claim 26, wherein if a frame immediately preceding a current frame at the first resolution is not positively acknowledged, then the current frame is encoded with respect to a last reference frame.

28. The system in claim 26, wherein the encoder is configured to update the list of reference frames upon receipt of a negative acknowledgement concerning a frame or a group of frames from the decoder.

29. The system in claim 1, wherein the encoder is further configured to compute a packet error rate and an available network bandwidth, and to determine whether or not to adjust a target bit rate of the encoder based on an analysis of the packet error rate and the available network bandwidth.

30. The system in claim 29, wherein the encoder is further configured to insert advertisements between frames that are sent to the decoder before encoding the data frames at the target bit rate.

31. The system in claim 30, wherein the advertisements are customized for individual viewers according to decoder media access controller (MAC) addresses, an EIN number, or another viewer identifier.

32. The system in claim 31, wherein the encoder is configured to insert customized advertisements, based on viewer identification, into a video stream on a frame-by-frame basis.

33. The system in claim 29, wherein the encoder is configured to adjust the target bit rate according to a comparison of the packet error rate (PER) with predetermined thresholds such that if the PER is greater than a predetermined high-PER threshold, the target bit rate is decreased, otherwise, if the PER is lower than a predetermined low-PER threshold, the target bit rate is increased, and if the PER is within the high-PER and low-PER thresholds, the target bit rate may, but need not be, modified.

34. The system in claim 29, wherein the encoder is configured to adjust the target bit rate with respect to the available network bandwidth, which is computed by monitoring a size of the encoder'"'"'s transmission queue and arrivals of packets at a sliding window with respect to a historical size of the transmission queue, such that if the transmission queue increases in size at a rate faster than a rate of packet arrivals, the encoder treats this as a decrease in available network bandwidth, and decreases the target bit rate, while if if the transmission queue decreases in size at a rate faster than the rate of packet arrivals, the encoder treats this as an increase in available network bandwidth, and increases the target bit rate.

35. The system in claim 29, wherein the encoder is configured to use frame quantization to produce an equivalent frame that is quantized to a desired target bit rate if the target bit rate is greater than or equal to a predetermined fraction of an incoming bit rate, and to drop frames if the available target bit rate is less than the predetermined fraction of the incoming bit rate.

36. The system in claim 1, wherein the decoder is configured to produce a pseudo-first resolution frame from a second resolution frame that was substituted in place of a first resolution frame by the encoder using first-resolution frame information resident in first resolution frames preceding and/or following a subject second resolution frame.

37. The system in claim 36, wherein the decoder is configured to produce the pseudo-first resolution frame by up-sampling the second resolution frame to produce an intermediate frame, performing a motion search between a first first resolution frame and the intermediate frame to identify a first motion vector, computing a first confidence value and comparing the first confidence value to a predetermined threshold value, wherein if the first confidence value is greater than the predetermined threshold, the decoder uses the prior first resolution frame translated by the first motion vector to produce the pseudo-first resolution frame;

but if the first confidence value is less than the predetermined threshold, a second motion vector is generated by performing a motion search between a second first resolution frame and the intermediate frame, and a second confidence value is calculated therefrom and compared to the predetermined threshold value, so that if the second confidence value is greater than the predetermined threshold, the decoder uses the second first resolution frame translated by the second motion vector to produce a restructured first resolution frame (RHR);

otherwise the decoder uses the intermediate frame as the pseudo-first resolution frame.

38. The system in claim 1, further comprising multiple decoders, each decoder configured to receive an encoded broadcast frame from the encoder via a broadcast channel, and to return feedback information to the encoder.

39. The system in claim 38, wherein the decoders share one or more common feedback channels, with feedback information from each decoder being “

personalized”

to a respective decoder through use of decoder identification information included in the feedback information.

40. The system in claim 1 wherein the encoder is configured to use a different channel to transmit the second resolution frames or retransmit the portions of the first resolution frames than was used to transmit the corresponding first resolution fames.

41. A projector, comprising a server storing software to be installed on a computing device to be used with the projector, the projector being configured to display an instruction screen upon power up, the instruction screen including a network address for the server at which the software to be installed on the computing device can be obtained.

42. The system in claim 41, wherein the projector is configured to broadcast a projector ID, using which the computing device can establish a wireless network connection with the server.