Effective intra-frame refresh in multimedia communications over packet networks

• US 10,230,651 B2
• Filed: 04/13/2017
• Issued: 03/12/2019
• Est. Priority Date: 11/14/2014
• Status: Active Grant

• Current Assignee: Dialogic, Inc. (Enghouse Systems Limited), GM Global Technology Operations LLC (General Motors Company)
• Original Assignee: Dialogic, Inc. (Enghouse Systems Limited)
• Inventors: Yang, Kyeong Ho
• Primary Examiner(s): Zhao, Wei
• Application Number:
  US15/486,566
• Publication Number:
  US 20170279729A1
• Time in Patent Office: 698 Days
• CPC Class Codes: H04L 47/32 by discarding or delaying data units, e.g. packets or frames
• CPC Class Codes: H04L 47/743 Reaction at the end points
• CPC Class Codes: H04L 47/801 Real time traffic
• CPC Class Codes: H04L 65/613 for the control of the source by the destination control signals issued by the client directed to the server or network components specially adapted for selective content distribution H04N21/637
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04N 21/2402 Monitoring of the downstream path of the transmission network, e.g. bandwidth available traffic monitoring in data switching networks H04L43/00; monitoring data switching networks utilization H04L43/0876
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/44209 Monitoring of downstream path of the transmission network originating from a server, e.g. bandwidth variations of a wireless network arrangements for maintenance or administration in data switching networks involving bandwidth and capacity management H04L41/0896
• CPC Class Codes: H04N 21/6375 for requesting retransmission , e.g. of data packets lost or corrupted during transmission from serverARQ protocols H04L1/18; implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP] H04L69/16
• CPC Class Codes: H04N 21/6582 Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number arrangements where receivers interact with the broadcast H04H20/38
• CPC Class Codes: H04N 21/8453 by locking or enabling a set of features, e.g. optional functionalities in an executable program
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q

Abstract

Systems and methods of performing intra-frame refresh in multimedia communications over lossy packet networks, in which a video receiver can provide packet loss feedback information to a video transmitter, and the video transmitter can respond to the feedback information, in a manner that makes efficient use of available bandwidth. By providing one or more PLI messages from the video receiver to the video transmitter based on criteria related to the detection of an eventual missing video packet and/or the determination that the quality of a current reference frame is bad, and by pausing the providing of GNACK messages from the video receiver to the video transmitter while the PLI messages are being provided to the video transmitter, the total number of video packets required for transmission/retransmission can be reduced, thereby limiting the impact of the transmitted/retransmitted video packets on the available bandwidth while enhancing the video QoE of system users.

Claims

1. A method of performing intra-frame refresh in multimedia communications between a video transmitter and a video receiver over a packet communications network, the method comprising:

detecting, at the video receiver, at least one eventual missing video packet in a plurality of video packets from the video transmitter, the at least one eventual missing video packet corresponding to at least one missing packet not received at the video receiver after a predetermined wait time has passed;

having detected the eventual packet loss at the video receiver;

detecting an absence of at least one packet associated with an intra-frame after the at least one eventual missing video packet in the plurality of video packets from the video transmitter is detected; and

transmitting, by the video receiver, one or more feedback packets to the video transmitter to request transmission of an intra-frame while decoding of the plurality of video packets at the video receiver is paused; and

upon receipt of the requested intra-frame at the video receiver, stopping the transmitting of the one or more feedback packets to the video transmitter while the decoding of the plurality of video packets at the video receiver is resumed.

2. The method of claim 1 further comprising:

determining a quality of a current reference frame at the video receiver.

3. The method of claim 2 further comprising:

assigning a first quality designation to the current reference frame in the event a complete intra-frame is received at the video receiver; and

assigning a second quality designation to the current reference frame in the event of (1) the subsequent detection of packet loss at the video receiver, or (2) detection of a bad packet at the video receiver.

4. The method of claim 3 further comprising:

having assigned the second quality designation to the current reference frame;

transmitting, by the video receiver, one or more additional feedback packets to the video transmitter to request transmission of a further intra-frame while the decoding of the plurality of video packets at the video receiver is paused.

5. The method of claim 4 further comprising:

upon receipt of the requested further intra-frame at the video receiver, stopping the transmitting of the one or more additional feedback packets to the video transmitter while the decoding of the plurality of video packets at the video receiver is resumed.

6. A system for performing intra-frame refresh in multimedia communications over a packet communications network, comprising:

a video receiver including a video decoder, a network packet processor, and a packet parser,

wherein the network packet processor includes a jitter buffer, a missing packet detector, and a feedback packet transmitter,

wherein the packet parser includes an intra-frame detector,

wherein the missing packet detector is configured to detect at least one eventual missing video packet among a plurality of video packets in the jitter buffer, the at least one eventual missing video packet corresponding to at least one missing packet not received at the video receiver after a predetermined wait time has passed;

wherein, upon detection of the eventual packet loss at the video receiver, the intra-frame detector is configured;

to detect an absence of at least one packet associated with an intra-frame in the jitter buffer after the at least one eventual missing video packet in the plurality of video packets in the jitter buffer is detected; and

to direct the feedback packet transmitter to transmit one or more feedback packets to the video transmitter to request transmission of an intra-frame while decoding of the plurality of video packets by the video decoder is paused; and

upon receipt of the requested intra-frame, to direct the feedback packet transmitter to stop transmitting the one or more feedback packets to the video transmitter while the decoding of the plurality of video packets by the video decoder is resumed.

7. The system of claim 6 wherein the packet parser further includes a reference frame quality estimator configured to determine a quality of a current reference frame at the video receiver.

8. The system of claim 7 wherein the reference frame quality estimator is further configured to assign a first quality designation to the current reference frame in the event a complete intra-frame is received at the video receiver, and to assign a second quality designation to the current reference frame in the event of (1) the subsequent detection of packet loss at the video receiver, or (2) detection of a bad packet at the video receiver.

9. The system of claim 8 wherein, having assigned the second quality designation to the current reference frame, the intra-frame detector is further configured to direct the feedback packet transmitter to transmit one or more additional feedback packets to the video transmitter to request transmission of a further intra-frame while the decoding of the plurality of video packets at the video receiver is paused.

10. The system of claim 9 wherein, upon receipt of the requested further intra-frame at the video receiver, the intra-frame detector is further configured to direct the feedback packet transmitter to stop transmitting of the one or more additional feedback packets to the video transmitter while the decoding of the plurality of video packets at the video receiver is resumed.

11. The system of claim 6 wherein the network packet processor is configured to remove, from the jitter buffer, one or more video packets arriving before the receipt of the requested intra-frame at the video receiver.

12. The system of claim 11 wherein the video receiver further includes frame constructor configured to construct one or more video frames from one or more video packets remaining at the jitter buffer subsequent to the removing of the one or more video packets from the jitter buffer.

13. The system of claim 6 wherein the network packet processor is configured to adjust a size of the jitter buffer based at least on one or more of a round trip delay between the video transmitter and the video receiver, and a target bitrate used by the video transmitter for encoding a video data stream.

14. The system of claim 6 wherein the feedback packet transmitter is configured to adjust a time interval for transmitting the one or more feedback packets to the video transmitter based at least on one or more of an available bandwidth from the video transmitter to the video receiver, a video frame size in use by the video receiver, and a packet loss ratio from the video transmitter.

15. A system for performing intra-frame refresh in multimedia communications over a packet communications network, comprising:

a video transmitter including a video encoder, a packetizer, and a packet transmitter,

wherein the video encoder is configured to encode a video data stream at a target bitrate,

wherein the packetizer is configured to generate a plurality of video packets corresponding to the encoded video data stream, and

wherein the packet transmitter is configured to transmit the plurality of video packets over the packet communications network at a video data transmit rate;

a video receiver including a video decoder, a network packet processor, and a packet parser,

wherein the network packet processor includes a jitter buffer, a missing packet detector, and a feedback packet transmitter,

wherein the video receiver is configured to receive the plurality of video packets in the jitter butter,

wherein the packet parser includes an intra-frame detector,

wherein the missing packet detector is configured to detect at least one eventual missing video packet among the plurality of video packets in the jitter buffer, the at least one eventual missing video packet corresponding to at least one missing packet not received at the video receiver after a predetermined wait time has passed;

wherein, upon detection of the eventual packet loss at the video receiver, the intra-frame detector is configured;

to detect an absence of at least one packet associated with an intra-frame in the jitter buffer after the at least one eventual missing video packet in the plurality of video packets in the jitter buffer is detected; and

to direct the feedback packet transmitter to transmit one or more feedback packets to the video transmitter to request transmission of an intra-frame while decoding of the plurality of video packets by the video decoder is paused; and

upon receipt of the requested intra-frame from the video transmitter, to direct the second feedback packet transmitter to stop transmitting the one or more feedback packets while the decoding of the plurality of video packets by the video decoder is resumed.

16. The system of claim 15 wherein the video transmitter further includes a bandwidth estimator and a bitrate controller, wherein the bandwidth estimator is configured to estimate an available bandwidth from the video transmitter to the video receiver, and wherein the bitrate controller is configured, in the event the available bandwidth is insufficient for transmitting the requested intra-frame to the video receiver, to reduce the target bitrate by a predetermined amount to allow the video encoder and the packetizer to generate the requested intra-frame from the encoded video data stream with fewer bytes of information.

17. The system of claim 15 wherein the video transmitter further includes a retransmission buffer, and wherein, in the event the eventual missing video packet is absent from the retransmission buffer, the video encoder and the packetizer are configured to generate a further intra-frame from the encoded video data stream for subsequent transmission to the video receiver.

18. The system of claim 15 wherein the video transmitter is configured to determine a time period since transmission of a last intra-frame by the packet transmitter to the video receiver, and wherein the packet transmitter is further configured to transmit a further intra-frame to the video receiver in the event the time period since the transmission of the last intra-frame satisfies one or more conditions based at least on a round trip delay between the video transmitter and the video receiver.