Intelligent buffering process for network conference video

• US 7,843,818 B2
• Filed: 02/17/2006
• Issued: 11/30/2010
• Est. Priority Date: 11/14/2001
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Chu, Lon-Chan, Collins, Leonard Alan, Ben-Shachar, Ido M., Xu, Yikang, Giloi, Claus T., Leichtling, Ivan J.
• Primary Examiner(s): Wong; Warner
• Application Number:
  US11/356,935
• Publication Number:
  US 20060153217A1
• Time in Patent Office: 1,747 Days
• Field of Search: 370/235, 370/412, 370/428, 370/429, 348/423, 348/500, 348/412, 725/94, 725/144, 725/151
• US Class Current: 370/230
• CPC Class Codes: H04L 12/1827 Network arrangements for conference optimisation or adaptation
• CPC Class Codes: H04L 47/2416 Real-time traffic
• CPC Class Codes: H04L 47/32 by discarding or delaying data units, e.g. packets or frames
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/4038 with floor control
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04N 21/23406 involving management of server-side video buffer
• CPC Class Codes: H04N 21/44004 involving video buffer management, e.g. video decoder buffer or video display buffer
• CPC Class Codes: H04N 7/152 Multipoint control units therefor

Abstract

In order to minimize the delay of the video images viewed by a network conference attendee, an intelligent buffering process (IB process) selectively discards video frames from at least one point, and in an embodiment, many network points, such as at “in” and/or “out” buffers of clients, servers, routers, etc. Packets of video frame data arrive at a buffer, which can fill to a predetermined limit if the packets cannot be forwarded due to delays or slow connections. To forward the most current video images, old frames in the buffer are discarded rather than forwarded. In a particular embodiment, when the buffer is full, the next arriving delta frame packet is discarded. To avoid distortion, each subsequent delta frame is discarded until a new key frame eventually arrives. If the buffer is still full when the key frame arrives, the buffer is purged and the new key frame is added.

Claims

1. An intelligent buffering process to reduce lag in a conference video stream after a data forwarding delay, the stream including a plurality of packets defining either a key frame or a delta frame, the process comprising the steps of:

providing a plurality of buffers at a conference server, each of the buffers corresponding to a respective attendee destination;

receiving a newly arrived packet;

for each of the plurality of buffers, determining, at a computer, whether the newly arrived packet is a key frame, wherein the key frame contains data to construct an entire frame image; and

if the packet is a key frame;

detecting whether a buffer has reached a predetermined limit and discarding all of a number of currently buffered packets in the buffer if the predetermined limit has been reached; and

adding the newly arrived packet as the only packet in the buffer;

determining that the newly arrived packet is a delta frame;

automatically discarding the delta frame when the buffer contents have reached the predetermined limit;

forwarding at least one packet from the buffer to reduce the buffer contents below the predetermined limit; and

upon forwarding the at least one packet from the buffer to reduce the buffer contents below the predetermined limit, preventing distortion by continuing to discard subsequent consecutively arriving delta frames until a new key frame has been added to the buffer, wherein the new key frame prevents distortion by containing the data to construct an entire frame image, and wherein the delta frames cause the distortion by being encrypted to contain data representing only changes relative to an immediately preceding key frame or delta frame.

2. The intelligent buffering process of claim 1, wherein the at least one packet is forwarded from the buffer in the order received.

3. The intelligent buffering process of claim 2, wherein the forwarding step occurs as soon as downstream data traffic permits.

4. The intelligent buffering process of claim 2, wherein both the receiving step and forwarding step are executed above a network protocol level.

5. The intelligent buffering process of claim 2, wherein after the forwarding step the process includes transmitting the packets over a network.

6. The intelligent buffering process of claim 1, wherein the receiving step includes receiving packets sent from a conference attendee computer over the network.

7. An intelligent buffering process to reduce lag in a conference video stream after a data forwarding delay, the stream including a plurality of packets defining either a key frame or a delta frame, the process comprising the steps of:

providing a plurality of buffers at a conference server, each of the buffers corresponding to a respective attendee destination;

receiving a newly arrived packet;

for each of the plurality of buffers, determining, at a computer, whether the newly arrived packet is a delta frame, and if so;

detecting whether the previous packet was discarded, and, if so, discarding the newly arrived delta frame if the previous packet was discarded;

detecting whether a buffer has reached a predetermined limit; and

discarding the newly arrived packet if the buffer has reached the predetermined limit;

forwarding at least one packet from the buffer to reduce the buffer below the predetermined limit; and

upon forwarding the at least one packet from the buffer to reduce the buffer below the predetermined limit, preventing distortion by continuing to discard subsequent consecutively arriving delta frames until a new key frame has been added to the buffer, wherein the new key frame prevents distortion by containing the data to construct an entire frame image, and wherein the delta frames cause the distortion by being encrypted to contain data representing only changes relative to an immediately preceding key frame or delta frame.

8. The intelligent buffering process of claim 7, further comprising the step of adding the newly arrived delta frame packet to the buffer if the previous frame was not discarded and if the buffer has not reached the predetermined limit.

9. The intelligent buffering process of claim 7, further comprising:

deleting packets from the buffer if the newly arrived packet is a key frame, and if the buffer has reached the predetermined limit; and

adding the key frame to the buffer, wherein the key frame contains data to construct an entire frame image.

10. A process for managing a video conference among a plurality of attendee computers, the process comprising:

providing a plurality of buffers at a conference server, each of the buffers corresponding to a respective attendee destination;

receiving a newly arrived packet of a video stream, the packet defining respective key frames and delta frames, each of the key frames containing data to construct an entire frame image;

determining whether the newly arrived packet is a key frame;

if the packet is a key frame;

discarding all of a number of currently buffered packets in the buffer if the buffer contents have reached a predetermined limit;

adding the key frame as the only packet in the buffer;

forwarding non-discarded packets from the buffer to the corresponding attendee destination;

determining that the newly arrived packet is a delta frame;

automatically discarding the delta frame when the buffer contents have reached the predetermined limit;

forwarding at least one packet from the buffer to reduce the buffer contents below the predetermined limit; and

upon forwarding the at least one packet from the buffer to reduce the buffer contents below the predetermined limit, preventing distortion by continuing to discard subsequent consecutively arriving delta frames until a new key frame has been added to the buffer, wherein the new key frame prevents distortion by containing the data to construct an entire frame image, and wherein the delta frames cause the distortion by being encrypted to contain data representing only changes relative to an immediately preceding key frame or delta frame.

11. The process of claim 10, wherein after the forwarding of the non-discarded packets from the buffer to the corresponding attendee destination, the process includes transmitting the packets over a network to an attendee destination according to a verified-delivery protocol.

12. The process of claim 11, wherein the protocol is TCP/IP.

13. The process of claim 10, wherein the process further comprises providing at least one buffer at at least one of the attendee computers, said receiving, determining, discarding, and adding steps being performed by the attendee computer on the packets forwarded from the server.

14. The process of claim 10, wherein the process further comprises providing at least one buffer at at least one of the attendee computers, the attendee computer performing said receiving, determining, discarding, adding and forwarding steps on the packets prior to the receiving of packets by the server.

15. A non-signal computer-readable medium having computer-executable instructions for performing a process for managing a buffer, the process comprising:

providing a plurality of buffers at a conference server, each of the buffers corresponding to a respective attendee destination;

receiving packets of a video stream data, the packets defining a plurality of key frames and delta frames, each of the key frames containing data to construct an entire frame image;

for each of the plurality of buffers, determining whether a newly arrived packet contains a key frame; and

if the packet is a key frame;

detecting whether the buffer contents have reached a predetermined limit and discarding all of a number of currently buffered packets in the buffer if the predetermined limit has been reached;

adding the key frame as the only packet in the buffer,

determining that the newly arrived packet is a delta frame;

automatically discarding the delta frame when the buffer contents have reached the predetermined limit;

forwarding at least one packet from the buffer to reduce the buffer contents below the predetermined limit; and

upon forwarding the at least one packet from the buffer to reduce the buffer contents below the predetermined limit, preventing distortion by continuing to discard subsequent consecutively arriving delta frames until a new key frame has been added to the buffer, wherein the new key frame prevents distortion by containing the data to construct an entire frame image, and wherein the delta frames cause the distortion by being encrypted to contain data representing only changes relative to an immediately preceding key frame or delta frame.