Network streaming of a video stream over multiple communication channels

• US 8,356,109 B2
• Filed: 05/13/2010
• Issued: 01/15/2013
• Est. Priority Date: 05/13/2010
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Martinez, Martin, Riggert, Eric
• Primary Examiner(s): DONAGHUE, LARRY D
• Application Number:
  US12/779,905
• Publication Number:
  US 20110283009A1
• Time in Patent Office: 978 Days
• Field of Search: 709/231, 709/232
• US Class Current: 709/231
• CPC Class Codes: H04N 21/6125 involving transmission via Internet transmission by internet of broadcast information H04H60/82
• CPC Class Codes: H04N 21/631 Multimode Transmission, e.g. transmitting basic layers and enhancement layers of the content over different transmission paths or transmitting with different error corrections, different keys or with different transmission protocols
• CPC Class Codes: H04N 21/64322 IP

Abstract

The present disclosure is directed to streaming a video from a sending endpoint to a receiving endpoint. Both of the sending endpoint and the receiving endpoint have multiple communication channels connecting the sending endpoint and the receiving endpoint to one or more networks, respectively, and the streaming video includes a plurality of intra-frame coded frames and a plurality of inter-frame coded frames. The sending endpoint sends different ones of the plurality of intra-frame coded frames and different ones of high priority inter-frame coded frames to the receiving endpoint over more than one of the multiple communication channels having a connection-oriented protocol. In addition, the sending endpoint sends different ones of non-high priority inter-frame coded frames to the receiving endpoint over more than one of the multiple communication channels having a connectionless-oriented protocol.

Claims

1. A method for streaming a video from a sending endpoint to a receiving endpoint, wherein both of the sending endpoint and the receiving endpoint have multiple communication channels connecting the sending endpoint and the receiving endpoint to one or more networks, respectively, wherein the streaming video includes a plurality of intra-frame coded frames and a plurality of inter-frame coded frames, wherein in the method the sending endpoint performs:

sending different ones of the plurality of intra-frame coded frames over more than one of the multiple communication channels having a connection-oriented protocol;

determining whether an inter-frame coded frame is or is not a high priority frame, wherein an inter-frame coded frame is a high priority frame if a number of references to the inter-frame coded frame by other inter-frame coded frames is more than a threshold number;

sending different ones of the plurality of inter-frame coded frames which are determined to be high priority frames over more than one of the multiple communication channels having a connection-oriented protocol; and

sending different ones of the plurality of inter-frame coded frames which are determined not to be high priority frames over more than one of the multiple communication channels having a connectionless-oriented protocol.

2. A method according to claim 1, wherein the sent intra-frame coded frames and inter-frame coded frames are received at the receiving endpoint, and recombined to reconstruct the video stream.

3. A method according to claim 1, wherein before the video stream is split into the plurality of intra-frame coded frames and the plurality of inter-frame coded frames, an amount of preloaded data including both intra-frame coded frames and inter-frame coded frames is sent over one of the multiple communication channels having a connection-oriented protocol.

4. A method according to claim 3, wherein the amount of preloaded data is determined based upon a size of a buffer included in the receiving endpoint and a playback rate of the video stream at the receiving endpoint.

5. A method according to claim 1, wherein a determination is made as to an expected difference in arrival time between intra-frame coded frames and/or high priority inter-frame coded frames sent concurrently over different ones of the multiple communication channels having a connection-oriented protocol, and a determination is made as to an expected difference in arrival time between non-high priority inter-frame coded frames sent concurrently over different ones of the multiple communication channels having a connectionless-oriented protocol,

wherein in the step of sending the intra-frame coded frames or high priority inter-frame coded frames, a next available intra-frame coded frame or a next available high priority inter-frame coded frame is sent over a fastest one of the multiple communication channels having a connection-oriented protocol, the fastest one of the multiple communication channels being determined based on the determined expected difference in arrival time,

wherein an intra-frame coded frame or a high priority inter-frame coded frame is extracted from further back in the plurality of intra-frame coded frames or plurality of inter-frame coded frames in accordance with the determined expected difference in arrival time, and the extracted intra-frame coded frame or high priority inter-frame coded frame is sent on a corresponding slower one of the multiple communication channels,

wherein the next available intra-frame coded frame or high priority inter-frame coded frame is sent from the sending endpoint nearly simultaneously as the extracted intra-frame coded frame or high priority inter-frame coded frame is sent from the sending endpoint,

wherein in the step of sending the non-high priority inter-frame coded frames, a next non-high priority inter-frame coded frame of the plurality of inter-frame coded frames is sent over a fastest one of the multiple communication channels having a connectionless-oriented protocol, the fastest one of the multiple communication channels being determined based on the determined expected difference in arrival time,

wherein a non-high priority inter-frame coded frame is extracted from further back in the plurality of inter-frame coded frames in accordance with the determined expected difference in arrival time, and the extracted non-high priority inter-frame coded frame is sent on a corresponding slower one of the multiple communication channels, and

wherein the next available non-high priority inter-frame coded frame is sent from the sending endpoint nearly simultaneously as the extracted non-high priority inter-frame coded frame is sent from the sending endpoint.

6. A method according to claim 5, wherein each determination is made using historical statistical data regarding performances of each of the communication channels.

7. A method according to claim 1, wherein each of the intra-frame coded frames and the high priority inter-frame coded frames are split into a series of data packets, each of the non-high priority inter-frame coded frames are split into a series of data packets, and when sending the frames over the multiple communication channels, different ones of the series of data packets are sent over the multiple communication channels.

8. A method according to claim 7, wherein a determination is made as to an expected difference in arrival time between data packets of the intra-frame coded frames and/or high priority inter-frame coded frames sent concurrently over different ones of the multiple communication channels having a connection-oriented protocol, and a determination is made as to an expected difference in arrival time between data packets of the non-high priority inter-frame coded frames sent concurrently over different ones of the multiple communication channels having a connectionless-oriented protocol,

wherein in the step of sending data packets of the intra-frame coded frames or high priority inter-frame coded frames, a next available data packet of the series of data packets of the intra-frame coded frames or high priority inter-frame coded frames is sent over a fastest one of the multiple communication channels having a connection-oriented protocol, the fastest one of the multiple communication channels being determined based on the determined expected difference in arrival time,

wherein a data packet of the series of data packets of the intra-frame coded frames or the high priority inter-frame coded frames is extracted from further back in the series of data packets in accordance with the determined expected difference in arrival time, and the extracted data packet is sent on a corresponding slower one of the multiple communication channels,

wherein the next available data packet of the series of data packets of the intra-frame coded frames or the high priority inter-frame coded frames is sent from the sending endpoint nearly simultaneously as the extracted data packet is sent from the sending endpoint,

wherein in the step of sending data packets of the non-high priority inter-frame coded frames, a next available data packet of the series of data packets of the non-high priority inter-frame coded frames is sent over a fastest one of the multiple communication channels having a connectionless-oriented protocol, the fastest one of the multiple communication channels being determined based on the determined expected difference in arrival time,

wherein a data packet of the series of data packets of the non-high priority inter-frame coded frames is extracted from further back in the series of data packets in accordance with the determined expected difference in arrival time, and the extracted data packet is sent on a corresponding slower one of the multiple communication channels, and

wherein the next available data packet is sent from the sending endpoint nearly simultaneously as the extracted data packet is sent from the sending endpoint.

9. A method according to claim 1, wherein the multiple communication channels having a connection-oriented protocol comprised of a first bondable virtual interface, and the multiple communication channels having a connectionless-oriented protocol comprised of a second bondable virtual interface,

wherein a determination is made as to an expected difference in arrival time for data sent concurrently over the first bondable virtual interface and the second bondable virtual interface,

wherein an amount of preloaded data including intra-frame coded frames and inter-frame coded frames is sent over a fastest one of the first bondable virtual interface and the second bondable virtual interface, the fastest one being determined based on the determined expected difference in arrival time,

wherein in a case that the first bondable virtual interface is the fastest, a next available intra-frame coded frame or high priority inter-frame coded frame is sent over the first bondable virtual interface, and a non-high priority inter-frame coded frame is extracted from further back in the video stream and sent over the second bondable virtual interface, the extracted non-high priority inter-frame coded frame being sent nearly simultaneously as the next available intra-frame coded frame or high priority inter-frame coded frame, and

wherein in a case that the second bondable virtual interface is the fastest, a next available non-high priority inter-frame coded frame is sent over the second bondable virtual interface, and an intra-frame coded frame or high priority inter-frame coded frame is extracted from further back in the video stream and sent over the first bondable virtual interface, the extracted intra-frame coded frame or high priority inter-frame coded frame being sent nearly simultaneously as the next available non-high priority inter-frame coded frame.

10. A method according to claim 1, wherein the threshold number is set by a user.

11. A method according to claim 1, wherein the threshold number is determined based upon a number of frames that are dropped before a quality of service (QoS) is reduced such that the reduction in QoS is noticeable to a viewer.

12. A method according to claim 1, wherein in a case that an inter-frame coded frame is not received by the receiving endpoint, feedback is sent from the receiving endpoint to the sending endpoint indicating the inter-frame coded frame that was not received, and the inter-frame coded frame is re-sent over one of the multiple communication channels having a connection-oriented protocol.

13. The method according to claim 1, wherein the multiple communication channels are carried over multiple physical interfaces connecting the sending endpoint and the receiving endpoint to one or more networks, respectively.

14. The method according to claim 1, wherein the multiple communication channels are carried over a single physical interface.

15. A method for streaming a video from a sending endpoint to a receiving endpoint, wherein both of the sending endpoint and the receiving endpoint have multiple communication channels connecting the sending endpoint and the receiving endpoint to one or more networks, respectively, wherein the streaming video includes a plurality of intra-frame coded frames and a plurality of inter-frame coded frames, wherein in the method the receiving endpoint performs:

receiving intra-frame coded frames and inter-frame coded frames sent from the sending endpoint; and

recombining the intra-frame coded frames and inter-frame coded frames to reconstruct the video stream,

wherein different ones of the plurality of intra-frame coded frames are sent by the sending endpoint and received by the receiving endpoint over more than one of the multiple communication channels having a connection-oriented protocol,

wherein a determination is made as to whether an inter-frame coded frame is or is not a high priority frame, wherein an inter-frame coded frame is a high priority frame if a number of references to the inter-frame coded frame by other inter-frame coded frames is more than a threshold number,

wherein different ones of the plurality of inter-frame coded frames which are determined to be high priority frames are sent by the sending endpoint and received by the receiving endpoint over more than one of the multiple communication channels having a connection-oriented protocol, and

wherein different ones of the plurality of inter-frame coded frames which are determined not to be high priority frames are sent by the sending endpoint and received by the receiving endpoint over more than one of the multiple communication channels having a connectionless-oriented protocol.

16. A method according to claim 15, wherein before the split video stream is received by the receiving endpoint, an amount of preloaded data including both intra-frame coded frames and inter-frame coded frames is received over one of the multiple communication channels having a connection-oriented protocol.

17. A method according to claim 16, wherein the amount of preloaded data is determined based upon a size of a buffer included in the receiving endpoint and a playback rate of the video stream at the receiving endpoint.

18. A method according to claim 15, wherein each of the intra-frame coded frames and the high priority inter-frame coded frames are split into a series of data packets by the sending endpoint, each of the non-high priority inter-frame coded frames are split into a series of data packets by the sending endpoint, and when receiving the frames over the multiple communication channels, different ones of the series of data packets are received by the receiving endpoint over the multiple communication channels, and are recombined by the receiving endpoint into the intra-frame coded frames and the inter-frame coded frames.

19. A method according to claim 15, wherein the threshold number is set by a user.

20. A method according to claim 15, wherein the threshold number is determined based upon a number of frames that are dropped before a quality of service (QoS) is reduced such that the reduction in QoS is noticeable to a viewer.

21. A method according to claim 15, wherein in a case that an inter-frame coded frame is not received by the receiving endpoint, the receiving endpoint sends feedback to the sending endpoint indicating the inter-frame coded frame that was not received, and the inter-frame coded frame is re-sent over one of the multiple communication channels having a connection-oriented protocol.

22. The method according to claim 15, wherein the multiple communication channels are carried over multiple physical interfaces connecting the sending endpoint and the receiving endpoint to one or more networks, respectively.

23. The method according to claim 15, wherein the multiple communication channels are carried over a single physical interface.

24. A sending endpoint comprising:

an interface to multiple communication channels connecting the sending endpoint and a receiving endpoint to one or more networks;

a computer-readable memory constructed to store computer-executable process steps; and

a processor constructed to execute the computer-executable steps stored in the memory,

wherein the process steps stored in the memory cause the processor to stream a video from the sending endpoint to the receiving endpoint, wherein the streaming video includes a plurality of intra-frame coded frames and a plurality of inter-frame coded frames, and wherein the process steps stored in the memory include computer-executable process steps to;

send different ones of the plurality of intra-frame coded frames over more than one of the multiple communication channels having a connection-oriented protocol;

determine whether an inter-frame coded frame is or is not a high priority frame, wherein an inter-frame coded frame is a high priority frame if a number of references to the inter-frame coded frame by other inter-frame coded frames is more than a threshold number;

send different ones of the plurality of inter-frame coded frames which are determined to be high priority frames over more than one of the multiple communication channels having a connection-oriented protocol; and

send different ones of the plurality of inter-frame coded frames which are determined not to be high priority frames over more than one of the multiple communication channels having a connectionless-oriented protocol.

25. A receiving endpoint comprising:

an interface to multiple communication channels connecting a sending endpoint and the receiving endpoint to one or more networks;

a computer-readable memory constructed to store computer-executable process steps; and

a processor constructed to execute the computer-executable steps stored in the memory,

wherein the process steps stored in the memory cause the processor to stream a video from the sending endpoint to the receiving endpoint, wherein the streaming video includes a plurality of intra-frame coded frames and a plurality of inter-frame coded frames, and wherein the process steps stored in the memory include computer-executable process steps to;

receive intra-frame coded frames and inter-frame coded frames sent from the sending endpoint; and

recombine the intra-frame coded frames and inter-frame coded frames to reconstruct the video stream,

wherein different ones of the plurality of intra-frame coded frames are sent by the sending endpoint and received by the receiving endpoint over more than one of the multiple communication channels having a connection-oriented protocol,

wherein a determination is made as to whether an inter-frame coded frame is or is not a high priority frame, wherein an inter-frame coded frame is a high priority frame if a number of references to the inter-frame coded frame by other inter-frame coded frames is more than a threshold number,

wherein different ones of the plurality of inter-frame coded frames which are determined to be high priority frames are sent by the sending endpoint and received by the receiving endpoint over more than one of the multiple communication channels having a connection-oriented protocol, and

wherein different ones of the plurality of inter-frame coded frames which are determined not to be high priority frames are sent by the sending endpoint and received by the receiving endpoint over more than one of the multiple communication channels having a connectionless-oriented protocol.

26. A computer-readable memory medium on which is stored computer-executable process steps for causing a processor to stream a video from a sending endpoint to a receiving endpoint, wherein both of the sending endpoint and the receiving endpoint have multiple communication channels connecting the sending endpoint and the receiving endpoint to one or more networks, respectively, wherein the streaming video includes a plurality of intra-frame coded frames and a plurality of inter-frame coded frames, said processing steps comprising:

sending different ones of the plurality of intra-frame coded frames over more than one of the multiple communication channels having a connection-oriented protocol;

determining whether an inter-frame coded frame is or is not a high priority frame, wherein an inter-frame coded frame is a high priority frame if a number of references to the inter-frame coded frame by other inter-frame coded frames is more than a threshold number;

sending different ones of the plurality of inter-frame coded frames which are determined to be high priority frames over more than one of the multiple communication channels having a connection-oriented protocol; and

sending different ones of the plurality of inter-frame coded frames which are determined not to be high priority frames over more than one of the multiple communication channels having a connectionless-oriented protocol.

27. A computer-readable memory medium on which is stored computer-executable process steps for causing a processor to streaming a video from a sending endpoint to a receiving endpoint, wherein both of the sending endpoint and the receiving endpoint have multiple communication channels connecting the sending endpoint and the receiving endpoint to one or more networks, respectively, wherein the streaming video includes a plurality of intra-frame coded frames and a plurality of inter-frame coded frames, said processing steps comprising:

receiving intra-frame coded frames and inter-frame coded frames sent from the sending endpoint; and

recombining the intra-frame coded frames and inter-frame coded frames to reconstruct the video stream,

wherein different ones of the plurality of intra-frame coded frames are sent by the sending endpoint and received by the receiving endpoint over more than one of the multiple communication channels having a connection-oriented protocol,

wherein a determination is made as to whether an inter-frame coded frame is or is not a high priority frame, wherein an inter-frame coded frame is a high priority frame if a number of references to the inter-frame coded frame by other inter-frame coded frames is more than a threshold number,

wherein different ones of the plurality of inter-frame coded frames which are determined to be high priority frames are sent by the sending endpoint and received by the receiving endpoint over more than one of the multiple communication channels having a connection-oriented protocol, and

wherein different ones of the plurality of inter-frame coded frames which are determined not to be high priority frames are sent by the sending endpoint and received by the receiving endpoint over more than one of the multiple communication channels having a connectionless-oriented protocol.