System and method for improved view layout management in scalable video and audio communication systems

• US 8,421,840 B2
• Filed: 06/09/2009
• Issued: 04/16/2013
• Est. Priority Date: 06/09/2008
• Status: Active Grant

• Current Assignee: Vidyo Incorporated (Enghouse Systems Limited)
• Original Assignee: Vidyo Incorporated (Enghouse Systems Limited)
• Inventors: Eleftheriadis, Alexandros, Adragna, James P.
• Primary Examiner(s): Gauthier, Gerald
• Application Number:
  US12/481,354
• Publication Number:
  US 20100002069A1
• Time in Patent Office: 1,407 Days
• Field of Search: 348/14.07, 348/14.08, 348/14.09, 348/14.16, 348/14.12, 375/240.08, 375/240.02, 375/240.081, 382/233, 382/239, 709/223, 709/231, 709/232, 709/204, 725/105
• US Class Current: 348/14.08
• CPC Class Codes: H04N 21/234327 by decomposing into layers, e.g. base layer and one or more enhancement layers
• CPC Class Codes: H04N 21/2662 Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 21/4314 for fitting data in a restricted space on the screen, e.g. EPG data in a rectangular grid
• CPC Class Codes: H04N 21/4622 Retrieving content or additional data from different sources, e.g. from a broadcast channel and the Internet web site content organization and management for information retrieval from the Internet G06F16/958; transmission by internet of broadcast information H04H60/82; stock exchange data over packet-switching network H04L12/1804; push services including data channel over packet-switching network H04L12/1859
• CPC Class Codes: H04N 21/4858 for modifying screen layout parameters, e.g. fonts, size of the windows
• CPC Class Codes: H04N 21/654 Transmission by server directed to the client
• CPC Class Codes: H04N 21/6587 Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 7/152 Multipoint control units therefor

Abstract

A system and method for transmitting a plurality of video signals scalably coded into layers including a base layer and one or more enhancement layers and associated audio signals, if any, over a communication network for presentation to one or more end users. A layout to display the plurality of video signals is determined based on a set of criteria and only the data of the video signal layers that are necessary for displaying the video signals in the determined layout, and any associated audio signals, is selectively transmitted over the communication network.

Claims

1. An audiovisual communication system for transmitting a plurality of video signals and associated signals, if any, over a communication network for presentation to one or more end users, wherein the video signals are scalably coded into layers including a base layer and one or more enhancement layers, the system comprising:

a receiver configured to receive the plurality of video and any associated audio signals over the communication network, determine a layout for the displayed video signals based on a set of criteria, and communicate information about the determined layout;

a display, connected to the receiver, configured to display one or more of the received video signals using the determined layout; and

one or more servers, connected to the communication network, adapted to receive the information about the determined layout, and configured to transmit the plurality of video and any associated audio signals over the communication network;

wherein the one or more servers are configured to selectively transmit, for each of the plurality of video signals, no layers, the base layer, or the base layer and one or more enhancement layers, if any, that are necessary for displaying the video signals in the determined layout;

wherein the one or more servers are further configured to determine which of the audio signals to selectively transmit by using audio activity indicators for each of the associated audio signals and an indication of whether the associated video signal is displayed in the determined layout, such that audio signal data is transmitted for one or more of the audio signals with the highest activity and audio signal data associated with one or more of the video signals that are displayed in the determined layout is transmitted, and no audio signal data is transmitted associated with the remaining video signals;

wherein the receiver is further configured to maintain for the determined layout the number of views N and the number of enlarged views P, where P ranges from 0 to N−

1, as well as the total number M of video signals each having an associated audio signal, and upon initialization or upon a change in N or P, determine a new layout and assign views to the new layout by;

sorting the M signals by decreasing audio activity into a list F;

if N or P changed, first assigning each video signal in F that was occupying a view in the determined layout to a view in the new layout that has the maximum overlap with the view that the video signal was assigned in the determined layout;

then assigning each video signal remaining unassigned in F sequentially into the next available view in the new layout until all video signals in F are processed or until all views in the new layout are assigned a video signal.

2. An audiovisual communication system for transmitting a plurality of video signals and associated audio signals, if any, over a communication network for presentation to one or more end users, wherein the video signals are scalably coded into layers including a base layer and one or more enhancement layers, the system comprising:

a receiver configured to receive the plurality of video and any associated audio signals over the communication network, determine a layout for the displayed video signals based on a set of criteria, and communicate information about the determined layout;

a display, connected to the receiver, configured to display one or more of the received video signals using the determined layout; and

one or more servers, connected to the communication network, adapted to receive the information about the determined layout, and configured to transmit the plurality of video and any associated audio signals over the communication network;

wherein the one or more servers are configured to selectively transmit, for each of the plurality of video signals, no layers, the base layer, or the base layer and one or more enhancement layers, if any, that are necessary for displaying the video signals in the determined layout;

wherein the one or more servers are further configured to determine which of the audio signals to selectively transmit by using audio activity indicators for each of the associated audio signals and an indication of whether the associated video signal is displayed in the determined layout, such that audio signal data is transmitted for one or more of the audio signals with the highest activity and audio signal data associated with one or more of the video signals that are displayed in the determined layout is transmitted, and no audio signal data is transmitted associated with the remaining video signals;

wherein the receiver is further configured to maintain for the determined layout the number of views N and the number of enlarged views P, where P ranges from 0 to N−

1, as well as the total number M of video signals each having an associated audio signal, and wherein at certain time instances or upon a change in M, reassign views to the determined layout by;

sorting the M video signals by decreasing associated audio activity into a list F;

sequentially assign the first P video signals in F such that;

if a video signal is already assigned an enlarged view it retains its current enlarged view,

otherwise if a replacing video signal was assigned a standard view it is assigned the enlarged view that was assigned to the video signal with the smallest associated audio activity among the video signals that were previously assigned an enlarged view, and the video signal with the smallest associated audio activity among the video signals that was previously assigned an enlarged view is assigned the replacing video signal'"'"'s original standard view,

otherwise if a video signal was not previously assigned an enlarged or standard view it is assigned an enlarged view that was assigned to the video signal with the smallest associated audio activity among the video signals that were previously assigned an enlarged view, whereas the video signal with the smallest associated audio activity among the video signals that were previously assigned an enlarged view is inserted in a list G;

sequentially assign the top max{N-P-|G|, 0} video signals in F to the next available free views in the current layout;

sequentially assign the video signals in G to the remaining available free views in the current layout.

3. An audiovisual communication system for transmitting a plurality of video signals and associated audio signals, if any, over a communication network for presentation to one or more end users, wherein the video signals are scalably coded into layers including a base layer and one or more enhancement layers, the system comprising:

a receiver configured to receive the plurality of video and any associated audio signals over the communication network, determine a layout for the displayed video signals based on a set of criteria, and communicate information about the determined layout;

a display, connected to the receiver, configured to display one or more of the received video signals using the determined layout; and

one or more servers, connected to the communication network, adapted to receive the information about the determined layout, and configured to transmit the plurality of video and any associated audio signals over the communication network;

wherein the one or more servers are configured to selectively transmit, for each of the plurality of video signals, no layers, the base layer, or the base layer and one or more enhancement layers, if any, that are necessary for displaying the video signals in the determined layout;

wherein the receiver is further configured to determine a layout for a target screen size by accessing a table of potential layouts, selecting one or more candidate layouts from the table of potential layouts using a first set of criteria, fitting each candidate layout in the target screen size, and selecting to use zero or one of the candidate layouts based on a second set of criteria;

wherein;

the table of potential layouts represents each layout as a set of parameters comprising at least the horizontal and vertical dimensions of the layout in units of fundamental view building blocks that form a two-dimensional array, and, for each view, a scale factor and the horizontal and vertical positioning of the top left corner of the view within the two-dimensional array of view building blocks;

the fitting each candidate layout in the target screen size is performed by selecting appropriate view building block width and height such that the layout fits the target screen size in at least one of the vertical or horizontal dimensions, and such that the resulting view building block aspect ratio is within a given range of minimum and maximum values; and

the second set of criteria includes scoring of each candidate layout by computing a scoring function, selecting the candidate layouts with the highest value of the scoring function, and computing a score using a filtering function if multiple candidate layouts are selected, thus identifying a single candidate layout, and using the single candidate layout only if the single candidate layout'"'"'s smallest view size is greater than a specified value.

4. The system of claim 3 wherein the scoring function of a layout with N views is a weighted average of the minimum view size of the P largest views and the minimum size of the remaining N-P views, where P may be 0.

5. The system of claim 4 wherein the filtering function includes an additional weighted term that represents an ad-hoc preference bias for a particular layout, and which is stored in each layout'"'"'s entry in the table of potential layouts.

6. The system of claim 3 wherein the filtering function is equal to the total size of all views of a layout.

7. A method for transmitting a plurality of video signals scalably coded into layers including a base layer and one or more enhancement layers and associated audio signals, if any, over a communication network for presentation to one or more end users, comprising:

determining a layout to display the plurality of video signals based on a set of criteria;

communicating information about the determined layout;

selectively transmitting only the data of the video signal layers that are necessary for displaying the video signals in the determined layout and any associated audio signals;

receiving the selectively transmitted data;

displaying the plurality of video signals utilizing the determined layout;

determining which of the audio signals to selectively transmit by using audio activity indicators for each of the associated audio signals and an indication of whether the associated video signal is displayed in the determined layout, such that audio signal data is transmitted for one or more of the audio signals with the highest activity and audio signal data associated with one or more of the video signals that are displayed in the determined layout is transmitted, and no audio signal data is transmitted associated with the remaining video signals; and

maintaining for the determined layout the number of views N and the number of enlarged views P, where P ranges from 0 to N−

1, as well as the total number M of video signals each having an associated audio signal, and wherein upon initialization or upon a change in N or P, determining a new layout and assigning views to the new layout by;

sorting the M signals by decreasing audio activity into a list F;

if N or P changed, first assigning each video signal in F that was occupying a view in the determined layout to the view in the new layout that has the maximum overlap with the view that the video signal was assigned to in the determined layout;

then assigning each video signal remaining unassigned in F sequentially into the next available view in the new layout until all video signals in F are processed or until all views in the new layout are assigned a video signal.

8. A method for transmitting a plurality of video signals scalably coded into layers including a base layer and one or more enhancement layers and associated audio signals, if any, over a communication network for presentation to one or more end users, comprising:

determining a layout to display the plurality of video signals based on a set of criteria;

communicating information about the determined layout;

selectively transmitting only the data of the video signal layers that are necessary for displaying the video signals in the determined layout and any associated audio signals;

receiving the selectively transmitted data;

displaying the plurality of video signals utilizing the determined layout;

determining which of the audio signals to selectively transmit by using audio activity indicators for each of the associated audio signals and an indication of whether the associated video signal is displayed in the determined layout, such that audio signal data is transmitted for one or more of the audio signals with the highest activity and audio signal data associated with one or more of the video signals that are displayed in the determined layout is transmitted, and no audio signal data is transmitted associated with the remaining video signals; and

maintaining for the determined layout the number of views N and the number of enlarged views P, where P ranges from 0 to N−

1, as well as the total number M of video signals each having an associated audio signal, and wherein at certain time instances or upon a change in M, reassigning views to a new layout by;

sorting the M video signals by decreasing associated audio activity into a list F;

sequentially assigning the first P video signals in F such that;

if a video signal is assigned an enlarged view in the determined layout, it retains its current enlarged view in the new layout,

otherwise if a replacing video signal was assigned a standard view in the determined layout, it is assigned the enlarged view in the new layout that was assigned to the video signal with the smallest associated audio activity among the video signals that were assigned an enlarged view in the determined layout, and the video signal with the smallest associated audio activity among the video signals that was assigned an enlarged view in the deter mined layout is assigned the replacing video signal'"'"'s standard view from the determined layout in the new layout,

otherwise if a video signal was not previously assigned an enlarged or standard view in the determined layout, it is assigned an enlarged view in the new layout that was assigned to the video signal with the smallest associated audio activity among the video signals that were previously assigned an enlarged view in the determined layout, whereas the video signal with the smallest associated audio activity among the video signals that were previously assigned an enlarged view in the determined layout is inserted in a list G;

sequentially assigning the top max{N-P-|G|, 0} video signals in F to the next available free views in the new layout;

sequentially assigning the video signals in G to the remaining available free views in the new layout.

9. A method for transmitting a plurality of video signals scalably coded into layers including a base layer and one or more enhancement layers and associated audio signals, if any, over a communication network for presentation to one or more end users, comprising:

determining a layout to display the plurality of video signals based on a set of criteria;

communicating information about the determined layout;

selectively transmitting only the data of the video signal layers that are necessary for displaying the video signals in the determined layout and any associated audio signals;

receiving the selectively transmitted data;

displaying the plurality of video signals utilizing the determined layout determining a layout for a target screen size by accessing a table of potential layouts, selecting one or more candidate layouts from the table of potential layouts using a first set of criteria, fitting each candidate layout in the target screen size, and selecting to use zero or one of the candidate layouts based on a second set of criteria;

determining a layout for a target screen size by accessing a table of potential layouts, selecting one or more candidate layouts from the table of potential layouts using a first set of criteria, fitting each candidate layout in the target screen size, and selecting to use zero or one of the candidate layouts based on a second set of criteria; and

representing each layout in the table of potential layouts as a set of parameters comprising at least the horizontal and vertical dimensions of the layout in units of fundamental view building blocks that form a two-dimensional array, and, for each view, a scale factor and the horizontal and vertical positioning of the top left corner of the view within the two-dimensional array of view building blocks;

wherein fitting each candidate layout in the target screen size comprises;

selecting appropriate view building block width and height such that the layout fits the target screen size in at least one of the vertical or horizontal dimensions, and such that the resulting view building block aspect ratio is within a given range of minimum and maximum values; and

wherein the second set of criteria includes scoring of each candidate layout by computing a scoring function, selecting the candidate layouts with the highest value of the scoring function, and computing a score using a filtering function if multiple candidate layouts are selected, thus identifying a single candidate layout, and using the single candidate layout only if the single candidate layer'"'"'s smallest view size is greater than a specified value.

10. The method of claim 9 wherein the scoring function of a layout with N views is a weighted average of the minimum view size of the P largest views and the minimum size of the remaining N-P views, where P may be 0.

11. The method of claim 10 wherein the filtering function includes an additional weighted term that represents an ad-hoc preference bias for a particular layout, and which is stored in each layout'"'"'s entry in the table of potential layouts.

12. The method of claim 9 wherein the filtering function is equal to the total size of all views of a layout.