System And Method For Improved View Layout Management In Scalable Video And Audio Communication Systems

• US 20100002069A1
• Filed: 06/09/2009
• Published: 01/07/2010
• Est. Priority Date: 06/09/2008
• Status: Active Grant

• Current Assignee: Vidyo Incorporated (Enghouse Systems Limited)
• Original Assignee: Vidyo Incorporated (Enghouse Systems Limited)
• Inventors: Adragna, James P., Eleftheriadis, Alexandros
• Granted Patent: US 8,421,840 B2
• Time in Patent Office: Days
• US Class Current: 348/14.08
• CPC Class Codes: H04N 21/234327 by decomposing into layers, e.g. base layer and one or more enhancement layers
• CPC Class Codes: H04N 21/2662 Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 21/4314 for fitting data in a restricted space on the screen, e.g. EPG data in a rectangular grid
• CPC Class Codes: H04N 21/4622 Retrieving content or additional data from different sources, e.g. from a broadcast channel and the Internet web site content organization and management for information retrieval from the Internet G06F16/958; transmission by internet of broadcast information H04H60/82; stock exchange data over packet-switching network H04L12/1804; push services including data channel over packet-switching network H04L12/1859
• CPC Class Codes: H04N 21/4858 for modifying screen layout parameters, e.g. fonts, size of the windows
• CPC Class Codes: H04N 21/654 Transmission by server directed to the client
• CPC Class Codes: H04N 21/6587 Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 7/152 Multipoint control units therefor

Abstract

A system and method for transmitting a plurality of video signals scalably coded into layers including a base layer and one or more enhancement layers and associated audio signals, if any, over a communication network for presentation to one or more end users. A layout to display the plurality of video signals is determined based on a set of criteria and only the data of the video signal layers that are necessary for displaying the video signals in the determined layout, and any associated audio signals, is selectively transmitted over the communication network.

Claims

1. An audiovisual communication system for transmitting a plurality of video signals and associated audio signals, if any, over a communication network for presentation to one or more end users, wherein the video signals are scalably coded into layers including a base layer and one or more enhancement layers, the system comprising:

a receiver configured to receive the plurality of video and any associated audio signals over the communication network, determine a layout for the displayed video signals based on a set of criteria, and communicate information about the determined layout;

a display, connected to the receiver, configured to display one or more of the received video signals using the determined layout; and

one or more servers, connected to the communication network, adapted to receive the information about the determined layout, and configured to transmit the plurality of video and any associated audio signals over the communication network;

wherein the one or more servers are configured to selectively transmit, for each of the plurality of video signals, no layers, the base layer, or the base layer and one or more enhancement layers, if any, that are necessary for displaying the video signals in the determined layout.

2. The system of claim 1 wherein at least one of the plurality of video and associated audio signals are precompressed, and wherein the one or more servers are further configured to access said precompressed video and associated audio signals from local or remote storage.

3. The system of claim 1 wherein at least one of the plurality of video and associated audio signals are compressed in real-time, and wherein the one or more servers are further configured to encode said real-time compressed video and associated audio signals from a video or audio source attached to the one or more servers.

4. The system of claim 1 wherein at least one of the one or more servers are further configured to obtain one or more of said plurality of video and associated audio signals from transmitting systems connected to the at least one of the one or more servers over the communication network.

5. The system of claim 1 wherein the associated audio signals are scalably coded into layers including a base layer and one or more enhancement layers, and wherein the one or more servers are further configured to selectively transmit only the data of the audio signal layers that are necessary for the determined layout.

6. The system of claim 1 wherein the one or more servers are further configured to determine which of the audio signals to selectively transmit by using audio activity indicators for each of the associated audio signals and an indication of whether the associated video signal is displayed in the determined layout, such that audio signal data is transmitted for one or more of the audio signals with the highest activity and audio signal data associated with one or more of the video signals that are displayed in the determined layout is transmitted, and no audio signal data is transmitted associated with the remaining video signals.

7. The system of claim 6 wherein the associated audio signals are scalably coded into layers including a base layer and one or more enhancement layers, and wherein the one or more servers are further configured to selectively transmit both base and enhancement layer audio signal data for one or more audio signals with the highest activity indicators, and base layer audio signal data for one or more of the audio signals associated with the one or more of the video signals that are displayed in the determined layout.

8. The system of claim 6 wherein the audio activity indicators are computed at the one or more servers.

9. The system of claim 6 wherein the audio activity indicators are precomputed and are available to the one or more servers in combination with the video signals and associated audio signals.

10. The system of claim 6 wherein the receiver is further configured to maintain for the determined layout the number of views N and the number of enlarged views P, where P ranges from 0 to N−

1, as well as the total number M of video signals each having an associated audio signal, and upon initialization or upon a change in N or P, determine a new layout and assign views to the new layout by;

sorting the M signals by decreasing audio activity into a list F;

if N or P changed, first assigning each video signal in F that was occupying a view in the determined layout to a view in the new layout that has the maximum overlap with the view that the video signal was assigned in the determined layout;

then assigning each video signal remaining unassigned in F sequentially into the next available view in the new layout until all video signals in F are processed or until all views in the new layout are assigned a video signal.

11. The system of claim 6 wherein the receiver is further configured to maintain for the determined layout the number of views N and the number of enlarged views P, where P ranges from 0 to N−

1, as well as the total number M of video signals each having an associated audio signal, and wherein at certain time instances or upon a change in M, reassign views to the determined layout by;

sorting the M video signals by decreasing associated audio activity into a list F;

sequentially assign the first P video signals in F such that;

if a video signal is already assigned an enlarged view it retains its current enlarged view,

otherwise if a replacing video signal was assigned a standard view it is assigned the enlarged view that was assigned to the video signal with the smallest associated audio activity among the video signals that were previously assigned an enlarged view, and the video signal with the smallest associated audio activity among the video signals that was previously assigned an enlarged view is assigned the replacing video signal'"'"'s original standard view,

otherwise if a video signal was not previously assigned an enlarged or standard view it is assigned an enlarged view that was assigned to the video signal with the smallest associated audio activity among the video signals that were previously assigned an enlarged view, whereas the video signal with the smallest associated audio activity among the video signals that were previously assigned an enlarged view is inserted in a list G;

sequentially assign the top max{N-P-|G|, 0} video signals in F to the next available free views in the current layout;

sequentially assign the video signals in G to the remaining available free views in the current layout.

12. The system of claim 1 wherein the receiver is further configured to determine a layout for a target screen size by accessing a table of potential layouts, selecting one or more candidate layouts from the table of potential layouts using a first set of criteria, fitting each candidate layout in the target screen size, and selecting to use zero or one of the candidate layouts based on a second set of criteria.

13. The system of claim 12 wherein the first set of criteria includes the total number of views available in the layout being equal to a number N.

14. The system of claim 12 wherein the first set of criteria includes the total number of views available in the layout being equal to a number N and a number of views being equal to a number P with a size that is bigger than the remaining N-P views.

15. The system of claim 12 wherein the first set of criteria includes having a layout with a total number of views less than or equal to the number of video signals M received by the receiver.

16. The system of claim 12 wherein:

the table of potential layouts represents each layout as a set of parameters comprising at least the horizontal and vertical dimensions of the layout in units of fundamental view building blocks that form a two-dimensional array, and, for each view, a scale factor and the horizontal and vertical positioning of the top left corner of the view within the two-dimensional array of view building blocks;

the fitting each candidate layout in the target screen size is performed by selecting appropriate view building block width and height such that the layout fits the target screen size in at least one of the vertical or horizontal dimensions, and such that the resulting view building block aspect ratio is within a given range of minimum and maximum values; and

the second set of criteria includes scoring of each candidate layout by computing a scoring function, selecting the candidate layouts with the highest value of the scoring function, and computing a score using a filtering function if multiple candidate layouts are selected, thus identifying a single candidate layout, and using the single candidate layout only if the single candidate layout'"'"'s smallest view size is greater than a specified value.

17. The system of claim 16 wherein the scoring function of a layout with N views is a weighted average of the minimum view size of the P largest views and the minimum size of the remaining N-P views, where P may be 0.

18. The system of claim 17 wherein the filtering function includes an additional weighted term that represents an ad-hoc preference bias for a particular layout, and which is stored in each layout'"'"'s entry in the table of potential layouts.

19. The system of claim 16 wherein the filtering function is equal to the total size of all views of a layout.

20. The system of claim 1 wherein the receiver is further configured to determine a layout such that the number of views in the determined layout produces an aggregate bit rate of the video signal received by the receiver that is within a limit.

21. The system of claim 1 wherein the receiver is further configured to determine a layout such that the number of views in the determined layout produces an aggregate number of pixels to be decoded that is within a limit.

22. The system of claim 1 wherein the receiver is further configured to determine a layout automatically based on the number of received video signals.

23. The system of claim 1 wherein the receiver is further configured to allow a user to pan one or more of the displayed plurality of video signals manually within an assigned view if the one or more of the displayed plurality of video signals is cropped within the assigned view in the determined layout.

24. The system of claim 1 wherein the receiver is further configured to automatically detect the presence and location of a subject'"'"'s head in one or more of the plurality of displayed video signals, if the one or more of the plurality of displayed video signals is cropped within an assigned view in the determined layout, and to pan the one or more displayed video signals within the assigned view such that the subject'"'"'s head is in full view.

25. A method for transmitting a plurality of video signals scalably coded into layers including a base layer and one or more enhancement layers and associated audio signals, if any, over a communication network for presentation to one or more end users, comprising:

determining a layout to display the plurality of video signals based on a set of criteria;

communicating information about the determined layout; and

selectively transmitting only the data of the video signal layers that are necessary for displaying the video signals in the determined layout and any associated audio signals;

receiving the selectively transmitted data; and

displaying the plurality of video signals utilizing the determined layout.

26. The method of claim 25 wherein at least one of the plurality of video and associated audio signals are precompressed, further comprising:

accessing said precompressed video and associated audio signals from local or remote storage.

27. The method of claim 25 wherein at least one of the plurality of video and associated audio signals are compressed in real-time, further comprising:

encoding said real-time compressed video and associated audio signals from a video or audio source.

28. The method of claim 25, further comprising:

obtaining one or more of said plurality of video and associated audio signals from transmitting systems.

29. The method of claim 25 wherein the associated audio signals are scalably coded into layers including a base layer and one or more enhancement layers, further comprising:

selectively transmitting only the data of the associated audio signal layers that are necessary for the determined layout.

30. The method of claim 25, further comprising:

determining which of the audio signals to selectively transmit by using audio activity indicators for each of the associated audio signals and an indication of whether the associated video signal is displayed in the determined layout, such that audio signal data is transmitted for one or more of the audio signals with the highest activity and audio signal data associated with one or more of the video signals that are displayed in the determined layout is transmitted, and no audio signal data is transmitted associated with the remaining video signals.

31. The method of claim 30 wherein the associated audio signals are scalably coded into layers including a base layer and one or more enhancement layers, further comprising:

selectively transmitting both base and enhancement layer audio signal data for one or more audio signals with the highest activity indicators, and base layer audio signal data for one or more of the audio signals associated with the one or more of the video signals that are displayed in the determined layout.

32. The method of claim 30, further comprising:

maintaining for the determined layout the number of views N and the number of enlarged views P, where P ranges from 0 to N−

1, as well as the total number M of video signals each having an associated audio signal, and wherein upon initialization or upon a change in N or P, determining a new layout and assigning views to the new layout by;

sorting the M signals by decreasing audio activity into a list F;

if N or P changed, first assigning each video signal in F that was occupying a view in the determined layout to the view in the new layout that has the maximum overlap with the view that the video signal was assigned to in the determined layout;

then assigning each video signal remaining unassigned in F sequentially into the next available view in the new layout until all video signals in F are processed or until all views in the new layout are assigned a video signal.

33. The method of claim 30, further comprising:

maintaining for the determined layout the number of views N and the number of enlarged views P, where P ranges from 0 to N−

1, as well as the total number M of video signals each having an associated audio signal, and wherein at certain time instances or upon a change in M, reassigning views to a new layout by;

sorting the M video signals by decreasing associated audio activity into a list F;

sequentially assigning the first P video signals in F such that;

if a video signal is assigned an enlarged view in the determined layout, it retains its current enlarged view in the new layout,

otherwise if a replacing video signal was assigned a standard view in the determined layout, it is assigned the enlarged view in the new layout that was assigned to the video signal with the smallest associated audio activity among the video signals that were assigned an enlarged view in the determined layout, and the video signal with the smallest associated audio activity among the video signals that was assigned an enlarged view in the determined layout is assigned the replacing video signal'"'"'s standard view from the determined layout in the new layout,

otherwise if a video signal was not previously assigned an enlarged or standard view in the determined layout, it is assigned an enlarged view in the new layout that was assigned to the video signal with the smallest associated audio activity among the video signals that were previously assigned an enlarged view in the determined layout, whereas the video signal with the smallest associated audio activity among the video signals that were previously assigned an enlarged view in the determined layout is inserted in a list G;

sequentially assigning the top max{N-P-|G|, 0} video signals in F to the next available free views in the new layout;

sequentially assigning the video signals in G to the remaining available free views in the new layout.

34. The method of claim 25, further comprising:

determining a layout for a target screen size by accessing a table of potential layouts, selecting one or more candidate layouts from the table of potential layouts using a first set of criteria, fitting each candidate layout in the target screen size, and selecting to use zero or one of the candidate layouts based on a second set of criteria.

35. The method of claim 34 wherein the first set of criteria includes the total number of views available in the layout being equal to a number N.

36. The method of claim 34 wherein the first set of criteria includes the total number of views available in the layout being equal to a number N and a number of views being equal to a number P with a size that is bigger than the remaining N-P views.

37. The method of claim 34 wherein the first set of criteria includes having a layout with a total number of views less than or equal to the number of video signals M received by the receiver.

38. The method of claim 34, further comprising:

representing each layout in the table of potential layouts as a set of parameters comprising at least the horizontal and vertical dimensions of the layout in units of fundamental view building blocks that form a two-dimensional array, and, for each view, a scale factor and the horizontal and vertical positioning of the top left corner of the view within the two-dimensional array of view building blocks;

wherein fitting each candidate layout in the target screen size comprises;

selecting appropriate view building block width and height such that the layout fits the target screen size in at least one of the vertical or horizontal dimensions, and such that the resulting view building block aspect ratio is within a given range of minimum and maximum values; and

wherein the second set of criteria includes scoring of each candidate layout by computing a scoring function, selecting the candidate layouts with the highest value of the scoring function, and computing a score using a filtering function if multiple candidate layouts are selected, thus identifying a single candidate layout, and using the single candidate layout only if the single candidate layer'"'"'s smallest view size is greater than a specified value.

39. The method of claim 38 wherein the scoring function of a layout with N views is a weighted average of the minimum view size of the P largest views and the minimum size of the remaining N-P views, where P may be 0.

40. The method of claim 39 wherein the filtering function includes an additional weighted term that represents an ad-hoc preference bias for a particular layout, and which is stored in each layout'"'"'s entry in the table of potential layouts.

41. The method of claim 38 wherein the filtering function is equal to the total size of all views of a layout.

42. The method of claim 25, further comprising:

determining a layout such that the number of views in the determined layout produces an aggregate bit rate of the received video signal that is within a limit.

43. The method of claim 25, further comprising:

determining a layout such that the number of views in the determined layout produces an aggregate number of pixels to be decoded that is within a limit.

44. The method of claim 25, further comprising:

determining a layout automatically based on the number of received signals.

45. The method of claim 25, further comprising:

allowing a user to pan one or more of the plurality of displayed video signals manually within an assigned view if the one or more of the plurality of displayed video signals is cropped within the assigned view in the determined layout.

46. The method of claim 25, further comprising:

automatically detecting the presence and location of a subject'"'"'s head in one or more of the plurality of displayed video signals if the one or more of the plurality of video signals is cropped within an assigned view in the determined layout; and

panning the one or more displayed video signals within the assigned view such that the subject'"'"'s head is in full view.

47. A system for presenting a plurality of video signals and associated audio signals, if any, received over a communication network from one or more servers, and wherein the video signals are scalably coded into layers including a base layer and one or more enhancement layers, comprising:

a receiver configured to;

receive the plurality of video and any associated audio signals from the communication network;

determine a layout for the display of one or more of the received video signals based on a set of criteria; and

communicate the determined layout over the communications network as feedback; and

a display, connected to the receiver, configured to display one or more of the received video signals using the determined layout;

wherein the feedback comprises instructions for the selective transmission of the video signal layers, if any, in the determined layout.

48. The system of claim 47 wherein the associated audio signals are scalably coded into layers including a base layer and one or more enhancement layers, and wherein the feedback further comprises instructions for the selective transmission of audio signal layers in the determined layout.

49. The system of claim 47 wherein the receiver is further configured to determine a layout for a target screen size by accessing a table of potential layouts, selecting one or more candidate layouts from the table of potential layouts using a first set of criteria, fitting each candidate layout in the target screen size, and selecting to use zero or one of the candidate layouts based on a second set of criteria.

50. A system for transmitting a plurality of video signals and one or more associated audio signals, wherein the video signals are scalably coded into layers including a base layer and one or more enhancement layers, comprising:

one or more servers configured to;

receive instructions for the selective transmission of the video signal layers corresponding to a determined layout; and

selectively transmit only the data of the video signal layers that are necessary for displaying the video signals in the determined layout.

51. The system of claim 50 wherein the one or more associated audio signals are scalably coded into layers including a base layer and one or more enhancement layers, and wherein the instructions further comprise instructions for the selective transmission of audio signal layers in the determined layout.

52. The system of claim 50 wherein the one or more associated audio signals are scalably coded into layers including a base layer and one or more enhancement layers, and wherein the one or more servers are further configured to:

selectively transmit both base and enhancement layer audio signal data for one or more audio signals with the highest activity indicators, and base layer audio signal data for the audio signals associated with the one or more of the video signals that are contained in the determined layout.