Distributed presentations employing inputs from multiple video cameras located at multiple sites and customizable display screen configurations

• US 20070118868A1
• Filed: 11/23/2005
• Published: 05/24/2007
• Est. Priority Date: 11/23/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Rui, Yong, Zhang, Cha, Yu, Bin
• Granted Patent: US 7,589,760 B2
• Time in Patent Office: Days
• US Class Current: 725/105
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4316 for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A computer network-based distributed presentation system and process is presented that controls the display of one or more video streams output by multiple video cameras located across multiple presentation sites on display screens located at each presentation site. The distributed presentation system and process provides the ability for a user at a site to customize the screen configuration (i.e., what video streams are display at any one time and in what format) for that site via a two-layer display director module. In the design layer of the module, a user interface is provided for a user to specify display priorities dictating what video streams are to be displayed on the screen over time. These display priorities are then provided to the execution layer of the module which translates them into probabilistic timed automata and uses the automata to control what is displayed on the display screen.

Claims

1. A computer network-based distributed presentation system for displaying one or more video streams output by multiple video cameras located at multiple presentation sites, comprising:

a general purpose computing device located at each of the presentation sites;

a display screen located at each presentation site which is capable of simultaneously displaying one or more video streams in display sectors of the screen and which is controlled by said computing device;

at least one video camera located at one or more of the presentation sites, each of which outputs a video stream comprising views of the presentation site where the camera is located to the computer network;

a computer program comprising a display director program module executable by the computing device located at each presentation site, wherein the computing device is directed by the display director module to,

determine on an on-going basis which of the video streams output to the computer network are to be downloaded for display on the screen at the presentation site where the display director is running and select what sector of the screen the downloaded video stream or streams are to be displayed;

controlling the display screen so as to display the downloaded video stream or streams in the selected sectors.

2. The system of claim 1, wherein video stream locating data which allows a display director module to find and download a video stream output to the computer network from a video camera located at the same presentation site is provided to that display director module via a computing device controlling the video camera, and wherein each display director module is in communication with the other display director modules at the other presentation sites via the computer network and wherein the display director module further directs the local computing device to provide all video stream locating data received to the other display director modules so that each display director module can find and download for display on the local screen, videos streams output by any of the video cameras at any of the presentation sites.

3. The system of claim 2, wherein each display director module maintains a record of the current view being output by each camera across all the sites and whenever a video camera reports to the display director module at the same presentation site via the computing device controlling the video camera that it has changed the view it is capturing, the display director updates said record of current views and forward the view change information to the display director modules associated with other sites via the computer network for use in updating the record of view information maintained by each of those other display director modules.

4. A computer-implemented two-layer display director process for controlling the display of multiple video streams output to a computer network from video cameras located at multiple distributed sites on a display screen which is capable of simultaneously displaying one or more video streams in display sectors of the screen, comprising using a computer to perform the following process actions:

in the first layer, providing a user interface for a user to specify display priorities dictating what video streams are to be displayed on the screen over time, and providing the display priorities to the second layer of the display director process; and

in the second layer, translating the user-specified display priorities into probabilistic times automata and using the automata to control what is displayed on the display screen over time and where on the screen it is displayed.

5. The process of claim 4, wherein the process action of providing a user interface for the user to specify display priorities, comprises an action of providing as part of the user interface a provision for the user to enter camera view importance specifications which allows the user to specify for each view that can be output by any video camera across all the sites a level of importance indicative of the priority that view is to be given by the display director in deciding which views are to be displayed on the screen.

6. The process of claim 4, wherein the process action of providing a user interface for the user to specify display priorities, comprises an action of providing as part of the user interface a provision for the user to enter camera view importance specifications which allows the user to specify for each view that can be output by any video camera across all the sites any one of three progressive levels of importance indicative of the priority that view is to be given by the display director in deciding which views are to be displayed on the screen, wherein the highest of these levels indicates that a view is to be displayed most of the time.

7. The process of claim 4, wherein the process action of providing a user interface for the user to specify display priorities, comprises an action of providing as part of the user interface a provision for the user to enter timing constraints which allows the user to specify for each view that can be output by any video camera across all the sites a minimum time which the view is to be displayed on the screen and a maximum time the view should be displayed on the screen barring other display priorities.

8. The process of claim 7, wherein the process action of providing a user interface for the user to specify display priorities, comprises an action of providing as part of the user interface a provision for the user to enter transition probabilities which allows the user to specify for each view that can be output by any video camera across all the sites, a probability value that dictates the percentage of time that a view which has been displayed for its user-specified maximum time will be replaced with a different view.

9. The process of claim 4, wherein the process action of providing a user interface for the user to specify display priorities, comprises an action of providing as part of the user interface a provision for the user to enter a video composition selection which allows the user to specify which of a prescribed number of screen layout formats the display director is to use in displaying views output by the video cameras.

10. The process of claim 4, wherein the process action of providing a user interface for the user to specify display priorities, comprises an action of providing as part of the user interface a provision for the user to enter a transition idiom which allows the user to specify that if a current video camera view has been view X for over m seconds, and the last video camera view was Y, then view X is to be replaced with a view currently being captured by video camera Z for at least n seconds.

11. The process of claim 10, wherein view X can be specified as any video camera'"'"'s current view, and video camera Z can be specified as any video camera at any of the sites.

12. The process of claim 4, wherein the process action of translating the user-specified display priorities into probabilistic times automata comprises generating a series of state and transition tuples, wherein each state tuple describes a display configuration comprising what views are displayed, where they are displayed on the display screen and for how long, and what views are currently available from the various cameras at all the sites, and wherein each state tuple S takes the form S=<

View_List, Selection_List, SL, Max, Min>

, where the View_List field comprises a list of the possible views combinations of the video cameras present across all the sites, the Selection_List field comprises a list of the cameras whose views are found in the View_List field and whose outputs are selected for display on the display screen, the SL field specifies the screen configuration layout to be used in displaying the views output by the selected video cameras, and Max and Min fields represent the maximum and minimum time specified for displaying the screen configuration defined by the state tuple, and wherein each transition tuple specifies the conditions under which a displayed screen configuration is to be changed to another screen configuration, and wherein each transition tuple T takes the form T=<

Type, S_last, S_current, C_List, V_List, D, P, S_next>

, where the Type field is either “

Time”

or “

View”

, S_last is the state tuple defining the screen configuration displayed just before the currently displayed screen configuration defined by state tuple identified in S_current, S_next is the state tuple defining the screen configuration that is to be transitioned to after S_current, D refers to one of various time periods, P refers to the transition probability from current state S_current to next state S_next, the V_List field comprises a list of one or more views listed in the View_List field of a state tuple that remain after those views that are common with another state tuple are eliminated, and the C_List field comprises a list of the camera or cameras that output the view or views found in the V_List.

13. The process of claim 12, wherein the process action of generating the series of state and transition tuples, comprises the actions of:

generating a normal state tuple and a random state tuple for each possible combination of views that can be output at any one time by all the video cameras across all the sites, wherein a normal state tuple defines a screen configuration where all the views currently output by the cameras listed in the Selection_List field are displayed on the display screen, and wherein a random state tuple defines a screen configuration where one or more view listed in the View_List field is displayed on the display screen;

generating a view-type transition tuple for each state tuple in each possible pair of state tuples that exhibit at least one view that is different between their View_List fields, wherein a view-type tuple specifies a transition that occurs if the last screen configuration displayed before the current screen configuration was defined by a state tuple listed in the S_last field and the current screen configuration is defined by a state tuple listed in the S_current field and the view or views associated with the S_current state tuple have been displayed for at least D seconds, then when the video camera or cameras listed in the in the C_List field change the view or views being output to those listed in the V_List field, the screen configuration is changed to that defined by the state tuple listed in the S_next field.

generating two time-type transition tuples for each possible pair of normal and random state tuples that have the identical views listed in their View_List fields, one of said time-type transition tuples specifying a transition from the screen configuration defined by the normal state tuple of a pair to that defined by the random state tuple of the pair and the other of said time-type transition tuples specifying a transition from the screen configuration defined by the random state tuple of a pair to that defined by the normal state tuple of the pair, wherein a time-type tuple specifies a transition that occurs if the last screen configuration displayed before the current screen configuration was defined by a state tuple listed in the S_last field and the current screen configuration is defined by a state tuple listed in the S_current field and the view or views associated with the S_current state tuple have been displayed for at least the time period specified in the D field, the screen configuration is changed to that defined by the state tuple listed in the S_next field if not precluded by the transition probability listed in the P field.

14. The process of claim 13, wherein the process action of generating a normal state tuple and a random state tuple for each possible combination of views that can be output at any one time by all the video cameras across all the sites, comprises the actions of:

identifying every possible combination of views that can be output by the video cameras at all the sites at any one time; and

for each identified view combination,

identifying those views in the view combination under consideration that should be displayed based on the user-specified display priorities,

listing the video cameras responsible for outputting the identified views that should be displayed in the Selection_List field of the normal state tuple being generated,

setting the Selection_List field of the random state tuple being generated to “

random”

,

listing the views of the view combination under consideration in the View_List field of both the normal and random state tuple being generated,

setting the SL field of the normal state tuple being generated based on a screen layout format specified by the user,

setting the SL field of the random state tuple being generated so that a prescribed number of views from the View_List field will be the only views shown on the screen,

leaving the Min field of the random state tuple being generated open so that whenever the random state tuple being generated is employed the Min field is set to reflect the largest minimum time constraint specified by the user among randomly chosen view or views listed in the View_List field of the random state tuple that are equal in number to said prescribed number of views,

setting the Min field of the normal state tuple being generated to reflect the largest minimum time constraint specified by the user among the views output by the video cameras listed in the Selection_List field of the normal state tuple, and

setting the Max field of the normal state tuple being generated to reflect the smallest maximum time constraint specified by the user among the views output by the video cameras listed in the Selection_List field of the normal state tuple.

15. The process of claim 13, wherein the process action of generating a view-type transition tuple for each state tuple in each possible pair of state tuples that exhibit at least one view that is different between their View_List fields, comprises the actions of:

identifying all the possible state tuple pairs having at least one difference in their View_List fields; and

for each state tuple of each of the identified state tuple pairs,

listing the view or views found in the View_List field of the state tuple under consideration that are not found in the View_List field of the other state tuple of the pair in the V_List field of the view-type transition tuple being generated,

listing the camera or cameras that are responsible for outputting the view or views in the V_List field in the C_List field of the view-type transition tuple being generated,

setting the Type field of the view-type transition tuple being generated to “

View”

,

setting the S_last field of the view-type transition tuple being generated to identify the state tuple that defines the screen configuration that is to be shown before the screen configuration defined by the state tuple under consideration,

setting the S_current field of the view-type transition tuple being generated to identify the state tuple under consideration,

setting the S_next field of the view-type transition tuple being generated to identify the state tuple that defines the screen configuration that is to be shown after the screen configuration defined by the state tuple under consideration if the transition tuple being generated is executed, and

setting the D field of the view-type transition tuple being generated to the value of the Min field of the state tuple under consideration.

16. The process of claim 13, wherein the process action of generating two time-type transition tuples for each possible pair of normal and random state tuples that have the identical views listed in their View_List fields, comprises the actions of:

identifying pairs of normal and random state tuples that have the identical views listed in their View_List fields;

for each normal state tuple of each of the identified pairs of normal and random state tuples,

setting the Type field of the first time-type transition tuple being generated to “

Time”

,

setting the S_last field of the first time-type transition tuple being generated to identify the state tuple that defines the screen configuration that is to be shown before the screen configuration defined by the normal state tuple under consideration,

setting the S_current field of the first time-type transition tuple being generated to identify the normal state tuple under consideration,

setting the S_next field of the first time-type transition tuple being generated to identify the random state tuple that defines the screen configuration that is to be shown after the screen configuration defined by the normal state tuple under consideration if the first transition tuple being generated is executed,

setting the P field of the first time-type transition tuple being generated to the smallest of the transition probabilities assigned by the user to views output by the video cameras listed in the Selection_List field of the normal state tuple under consideration, and

setting the D field of the first time-type transition tuple being generated to the value of the Max field of the normal state tuple under consideration; and

for each random state tuple of each of the identified pairs of normal and random state tuples,

setting the Type field of the second time-type transition tuple being generated to “

time”

,

setting the S_last field of the second time-type transition tuple being generated to identify the state tuple that defines the screen configuration that is to be shown before the screen configuration defined by the random state tuple under consideration,

setting the S_current field of the second time-type transition tuple being generated to identify the random state tuple under consideration,

setting the S_next field of the second time-type transition tuple being generated to identify the normal state tuple that defines the screen configuration that is to be shown after the screen configuration defined by the random state tuple under consideration if the second transition tuple being generated is executed, and

setting the D field of the second time-type transition tuple being generated to the value of the Min field of the random state tuple under consideration.

17. The process of claim 12, wherein the process action of providing a user interface for the user to specify display priorities, comprises an action of providing as part of the user interface a provision for the user to enter one or more transition idioms, and wherein the process action of generating the series of state and transition tuples, comprises an action of generating two time-type transition tuples for each transition idiom entered by the user, one of said time-type transition tuples specifying a transition from a screen configuration defined by a current state tuple to screen configuration defined by another state tuple after a prescribed period, and the other of said time-type transition tuples specifying a transition back to the screen configuration defined by said current state tuple from the screen configuration defined by said other state tuple after a second prescribed time period, wherein each of the time-type tuples specifies a transition that occurs if the last screen configuration displayed before the current screen configuration was defined by a state tuple listed in the S_last field and the current screen configuration is defined by the state tuple listed in the S_current field and the view or views associated with the S_current state tuple have been displayed for at least the time period specified in the D field, the screen configuration is changed to that defined by the state tuple listed in the S_next field if not precluded by the transition probability listed in the P field.

18. The process of claim 17, wherein the process action of generating two time-type transition tuples for each transition idiom entered by the user, comprises the actions of:

setting the Type field of the first time-type transition tuple being generated to “

Idiom”

,

setting the S_last field of the first time-type transition tuple being generated to identify a state tuple that defines the screen configuration specified by the user as the last view when the transition idiom was created,

setting the S_current field of the first time-type transition tuple being generated to identify a state tuple that defines the screen configuration specified by the user as the current view when the transition idiom was created,

setting the S_next field of the first time-type transition tuple being generated to identify a state tuple that defines the screen configuration specified by the user as the screen configuration that is to be switched to if the first transition tuple being generated is executed,

setting the P field of the first time-type transition tuple being generated to the value specified by the user, if one, and to 1.0 otherwise, and

setting the D field of the first time-type transition tuple being generated to the value specified by the user which dictates how long the aforementioned current view is to be shown before switching; and

setting the Type field of the second time-type transition tuple being generated to “

Idiom”

,

setting the S_last field of the second time-type transition tuple being generated to identify the state tuple that defines the screen configuration specified by the user as the current view when the transition idiom was created,

setting the S_current field of the second time-type transition tuple being generated to identify the state tuple that defines the screen configuration specified by the user as the screen configuration that is to be switched to if the first transition tuple is executed,

setting the S_next field of the second time-type transition tuple being generated to identify the state tuple that defines the screen configuration specified by the user as the current view when the transition idiom was created,

setting the P field of the second time-type transition tuple being generated to a value of 1.0, and

setting the D field of the second time-type transition tuple being generated to the value specified by the user which dictates how long the aforementioned screen configuration that is to be switched to if the first transition tuple is executed is to be shown before switching back to the current view specified by the user when the transition idiom was created.

19. The process of claim 13, wherein the process action of using the automata to control what is displayed on the display screen over time and where on the screen it is displayed, comprises the actions of:

selecting a prescribed initial state tuple;

causing the screen configuration defined by the selected initial state tuple to be displayed on the display screen;

starting with the first screen configuration displayed, tracking how long each screen configuration is displayed on the screen, and detecting any view changes in the video streams being output by the video cameras across all the sites;

periodically checking each transition tuple to determine if any of the transition tuples are triggered by an elapsed time since a screen configuration has been displayed on the screen or view change in the output of a video camera; and

whenever, it is determined that a transition tuple is triggered, causing the screen configuration to change as specified by the triggered transition tuple.

20. The process of claim 19, wherein the process actions of periodically checking each transition tuple to determine if any of the transition tuples are triggered by a view change in the output of a video camera and causing the screen configuration to change as specified by the triggered transition tuple, comprises the actions of:

whenever a view change in one or more of the video streams being output by the video cameras across all the sites is detected, identifying the view-type transition tuple whose S_current field defines the current screen configuration;

determining if the video camera or cameras responsible for outputting the new view or views match camera or cameras listed in the C_List field of the identified view-type transition tuple and if the new view or views match view or views listed in the V_List field of the identified view-type transition tuple;

whenever a match is found,

determining if the screen configuration of the display screen immediately preceding the current screen configuration, if one existed, matches that defined by the S_last field of the identified view-type transition tuple, and

determining if the time that has elapsed since the current screen configuration was displayed on the screen exceeds the amount of time listed in the Min field of the identified view-type transition tuple; and

whenever it is determined that the screen configuration of the display screen immediately preceding the current screen configuration, if one existed, matches that defined by the S_last field of the identified view-type transition tuple and the time that has elapsed since the current screen configuration was displayed on the screen exceeds the amount of time listed in the Min field of the identified view-type transition tuple, causing the screen configuration currently displayed on the screen to change to that defined by the state tuple identified in the S_next field of the identified view-type transition tuple.

21. The process of claim 19, wherein the process actions of periodically checking each transition tuple to determine if any of the transition tuples are triggered by an elapsed time since a screen configuration has been displayed on the screen and causing the screen configuration to change as specified by the triggered transition tuple, comprises the actions of:

(a) periodically checking the time-type transition tuple whose S_current field identifies a state tuple that defines the currently displayed screen configuration and whose S_last field defines the preceding screen configuration to determine if the period of time identified in the D field of the transition tuple under consideration has expired since the current display configuration was displayed or designated to have been displayed;

(b) whenever it is found that said time period identified in the D field of the transition tuple under consideration has expired and the S_current field identifies a random state tuple, causing the screen configuration currently displayed on the screen to change to that defined by the state tuple identified in the S_next field of the time-type transition tuple under consideration;

(c) whenever it is found that said time period identified in the D field of the transition tuple under consideration has expired and the S_current field identifies a normal state tuple, testing the transition probability found in the P field of the time-type transition tuple under consideration to determine if implementing the transition tuple is precluded;

(d) whenever it is determined that implementing the transition tuple under consideration is not precluded, causing the screen configuration currently displayed on the screen to change to that defined by the state tuple identified in the S_next field of the time-type transition tuple under consideration; and

(e) whenever it is determined that implementing the transition tuple under consideration is precluded, designating the current display configuration to have been displayed at the time this determination is made and repeating process actions (a) through (e) as appropriate.

22. A computer network-based distributed presentation system for displaying one or more video streams output by multiple video cameras located at multiple presentation sites, comprising:

a general purpose computing device;

a display screen located at each presentation site which is capable of simultaneously displaying one or more video streams in display sectors of the screen and which is controlled by said computing device;

at least one video camera located at one or more of the presentation sites, each of which outputs a video stream comprising views of the presentation site where the camera is located to the computer network;

a computer program comprising a display director program module executable by the computing device, wherein the computing device is directed by the display director module to,

determine on an on-going basis which of the video streams output to the computer network are to be downloaded for display on the screen at one or more of the presentation sites and select what sector of the screen the downloaded video stream or streams are to be displayed;

controlling the display screen at one or more of the presentation sites so as to display the downloaded video stream or streams in the selected sectors.