Automatic multi-camera video composition

• US 20020105598A1
• Filed: 12/12/2000
• Published: 08/08/2002
• Est. Priority Date: 12/12/2000
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Yeo, Boon-Lock, Yeung, Minerva Ming-Yee, Tai, Li-Cheng
• Granted Patent: US 6,577,333 B2
• Time in Patent Office: Days
• US Class Current: 348/705
• CPC Class Codes: H04N 5/268 Signal distribution or switching
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

Methods and apparatus are disclosed for automatically selecting a video output from among several video input sources. In one method, one or more audio sensors are associated with each video input source. Preferably, an audio sensor is positioned to receive audio signals from directions that receive favorable coverage in the field of view of the associated video source. An autoselector calculates audio scores for each of the audio sensors over short (e.g., 0.5 seconds) examination intervals. At each examination interval, the potential exists for a different video source to be selected as the video output. The autoselector selects a video source based on the audio scores for an examination interval, as well as the recent time-history of video source selection. For instance, if a new video source has just been selected, selection of a different source may be disabled for a few seconds. The time-history is also used to increase the probability that source selection varies in a seemingly-natural manner.

Claims

1. An automatic video switching system comprising:

multiple audio inputs, each associated with at least one corresponding video source;

an audio scorer capable of calculating an audio score for each of the audio inputs, the audio score for a given audio input corresponding to an audio signal received at that input over an examination interval;

state memory to store at least a partial time-history of video source selection; and

switching logic to select a current video source for a given examination interval, based on a weighted comparison for that examination interval of the audio scores associated with each of the video sources, the comparison weighting based on the partial time-history of video source selection.

2. The system of claim 1, wherein the switching logic comprises a set of conversational rules that increase the probability of switching between video sources as one source remains selected for multiple consecutive examination intervals.

3. The system of claim 1, wherein the switching logic also bases selection of a current video source on one or more features extracted from one or more of the video sources.

4. The system of claim 1, wherein the system has the capability to operate in a real-time mode wherein the switching logic selects a current video source at a point in a video stream corresponding approximately to the end of the given examination interval.

5. The system of claim 1, wherein the system has the capability to operate in a posterior composition mode wherein the switching logic selects a current video source at a given temporal video switching point based on an examination interval that includes audio input that corresponds to a point in a video stream that occurs after the video switching point.

6. The system of claim 1, further comprising a video switcher to accept video signals from the video sources corresponding to the audio inputs, and to switch one of those video sources onto a video output in response to the current video source selection of the switching logic.

7. The system of claim 1, wherein each video source is selected from the group of sources comprising:

an analog video camera output signal, a video camera digital output signal, a digitally-compressed video signal, a recorded video signal, and a computer-generated video signal.

8. The system of claim 1, wherein the system comprises at least one network connection to receive one of the audio inputs from a remote location.

9. The system of claim 1, wherein the switching logic has the capability to select a different current video source for each of several remote locations.

10. A method of automatically selecting a video source from among multiple available video sources, the method comprising:

associating at least one audio source with each of the video sources;

evaluating, over an examination interval, an audio signal received from each audio source, thereby forming an audio score for each audio source that expresses the level of audio activity present at that audio source during the examination interval;

maintaining at least a partial time-history of video source selection;

performing a weighted comparison for the evaluation interval of the audio scores associated with each of the video sources, where the weighting is based on the partial time-history of video source selection; and

selecting a current video source based on the weighted comparison.

11. The method of claim 10, wherein when more than one audio score is associated with a given video source, selecting the largest of the audio scores associated with that video source as the score for weighted comparison for that evaluation interval.

12. The method of claim 10, wherein when more than one audio score is associated with a given video source, combining the audio scores associated with that video source by calculating a norm of those audio scores.

13. The method of claim 10, wherein associating at least one audio source with each of the video sources comprises expressing the association as a matrix that indicates a weight for each combination of audio source and video source, and wherein performing a weighted comparison comprises, for each video source, producing a combined score by weighting the audio scores by the corresponding weights in the matrix.

14. The method of claim 10, wherein performing a weighted comparison comprises weighting the audio score associated with a last-selected current video source according to a weighting function that varies according to the length of time that the last-selected current video source has remained selected.

15. The method of claim 14, wherein when the last-selected video source remains selected for an extended period of time, further weighting the audio score associated with that source by a fractional weight to decrease the probability of that source remaining selected.

16. The method of claim 10, wherein forming an audio score comprises dividing the examination interval into a number of temporal subwindows, and for each subwindow:

determining the variation in each of the audio signals during that subwindow; and

incrementing the audio score for the audio source having the largest variation for that subwindow.

17. The method of claim 10, the method further comprising, when one of the audio sources and a corresponding video source are at a location remote from the location where the current video source is selected, receiving the audio signal over a communications network, and notifying the remote location to transmit video when its video source has been selected as the current video source.

18. The method of claim 10, wherein the selection of a current video source occurs in real-time at approximately the end of an examination interval.

19. The method of claim 10, comprising relating the selection of a current video source to a point in the video stream that corresponds to a time preceding the end of the examination interval.

20. An apparatus comprising a computer-readable medium containing computer instructions that, when executed, cause a processor or multiple communicating processors to perform a method of automatically selecting a video source from among multiple available video sources, the method comprising:

associating at least one audio source with each of the video sources;

evaluating, over an examination interval, an audio signal received from each audio source, thereby forming an audio score for each audio source that expresses the level of audio activity present at that audio source during the examination interval;

maintaining at least a partial time-history of video source selection;

performing a weighted comparison for the evaluation interval of the audio scores associated with each of the video sources, where the weighting is based on the partial time-history of video source selection; and

selecting a current video source based on the weighted comparison.

21. The apparatus of claim 20, wherein associating at least one audio source with each of the video sources comprises expressing the association as a matrix that indicates a weight for each combination of audio source and video source, and wherein performing a weighted comparison comprises, for each video source, producing a combined score by weighting the audio scores by the corresponding weights in the matrix.

22. The apparatus of claim 20, wherein performing a weighted comparison comprises weighting the audio score associated with a last-selected current video source according to a weighting function that varies according to the length of time that the last-selected current video source has remained selected.

23. The apparatus of claim 20, wherein forming an audio score comprises dividing the examination interval into a number of temporal subwindows, and for each subwindow:

determining the variation in each of the audio signals during that subwindow; and

incrementing the audio score for the audio source having the largest variation for that subwindow.

24. The apparatus of claim 20, the method further comprising, when one of the audio sources and a corresponding video source are at a location remote from the location where the current video source is selected, receiving that audio signal over a communications network, and notifying the remote location to transmit video when its video source has been selected as the current video source.