Automated online broadcasting system and method using an omni-directional camera system for viewing meetings over a computer network

• US 6,937,266 B2
• Filed: 06/14/2001
• Issued: 08/30/2005
• Est. Priority Date: 06/14/2001
• Status: Expired due to Term

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Cadiz, Johnathan J, Rui, Yong, Gupta, Anoop, Cutler, Ross Garrett
• Primary Examiner(s): Eng, George
• Application Number:
  US09/681,843
• Publication Number:
  US 20020191071A1
• Time in Patent Office: 1,538 Days
• Field of Search: 348/14.01, 348/14.02, 348/14.03, 348/14.05, 348/14.07, 348/14.08, 348/14.09, 348/14.1, 348/14.11, 348/14.12, 348/14.13, 348/14.16, 348/36, 348/37, 348/39, 348/143, 348169-172, 348/211.99, 348/211.3
• US Class Current: 348/14.05
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

An automated system and method for broadcasting meetings over a computer network. The meeting is filmed using an omni-directional camera system and capable of being presented to a viewer both live and on-demand. The system of the present invention includes an automated camera management system for controlling the camera system and an analysis module determining the location of meeting participants in the meeting environments. The method of the present invention includes using the system of the present invention to broadcast an event to a viewer over a computer network. In particular, the method includes filming the event using an omni-directional camera system. Next, the method determines the location of each event participant in the event environment. Finally, a viewer is provided with a user interface for viewing the broadcast event. This user interface allows a viewer to choose which event participant that the viewer would like to view.

Claims

1. An automated event presentation system for capturing and viewing an event having event participants, comprising:

an omni-directional camera system that provides a seamless onmi-directional image of the event and that automatically tracks event participants simultaneously to determine the event participants that are speaking using audio analysis including a microphone-array sound source localization technique to alleviate camera view switching delays and films the event;

an automated online broadcasting system including a tracker module that controls and uses the omni-directional camera system and video tracking techniques to monitor and keep track of each of the tracked event participants simultaneously, and broadcasts the event;

a virtual director module using a probabilistic finite state machine and receiving as input audio and video tracking results to automatically select at least a portion of the omni-directional image as an output view; and

a viewer platform in communication with the automated online broadcasting system that allows a viewer to view the output view of the broadcasted event.

2. The automated event presentation system as set forth in claim 1, wherein the omni-directional camera system includes a camera having a wide-angle view of approximately 360 degrees.

3. The automated event presentation system as set forth in claim 2, wherein the camera includes a wide-angle imaging device.

4. The automated event presentation system as set forth in claim 1, wherein the omni-directional camera system includes a plurality of cameras that combined provide an approximately 360-degree field-of-view.

5. The automated event presentation system as set forth in claim 1, wherein the automated online broadcasting system further comprises a switching module that allows switching between of the omni-directional image of the event.

6. The automated event presentation system as set forth in claim 1, wherein the omni-direction camera system has a resolution of approximately 1000 by 1000 pixels.

7. The automated event presentation system as set forth in claim 1, wherein the automated online broadcasting system further comprises an analysis module for finding and indexing the event participants.

8. The automated event presentation system as set forth in claim 1, wherein the omni-directional camera system requires no physical movement to capture the event participants.

9. The automated event presentation system as set forth in claim 1, further comprising a user interface on the viewer platform that allows an arbitrary number of viewers to view an arbitrary number of viewpoints of the broadcasted event.

10. The automated event presentation system as set forth in claim 1, wherein the omni-directional image provides an infinite number of viewpoints, with each of the viewpoints corresponding to a portion of the omni-directional image, such that instantaneous switching is supported for an infinite number of viewers that select arbitrarily different viewpoints.

11. A method for filming and recording an event having event participants and presenting the event to a viewer, comprising:

filming and recording the event using an omni-directional camera system to provide a seamless omni-directional image that contains each of the event participants;

automatically determining a location of the event participants in the omni-directional image by using a speaker detection technique to determine the event participants that are speaking;

tracking multiple event participants simultaneously using the-speaker detection technique and a video tracking technique;

providing a user interface that allows a choice of which of the event participants in the omni-directional image to view, the choice being made by at least one of;

(a) manually by the viewer and (b) automatically by a virtual director that uses a probabilistic finite state machine and receives as input audio and video tracking results from the speaker detection and video tracking techniques; and

switching instantaneously between views of the event participants in the omni-directional image in response to the choice.

12. A computer-readable medium having computer-executable instructions for performing the method recited in claim 11.

13. The method as set forth in claim 12, further comprising storing annotations associated with the event and synchronizing these annotations with the event.

14. The method as set forth in claim 13, further comprising allowing the viewer to select which of the annotations to store.

15. The method as set forth in claim 13, wherein which annotations to store may be selected:

(a) while the event is occurring;

(b) after the event has occurred.

16. The method as set forth in claim 13, wherein the annotations include at least one of:

(a) a whiteboard;

(b) a digital chat regarding the event;

(c) a digital question and answer session over a computer network.

17. The method as set forth in claim 11, wherein multiple camera views are obtained from the omni-directional image and further comprising using the speaker detection technique to follow event participants that are speaking by switching from one camera view to another camera view.

18. The method as set forth in claim 11, wherein the omni-directional camera system is one of:

(a) a single panoramic camera;

(b) an array cameras having an approximately 360-degree field-of-view.

19. A method for displaying at least a portion of a seamless omni-directional image capturing an event occurring within an event environment, comprising:

filming the event and automatically tracking multiple event participants simultaneously using audio and video processing techniques and a single omni-directional camera system having a single camera to produce the seamless omni-directional image;

transmitting the omni-directional image from a broadcasting platform to a viewer platform using a computer network;

using a probabilistic finite state machine that receives as input audio and video tracking results from the audio and video processing techniques to automatically select which portion of the omni-directional image to view;

using the viewer platform to allow a viewer to select as one state of the probabilistic finite state machine which portion of the omni-directional image the viewer would like to view; and

switching instantaneously between views of the omni-directional image by presenting a desired portion of the omni-directional image as selected by the viewer.

20. The method as set forth in claim 19, wherein the viewer selects to view multiple portions of the omni-directional image.

21. The method asset forth in claim 19, wherein the omni-directional image contains all event participants within the event environment.

22. The method as set forth in claim 19, further comprising:

transmitting a low-resolution version of the omni-directional image to the viewer platform, wherein the omni-directional image produced by the onmi-directional camera system is a high-resolution omni-directional image;

selecting which portion of the omni-directional image to view, the selection being made by at least one of;

(a) manually by the viewer and (b) automatically by a virtual director module; and

transmitting a high-resolution version of the selected portion of the omni-directional image to the viewer platform.

23. An automated event presentation system for capturing an event, comprising:

a high-resolution onmi-directional camera system that provides an omni-directional image of the event, the omni-directional image containing multiple camera views;

an automated online broadcasting system capable of broadcasting the omni-directional image over a computer network;

a viewer platform in communication with computer network that receives the omni-directional image; and

a virtual director module within the automated online broadcasting system that uses a probabilistic finite state machine and receives as input audio and video tracking results to determine which of the multiple camera views within the omni-directional image to display on the viewer platform by applying a set of expert production rules based at least in part on a display history of an event participant.

24. The automated event presentation system as set forth in claim 23, wherein the virtual director module further comprises a switching module that provides switching between the multiple camera views of the event.

25. The automated event presentation system as set forth in claim 24, wherein the switching module provides instantaneous switching between the multiple camera views.

26. The automated event presentation system as set forth in claim 24, wherein the switching module is capable of providing negative switching that allows switching to a camera view of a person speaking before the person begins to speak.

27. An automated event presentation system for capturing and viewing an event having multiple event participants, comprising:

an omni-directional camera system that provides a seamless omni-directional image of the event;

a tracker module that determines a number of the multiple event participants and automatically tracks the number of multiple event participants simultaneously within the omni-directional image using audio-based tracking techniques and video-based tracking techniques combined in a probabilistic manner to obtain audio and video tracking results;

a virtual director module that uses a probabilistic finite state machine and receives as input the audio and video tracking results to automatically select without user intervention at least a portion of the omni-directional image for use as an output view;

an automated online broadcasting system that broadcasts the output view and the omni-directional image over a computer network; and

a viewer platform in communication with the automated online broadcasting system that allows a viewer to view at least one of;

(a) the output view;

(b) the omni-directional image.

28. The automated event presentation system as set forth in claim 27, further comprising a plurality of viewer platforms in communication with the automated online broadcasting system that allows a plurality of remote viewers to simultaneously view different views of the broadcasted event.