System and method for distributed meetings

• US 7,428,000 B2
• Filed: 06/26/2003
• Issued: 09/23/2008
• Est. Priority Date: 06/26/2003
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Rui, Yong, Gupta, Anoop, Cutler, Ross
• Primary Examiner(s): Ye; Lin
• Assistant Examiner(s): Yoder, III; Chriss S
• Application Number:
  US10/608,313
• Publication Number:
  US 20040263636A1
• Time in Patent Office: 1,916 Days
• Field of Search: 348/211.12, 348/14.09, 348/14.03, 348/14.08, 348/14.01, 348/211.13, 348/14.11, 348/14.12, 348/14.1, 348/14.16, 348/14.6, 348/14.05, 348/36, 348/143, 348/159, 348/169, 348/211.3, 348/211.99
• US Class Current: 348/14.11
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04N 7/152 Multipoint control units therefor
• CPC Class Codes: H04N 7/155 involving storage of or access to video conference sessions tracking arrangements for later retrieval of a computer conference content or participants activities H04L12/1831

Abstract

A system and method for teleconferencing and recording of meetings. The system uses a variety of capture devices (a novel 360° camera, a whiteboard camera, a presenter view camera, a remote view camera, and a microphone array) to provide a rich experience for people who want to participate in a meeting from a distance. The system is also combined with speaker clustering, spatial indexing, and time compression to provide a rich experience for people who miss a meeting and want to watch it afterward.

Claims

1. An automated system for capturing and viewing an event having event participants, comprising:

multiple cameras of different types simultaneously capturing images of sub-events occurring in a space associated with an event, wherein the multiple cameras of different types are at least two of;

a 360-degree camera centrally positioned to monitor in substantially 360-degrees the space in which the event occurs;

a remote view camera positioned so as to capture a view of event participants in said space associated with said event to be transmitted to a client over said network;

a presenter view camera positioned so as to capture a view of an overview of the space associated with the event wherein a presenter would typically be presenting; and

a whiteboard capture camera positioned so as to capture strokes written on a whiteboard;

a virtual director that automaticaljy determines which view of said multiple cameras of different types to display, wherein said virtual director determines which camera view to display by;

determining if a person is speaking and facing toward a display that displays at least one remote event participant, and if so using a camera view captured by said remote camera to display;

determining if a person is talking and the presenter view camera can track them and provide a higher resolution image than the 360-degree camera, and if so using a camera view captured by said presenter view camera for display; and

else, using a camera view captured by said 360-degree camera to display;

a server capable of recording and broadcasting the captured sub-events; and

one or more clients in network connection with said server that view portions of the captured event.

2. The system of claim 1 wherein said 360-degree camera comprises a set of cameras configured in a circular back to back fashion.

3. The system of claim 2 further comprising a panoramic stitcher that stitches together images captured from each camera to create a panoramic image of said space in which the event occurs.

4. The system of claim 1 wherein said presenter view camera is integrated with a microphone.

5. The system of claim 1 wherein said remote view camera is integrated with a microphone.

6. The system of claim 1 further comprising a microphone device that simultaneously captures audio data of said event.

7. The system of claim 6 wherein said microphone device is used to determine the direction from which an event participant is speaking by using sound source localization.

8. The system of claim 6 wherein captured images of the sub-events are used in combination with sound source localization to refine the determined direction from which an event participant is speaking.

9. The system of claim 1 wherein said system is used for at least one of:

broadcasting an event to one or more remote clients;

receiving data associated with the event from a client;

recording said event; and

browsing of a recording of said event.

10. The system of claim 1 further comprising a projector for projecting event materials onto a screen.

11. The system of claim 1 further comprising a monitor for displaying one or more remote participants or other meeting materials in said space where said event occurs.

12. The system of claim 1 further comprising an event kiosk which is used to control one of event broadcast, recording and recorded event playback.

13. The system of claim 12 wherein said kiosk further comprises a graphical user interface.

14. The system of claim 13 wherein said graphical user interface comprises at least one of:

an initial display showing initial status of the system;

a setup display for allowing a user to start recording of an event;

an event status display displaying images from said multiple cameras, event duration and a control for stopping or canceling said recording of said event; and

a stop meeting confirmation display that confirms the users action of stopping or canceling said recording of said event.

15. The system of claim 12 further wherein said event kiosk is located on one of said multiple cameras of different types.

16. The system of claim 1 further comprising a graphics capture device used to capture data presented in said event or transferred between the multiple cameras, server and one more clients.

17. The system of claim 1 further comprising an archive server on which recorded events are stored and wherein said archive server plays back said recorded events to said clients.

18. The system of claim 1 further comprising an archive server on which annotations to said captured sub-events are saved.

19. A system for conducting a distributed meeting, the system comprising:

a 360-degree camera for capturing images of meeting participants in a meeting in substantially 360 degrees about said 360-degree camera;

a whiteboard camera for capturing images of contents written on a whiteboard;

a presenter camera for capturing images of an overview of the meeting room in the area where a presenter would typically be presenting;

a microphone array for capturing the audio of the meeting that is synchronized with one of said images captured by said 360-degree camera, whiteboard camera or presenter camera; and

a virtual director that automatically determines which view of said 360 degree camera, whiteboard camera or presenter camera to display and switches to the determined view of the associated camera to display a view of one of said different sub-events, wherein said virtual director determines which camera view to display by;

determining if a person is talking and the presenter view camera can track them and provide a higher resolution image than the 360-degree camera, and if so using a camera view captured by said presenter view camera for display; and

else, using a camera view captured by said 360-degree camera to display; and

a meeting server for performing processing required to broadcast and record meeting data.

20. The system of claim 19 further comprising a network connecting said meeting server to at least one remote meeting participant, wherein said network is used to broadcast meeting images and audio from said server to participants and receive audio and images from said remote meeting participants at said server.

21. The system of claim 19 further comprising an archive server for performing processing required to playback recorded meeting data.

22. The system of claim 21 further comprising one or more archive clients capable of playing back said captured images and synchronized audio.

23. An automated system for capturing and viewing an event having event participants, comprising:

multiple cameras of different types simultaneously capturing images of different sub-events occurring in a space associated with an event, wherein the different types of cameras include a 360-degree camera, a remote view camera, a presenter view camera, or a whiteboard capture camera;

an event server, that processes in substantially real time said event data;

an event post processor that process said event data only when the event is completed;

a virtual director that automatically determines which of said multiple cameras of different types to display based on the position of a person speaking and the ability of one of the multiple cameras of different types to track a person speaking in the captured images and audio signals received and switches between said multiple cameras of different types to display a view of one of said different sub-events; and

at least one event client in connection with said event server wherein said event client allows viewing live events and archived events.

24. The automated system of claim 23 further comprising an archive server which acts as a store for said event data.

25. The automated system of claim 23 wherein the event server performs at least one of the following functions:

acquiring audio or video from said capture devices;

compressing said audio or video;

archiving said audio or video;

providing video or audio to said client; and

receiving audio or video generated at said event client from said event client.

26. A computer-readable medium having computer-executable instructions for viewing a recorded event, said computer-executable instructions comprising:

simultaneously capturing images of different sub-events of an event with multiple cameras of different types each capturing a different sub-events, wherein the different types of cameras include a 360-degree camera, a remote view camera, a presenter view camera, or a whiteboard capture camera;

capturing audio associated with the different sub-events;

automatically selecting which of the captured sub-events to transmit based on the position of a person speaking and the ability of one of the multiple cameras of different types to track a person speaking in the captured images of the different sub-events and the captured audio associated with the different sub-events; and

transmitting the selected captured sub-events and associated audio from a server to one or more clients in network connection with said server.

27. The computer-readable medium of claim 26 further comprising a computer-executable instruction for establishing a two-way connection between said server and said one or more clients for transferring images, audio and data between said server and said one or more clients.

28. The computer-readable medium of claim 26 further comprising a computer-executable instruction for projecting event data onto a screen.

29. The computer-readable medium of claim 28 wherein said event data that is projected onto a screen includes a view of one or more clients'"'"' meeting participants, a view of the event captured by one of said multiple cameras and a software application display.

30. A system for conducting a distributed meeting, the system comprising:

a 360-degree camera for capturing images of meeting participants in a meeting room in substantially 360 degrees about said 360-degree camera, wherein said 360-degree camera includes an integrated computer that performs processing required to broadcast said images and associated meeting data; and

a whiteboard camera for capturing images of contents written on a whiteboard;

a presenter view camera for capturing images of an overview of the meeting room in a space where a presenter would typically be presenting; and

a virtual director that automatically determines which view of said 360 degree camera, whiteboard view camera and presenter view camera to display based on determining if a person is speaking and is positioned in a certain manner relative to one of the cameras and the ability of one of the cameras to track the person speaking.

31. The system of claim 30 further comprising a microphone array for capturing the audio of the meeting that is synchronized with one of said images captured by said 360-degree camera, whiteboard camera or presenter view camera.

32. The system of claim 30 wherein said associated meeting data comprises audio that is synchronized with said images.