System and method for distributed meetings

• US 20040263636A1
• Filed: 06/26/2003
• Published: 12/30/2004
• Est. Priority Date: 06/26/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Rui, Yong, Gupta, Anoop, Cutler, Ross
• Granted Patent: US 7,428,000 B2
• Time in Patent Office: Days
• US Class Current: 348/211.12
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04N 7/152 Multipoint control units therefor
• CPC Class Codes: H04N 7/155 involving storage of or access to video conference sessions tracking arrangements for later retrieval of a computer conference content or participants activities H04L12/1831

Abstract

A system and method for teleconferencing and recording of meetings. The system uses a variety of capture devices (a novel 360° camera, a whiteboard camera, a presenter view camera, a remote view camera, and a microphone array) to provide a rich experience for people who want to participate in a meeting from a distance. The system is also combined with speaker clustering, spatial indexing, and time compression to provide a rich experience for people who miss a meeting and want to watch it afterward.

Claims

1. An automated system for capturing and viewing an event having event participants, comprising:

multiple cameras simultaneously capturing images of different sub-events occurring in a space associated with an event;

a server capable of recording and broadcasting the captured sub-events; and

one or more clients in network connection with said server that view portions of the captured event.

2. The system of claim 1, wherein the multiple cameras are one of:

a 360-degree camera centrally positioned to monitor in substantially 360-degrees the space in which the event occurs;

a remote view camera positioned so as to capture a view of event participants in said space associated with said event to be transmitted to a client over said network;

a presenter view camera positioned so as to capture a view of an overview of the space associated with the event; and

a whiteboard capture camera positioned so as to capture strokes written on a whiteboard.

3. The system of claim 2 wherein said omni-directional camera comprises a set of cameras configured in a circular back to back fashion.

4. The system of claim 3 further comprising a panoramic stitcher that stitches together images captured from each camera to create a panoramic image of said space in which the event occurs.

5. The system of claim 2 wherein said presenter view camera is integrated with a microphone.

6. The system of claim 2 wherein said remote view camera is integrated with a microphone.

7. The system of claim 2 further comprising a virtual director that allows switching between said multiple cameras to display a view of one of said different sub-events.

8. The system of claim 7 wherein said virtual director determines which camera view to display by:

determining if a person is speaking and facing toward a display that displays at least one remote event participant, and if so using a camera view captured by said remote camera to display;

determining if a person is talking and the presenter view camera can track them and provide a higher resolution image than the 360-degree camera, and if so using a camera view captured by said presenter view camera for display; and

else, using a camera view captured by said 360-degree camera to display.

9. The system of claim 1 further comprising a microphone device that simultaneously captures audio data of said event.

10. The system of claim 9 wherein said microphone device is used to determine the direction from which an event participant is speaking by using sound source localization.

11. The system of claim 9 wherein captured images of the sub-events are used in combination with sound source localization to refine the determined direction from which an event participant is speaking.

12. The system of claim 1 wherein said system is used for at least one of:

broadcasting an event to one or more remote clients;

receiving data associated with the event from a client;

recording said event; and

browsing of a recording of said event.

13. The system of claim 1 further comprising a projector for projecting event materials onto a screen.

14. The system of claim 1 further comprising a monitor for displaying one or more remote participants or other meeting materials in said space where said event occurs.

15. The system of claim 1 further comprising an event kiosk which is used to control one of event broadcast, recording and recorded event playback.

16. The system of claim 15 wherein said kiosk further comprises a graphical user interface.

17. The system of claim 16 wherein said graphical user interface comprises at least one of:

an initial display showing initial status of the system;

a setup display for allowing a user to start recording of an event;

an event status display displaying images from said multiple cameras, event duration and a control for stopping or canceling said recording of said event; and

a stop meeting confirmation display that confirms the user'"'"'s action of stopping or canceling said recording of said event.

18. The system of claim 15 further wherein said event kiosk is located on one of said multiple cameras.

19. The system of claim 1 further comprising a graphics capture device used to capture data presented in said event or transferred between the multiple cameras, server and one more clients.

20. The system of claim 1 further comprising an archive server on which recorded events are stored and wherein said archive server plays back said recorded events to said clients.

21. The system of claim 1 further comprising an archive server on which annotations to said captured sub-events are saved.

22. A computer-implemented process for recording an event, comprising the process actions of:

recording images captured by multiple different cameras of different types simultaneously capturing different portions of an event;

recording audio synchronized with said recorded images;

storing said images and synchronized audio as event data on a server capable of distributing the said event data.

23. The computer-implemented process of claim 22 further comprising the process action of said server transmitting said event data to one or more clients in network connection with said server.

24. The computer-implemented process of claim 23 further comprising the process action of said one or more clients playing back said stored images and synchronized audio.

25. The computer-implemented process of claim 23 wherein said multiple cameras are at least one of:

a 360-degree camera centrally positioned to monitor in substantially 360-degrees the space in which the event occurs;

a remote view camera positioned so as to capture a view of event participants in the space in which the event occurs to be transmitted to a viewer over said network;

a presenter view camera positioned so as to capture an overview of the space in which said event occurs; and

a whiteboard capture camera positioned so as to capture strokes written on a whiteboard.

26. The computer-implemented process of claim 22 further comprising the process action of capturing graphics data of said event.

27. The computer-implemented process of claim 26 further comprising the process action of determining key frames that summarize event sub-events by analyzing said captured graphics data.

28. The computer-implemented process of claim 27 further comprising a process action of indexing said stored event data, wherein said indexing process action comprises determining graphics key frames that summarize sub-events of said event by analyzing said images captured by a graphics capture device and using said key frames as an index to the corresponding portion of audio and images of the stored event data.

29. The computer-implemented process of claim 25 further comprising the process action of indexing said stored event data by at least one of the following process actions:

determining at least one whiteboard key frame that summarizes a sub-event by analyzing said images captured by said whiteboard camera and using said whiteboard key frames as an index to the corresponding portion of audio and images of the meeting;

determining a timeline outlining the sequence of sub-events occurring in the event and using said timeline as an index to a specific portion of the audio or images of the event; and

using speech recognition to transcribe the audio of the event and using word searches to locate specific portions of said audio.

30. The computer-implemented process of claim 29 wherein said word searches are by phonetic spelling.

31. The computer-implemented process of claim 22 further comprising the process action of indexing said stored event data for playback.

32. The computer-implemented process of claim 31 wherein the process action of indexing said stored event data comprises determining who is speaking by analyzing audio of said event and segmenting the audio of each speaker to be accessible on an individual basis to be used as an index of said stored event data.

33. The computer-implemented process of claim 31 wherein the process action of indexing said recorded event data comprises analyzing said event audio to categorize tone of voice for event participants speaking at said event and indexing said stored event data by tone of voice.

34. The computer-implemented process of claim 31 wherein the process action of indexing said stored event data comprises determining speaker turn rate, wherein speaker turn rate comprises the number of speakers of a given period of time, and using said speaker turn rate to index said stored event data.

35. The computer-implemented process of claim 31 wherein the process action of indexing said stored event data comprises determining when event recording was turned on and off using event turn on and turn off events to index said stored event data.

36. The computer-implemented process of claim 31 wherein the process action of indexing said stored event data comprises determining when a shared application is started or stopped and using this to index said stored event data.

37. The computer-implemented process of claim 31 wherein the process action of indexing said stored event data comprises determining when someone is pointing at an object and using the pointing event to index to said stored event data.

38. The computer-implemented process of claim 31 wherein the process action of indexing said stored event data comprises using data recorded by a client to be used as an index to said recorded meeting data.

39. The computer-implemented process of claim 24 further comprising the process action of adjusting meeting playback speed using a time compression feature, wherein said time compression feature appropriately slows down or speeds up playback of the audio signal.

40. The computer-implemented process of claim 39 wherein said time compression feature appropriately slows down or speeds up playback of an image sequence that is synchronized with said audio signal.

41. The computer-implemented process of claim 39 wherein said time compression feature is applied to the audio signal of a single speaker.

42. The computer-implemented process of claim 24 further comprising the process action of segmenting the audio from a given speaker in said event.

43. The computer-implemented process of claim 42 further comprising the process action of providing a fast forward function that allows skipping of a portion of the audio of one or more speakers.

44. A graphical user interface for viewing and playing back event data comprising:

a speaker image display pane displaying a speaker speaking at an event;

a whiteboard display pane displaying an image of content written on a whiteboard;

and a panoramic image display pane displaying a panoramic image of said event.

45. The graphical user interface of claim 44 further comprising a key frame image summarizing content written on a whiteboard.

46. The graphical user interface of claim 45 further comprising a thumbnail view of said key frame, wherein selecting said thumbnail view will display said key frame in said whiteboard pane.

47. The graphical user interface of claim 44 further comprising a graphics display pane displaying an image captured by a graphical capture device.

48. The graphical user interface of claim 47 further comprising a key frame image summarizing content of said data captured by said graphical capture device.

49. The graphical user interface of claim 47 wherein said key frame is an electronic slide of an electronic presentation.

50. The graphical user interface of claim 48 wherein a user can jump to a starting point of at least one of audio or video corresponding to the key frame image in a meeting playback timeline by selecting said key frame thumbnail.

51. A system for conducting a distributed meeting, the system comprising:

a 360-degree camera for capturing images of meeting participants in a meeting in substantially 360 degrees about said 360-degree camera;

a whiteboard camera for capturing images of contents written on a whiteboard;

a presenter camera for capturing images of an overview of the meeting room;

a microphone array for capturing the audio of the meeting that is synchronized with one of said images captured by said 360-degree camera, whiteboard camera or presenter camera; and

a meeting server for performing processing required to broadcast and record meeting data.

52. The system of claim 51 further comprising a network connecting said meeting server to at least one remote meeting participant, wherein said network is used to broadcast meeting images and audio from said server to participants and receive audio and images from said remote meeting participants at said server.

53. The system of claim 51 further comprising an archive server for performing processing required to playback recorded meeting data.

54. The system of claim 53 further comprising one or more archive clients capable of playing back said captured images and synchronized audio.

55. An automated system for capturing and viewing an event having event participants, comprising:

multiple cameras simultaneously capturing images of different sub-events occurring in a space associated with an event;

an event server, that processes in substantially real time said event data;

an event post processor that process said event data only when the event is completed; and

at least one event client in connection with said event server wherein said event client allows viewing live events and archived events.

56. The automated system of claim 55 further comprising an archive server which acts as a store for said event data.

57. The automated system of claim 55 wherein the event server performs at least one of the following functions:

acquiring audio or video from said capture devices;

compressing said audio or video;

archiving said audio or video;

providing video or audio to said client; and

receiving audio or video generated at said event client from said event client.

58. A computer-readable medium having computer-executable instructions for viewing a recorded event, said computer-executable instructions comprising:

simultaneously capturing images of different sub-events by of an event with multiple cameras each capturing a different sub-event;

capturing audio associated with the different sub-events; and

transmitting the captured sub-events and associated audio from a server to one or more clients in network connection with said server.

59. The computer-readable medium of claim 58 further comprising a computer-executable instruction for establishing a two-way connection between said server and said one or more clients for transferring images, audio and data between said server and said one or more clients.

60. The computer-readable medium of claim 58 further comprising a computer-executable instruction for projecting event data onto a screen.

61. The computer-readable medium of claim 60 wherein said event data that is projected onto a screen includes a view of one or more clients'"'"' meeting participants, a view of the event captured by one of said multiple cameras and a software application display.

62. A computer-implemented process for viewing an event comprising the process actions of:

simultaneously capturing images of different sub-events of an event with multiple cameras each capturing a different sub-event;

capturing audio associated with the different sub-events synchronized with said captured images;

storing said captured images and associated audio on a server; and

viewing the captured sub-events and associated audio by one or more clients in network connection with said server.

63. The computer-implemented process of claim 62 further comprising a process action of indexing said event data, wherein said indexing process action comprises determining graphics key frames that summarize sub-events of said event by analyzing images captured by a graphics capture device and using said key frames as an index to jump to the corresponding portion of audio and images of the event data while the event is ongoing.

64. The computer-implemented process of claim 62 further comprising a process action of indexing said event data, wherein said indexing process action comprises determining at least one whiteboard data key frame that summarizes a sub-event by analyzing said images captured by one of said multiple cameras and using said whiteboard key frames as an index to jump to the corresponding portion of audio and images of the meeting while the event is ongoing.

65. The computer-implemented process of claim 62 further comprising the process action of playing back a portion of said event while said event is ongoing using a time compression feature, wherein said time compression feature appropriately slows down or speeds up playback of the audio signal.

66. The computer-implemented process of claim 65 wherein said time compression feature appropriately slows down or speeds up playback of an image sequence that is synchronized with said audio signal.

67. The computer-implemented process of claim 65 wherein said time compression feature is applied to the audio signal of a single speaker.

68. The computer-implemented process of claim 65 further comprising the process action of providing a fast forward function that allows skipping of a portion of the audio of one or more speakers.

69. A system for conducting a distributed meeting, the system comprising:

a 360-degree camera for capturing images of meeting participants in a meeting room in substantially 360 degrees about said 360-degree camera, wherein said 360-degree camera includes an integrated computer that performs processing required to broadcast said images and associated meeting data.

70. The system of claim 69 further comprising at least one of:

a whiteboard camera for capturing images of contents written on a whiteboard; and

a presenter view camera for capturing images of an overview of the meeting room.

71. The system of claim 70 further comprising a microphone array for capturing the audio of the meeting that is synchronized with one of said images captured by said 360-degree camera, whiteboard camera or presenter view camera.

72. The system of claim 69 wherein said associated meeting data comprises audio that is synchronized with said images.