Multi-Device Capture and Spatial Browsing of Conferences

• US 20100085416A1
• Filed: 10/06/2008
• Published: 04/08/2010
• Est. Priority Date: 10/06/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Cha, Zhang, Zhengyou, Chou, Philip A., Liu, Zicheng, Junuzovic, Sasa, Hegde, Rajesh K.
• Granted Patent: US 8,537,196 B2
• Time in Patent Office: Days
• US Class Current: 348/14.80
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827

Abstract

Multi-device capture and spatial browsing of conferences is described. In one implementation, a system detects cameras and microphones, such as the webcams on participants'"'"' notebook computers, in a conference room, group meeting, or table game, and enlists an ad-hoc array of available devices to capture each participant and the spatial relationships between participants. A video stream composited from the array is browsable by a user to navigate a 3-dimensional representation of the meeting. Each participant may be represented by a video pane, a foreground object, or a 3-D geometric model of the participant'"'"'s face or body displayed in spatial relation to the other participants in a 3-dimensional arrangement analogous to the spatial arrangement of the meeting. The system may automatically re-orient the 3-dimensional representation as needed to best show the currently interesting event such as current speaker or may extend navigation controls to a user for manually viewing selected participants or nuanced interactions between participants.

Claims

1. A computer-executable method, comprising:

detecting devices in a meeting space;

selecting an array of the devices to capture spatial characteristics of the meeting space; and

combining media streams from the devices to create a composite media stream, wherein the composite media stream is browsable by a user to navigate a 3-dimensional representation of the meeting space.

2. The computer-executable method as recited in claim 1, wherein selecting the array of devices includes:

communicating with computers in the meeting space, including one of a desktop computer, a laptop computer, a notebook computer, or a pocket computer;

selecting cameras and microphones on the computers;

detecting a location and a view angle of at least some of the cameras; and

calibrating the locations and the view angles with each other to determine the spatial characteristics.

3. The computer-executable method as recited in claim 1, further comprising:

detecting microphones in the meeting space;

selecting an array of the cameras and the microphones to capture the spatial characteristics of the meeting space; and

combining video streams from the cameras and audio streams from the microphones to create a composite audio-video stream, wherein the composite audio-video stream is browsable by a user to navigate a 3-dimensional representation of the meeting space, including visual views displayed in a spatial arrangement, and spatial audio including stereo sound, and surround-sound.

4. The computer-executable method as recited in claim 1, wherein selecting an array of devices includes:

selecting cameras to obtain a view of at least some participants in a meeting of participants in the meeting space; and

corroborating information between the cameras to obtain at least a relative position of each participant associated with one of the views, the relative position being with respect to at least one of the other participants.

5. The computer-executable method as recited in claim 4, wherein combining video streams from the cameras to create the composite media stream further comprises producing a video stream that is browsable by a user to navigate a 3-dimensional representation of the participants in the meeting space, including at least the relative position of each participant with respect to the other participants.

6. The computer-executable method as recited in claim 1, further comprising:

capturing 3-dimensional shape information of a meeting participant via a depth camera or via an array of visible light cameras; and

encoding the 3-dimensional shape information of the meeting participant into the composite media stream for deriving a 3-dimensional model of the meeting participant for display.

7. A computer-executable method, comprising:

receiving a composite media stream containing component video streams, each component video stream showing a view of one of multiple meeting participants, the composite media stream containing spatial information including at least a spatial relationship of each meeting participant with respect to at least one other meeting participant; and

displaying the views contained in the component video streams in a 3-dimensional arrangement based on the spatial relationships among the meeting participants, wherein each view is shown in a perspective according to the 3-dimensional arrangement.

8. The computer-executable method as recited in claim 7, wherein displaying the views contained in the component video streams in a 3-dimensional arrangement based on the spatial relationships between the meeting participants further includes:

deriving a 3-dimensional model of a meeting participant from one of the views;

displaying the 3-dimensional model of the meeting participant in the 3-dimensional arrangement, wherein the 3-dimensional arrangement of the views is based on the spatial relationships between the meeting participants; and

wherein the 3-dimensional model of the meeting participant is displayed in compatible perspective with the 3-dimensional arrangement of the views.

9. The computer-executable method as recited in claim 7, wherein displaying the views contained in the component video streams in a 3-dimensional arrangement based on the spatial relationships between the meeting participants further includes:

segmenting one of the component video streams to separate a foreground from a visual background of the component video stream, wherein the foreground depicts a meeting participant; and

displaying the foreground depicting the meeting participant while discarding the visual background of the component video stream, when displaying the component video stream in the 3-dimensional arrangement of the views.

10. The computer-executable method as recited in claim 9, further comprising displaying an arbitrary visual background for the 3-dimensional arrangement of the views, wherein each view used with the arbitrary visual background consists of a foreground depicting a meeting participant from one of the component video streams minus the visual background of the component video stream.

11. The computer-executable method as recited in claim 7, further comprising dynamically orienting the 3-dimensional arrangement to facilitate emphasizing a currently interesting meeting event, while maintaining the spatial relationships in the 3-dimensional arrangement;

wherein emphasizing the currently interesting meeting event includes one of;

increasing a focus on the selected meeting participant;

enlarging the view of the selected meeting participant;

centering a view of the selected meeting participant on a 2-dimensional display;

decreasing a transparency of the view of the selected meeting participant;

fading-in to the selected meeting participant while fading-out at least one other meeting participant;

highlighting the selected meeting participant;

moving the view of the selected meeting participant to a reserved part of the 2-dimensional display;

or

displaying a notification associated with a display location of the currently interesting meeting event, wherein the notification includes one of a flashing border, a highlighted border, a cartoon-bubble, a text, a color, or an emoticon.

12. The computer-executable method as recited in claim 11, wherein dynamically orienting the 3-dimensional arrangement to emphasize a selected meeting participant includes automatically reorienting the 3-dimensional arrangement to emphasize each currently selected meeting participant, while maintaining the spatial relationships between the views of the other meeting participants as a context view.

13. The computer-executable method as recited in claim 12, further comprising when two meeting participants rapidly alternate speaking, then emphasizing both of the two meeting participants in a single viewing pane or in a view reserved for one or more currently selected meeting participants.

14. The computer-executable method as recited in claim 11, further comprising extending at least one navigation control to a user for one of:

selecting and emphasizing one of the meeting participants;

manually orienting at least the context view of the 3-dimensional arrangement;

adjusting an amount of display space allotted to each of a view of a currently interesting event and a context view, when simultaneously displaying the currently interesting event and the context view;

or

zooming, panning, rotating, or tilting the context view, to achieve a better view of a particular meeting participant.

15. The computer-executable method as recited in claim 11, wherein dynamically orienting the 3-dimensional arrangement to emphasize a selected meeting participant further includes orienting the 3-dimensional arrangement to indicate an interaction between a first meeting participant who is currently speaking and a second meeting participant, wherein the second meeting participant is

being addressed by the currently speaking meeting participant;

being observed by the currently speaking meeting participant;

or

interrupting the currently speaking meeting participant.

16. The computer-executable method as recited in claim 11, wherein displaying the views contained in the video stream in a 3-dimensional arrangement based on the spatial relationships among the meeting participants comprises:

displaying at least a part of a first view of a first meeting participant visually in front of at least a part of a second view of a second meeting participant in order to provide a perception of

3-dimensionality on a 2-dimensional display; and

rendering the parts of the first view as semi-transparent in order to enable a user to view the second view.

17. The computer-executable method as recited in claim 11, wherein the views displayed as a 3-dimensional arrangement based on the spatial relationships between the meeting participants portray a seating arrangement around a table;

and

wherein dynamically orienting the 3-dimensional arrangement to emphasize a selected meeting participant includes visually rotating the 3-dimensional arrangement around a center point of the portrayed seating arrangement.

18. The computer-executable method as recited in claim 17, wherein the views comprise frontal views of each meeting participant;

and

wherein when the portrayed seating arrangement implies a rear view of a meeting participant, then the frontal view of the meeting participant is horizontally inverted to preserve correct directionality of eye movements in the 3-dimensional arrangement.

19. A computing-device-executable browser for viewing a video depicting attendees of a meeting interacting with each other in an associated meeting space, comprising:

a media layer engine to extract a component video stream of each attendee from the video, each component video stream displaying a view of the associated attendee;

a 3-D assembler to extract spatial information from the video and arrange a viewing pane for each component video stream in a 3-dimensional arrangement based on the spatial information;

wherein the spatial information includes at least a spatial relationship of each attendee with respect to at least one other attendee;

wherein each pane is shown in a perspective view according to the 3-dimensional arrangement;

wherein verbal and visual interactions between the attendees are apparent within the 3-dimensional arrangement of the views; and

a 3-D navigation engine to change an orientation of the 3-dimensional arrangement to focus on an individual attendee while maintaining the spatial relationships between the viewing panes of the attendees.

20. The computing-device-executable browser as recited in claim 19, further comprising:

a mode selector to transition between

a first mode, wherein the 3-D navigation engine automatically reorients the 3-dimensional arrangement to emphasize each currently interesting event while maintaining the spatial relationships between the views of the other attendees; and

a second mode, wherein navigation controls extended to a user enable the user to manually orient the 3-dimensional arrangement to focus on a selected attendee or on a selected interaction between attendees while maintaining the spatial relationships between the views of the attendees.