Multi-device capture and spatial browsing of conferences

• US 9,065,976 B2
• Filed: 07/09/2013
• Issued: 06/23/2015
• Est. Priority Date: 10/06/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Hegde, Rajesh K., Zhang, Zhengyou, Chou, Philip A., Zhang, Cha, Liu, Zicheng, Junuzovic, Sasa
• Primary Examiner(s): Brandt, Christopher M
• Application Number:
  US13/938,144
• Publication Number:
  US 20140009562A1
• Time in Patent Office: 714 Days
• Field of Search: 348/14.08, 348/14.09, 709/204, 709/205, 715/751, 715/753, 715/755, 715/756, 715/757, 715/759
• US Class Current: 1/1
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827

Abstract

Multi-device capture and spatial browsing of conferences is described. In one implementation, a system detects cameras and microphones, such as the webcams on participants'"'"' notebook computers, in a conference room, group meeting, or table game, and enlists an ad-hoc array of available devices to capture each participant and the spatial relationships between participants. A video stream composited from the array is browsable by a user to navigate a 3-dimensional representation of the meeting. Each participant may be represented by a video pane, a foreground object, or a 3-D geometric model of the participant'"'"'s face or body displayed in spatial relation to the other participants in a 3-dimensional arrangement analogous to the spatial arrangement of the meeting. The system may automatically re-orient the 3-dimensional representation as needed to best show a currently interesting event.

Claims

1. A computing system comprising:

a memory device or storage device storing software; and

a processing device configured to execute the software, wherein the software causes the processing device to;

obtain multiple video streams from multiple cameras in a physical location where an activity occurs;

gather, from individual video streams, three-dimensional shape information of participants in the activity; and

encode the three-dimensional shape information into an output video stream.

2. The computing system of claim 1, wherein the software causes the processing device to:

determine spatial configuration information reflecting a spatial configuration of the participants in the activity; and

incorporate the spatial configuration information into the output video stream.

3. The computing system of claim 1, wherein the activity is a meeting or a conference.

4. The computing system of claim 1, wherein the software causes the processing device to:

send the output video stream to at least one other computing system that renders one or more views of the activity from the output video stream.

5. The computing system of claim 4, wherein the software causes the processing device to:

construct the output video stream such that the output stream is browsable by the at least one other computing system to navigate to different views of the physical location.

6. The computing system of claim 1, wherein the software causes the processing device to:

create a geometric model of an individual meeting participant.

7. The computing system of claim 6, wherein the geometric model is a geometric mesh.

8. The computing system of claim 1, wherein the software causes the processing device to:

calibrate locations of the multiple cameras;

use the calibrated locations to determine spatial characteristics of the physical location; and

incorporate the spatial characteristics of the physical location into the output video stream.

9. A computing system comprising:

a memory device or storage device storing software; and

a processing device configured to execute the software, wherein the software causes the processing device to;

receive a video stream of participants in an activity;

obtain, using the video stream, a first geometric model of a first individual participant; and

display a view showing a first representation of the first individual participant, the first representation being based on the first geometric model.

10. The computing system of claim 9, wherein the software causes the processing device to:

obtain a second geometric model of a second individual participant, the second geometric model also being obtained using the video stream; and

show a second representation of the second individual participant in the view with the first representation of the first individual participant, the second representation being based on the second geometric model.

11. The computing system of claim 10, wherein the view maintains a spatial relationship between the first individual participant and the second individual participant.

12. The computing system of claim 11, wherein the software causes the processing device to:

zoom, pan, rotate, or tilt the view while preserving the spatial relationship between the first individual participant and the second individual participant.

13. The computing system of claim 9, wherein the software causes the processing device to:

obtain the first geometric model by receiving the first geometric model in the video stream.

14. The computing system of claim 9, wherein the software causes the processing device to:

obtain the first geometric model by receiving spatial or three-dimensional information and computing the first geometric model using the spatial or three-dimensional information.

15. A method performed by at least one computing device, the method comprising:

obtaining multiple video streams from multiple cameras in a physical location where an activity occurs;

obtaining, using the multiple video streams, three-dimensional shape information of participants in the activity; and

computing a geometric model of an individual participant in the activity, the geometric model being computed using the three-dimensional shape information.

16. The method of claim 15, further comprising:

generating a view showing a representation of the individual participant, the representation being based on the geometric model.

17. The method of claim 16, further comprising:

panning the view responsive to a user input.

18. The method of claim 17, wherein the user input is a dragging input.

19. The method of claim 16, further comprising:

automatically reorienting the view to emphasize a particular event at the activity.

20. The method of claim 15, wherein the multiple cameras comprise infrared depth cameras.