Combining views of a plurality of cameras for a video conferencing endpoint with a display wall

• US 8,164,617 B2
• Filed: 03/25/2009
• Issued: 04/24/2012
• Est. Priority Date: 03/25/2009
• Status: Active Grant

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Inventors: Mauchly, J. William
• Primary Examiner(s): Ramakrishnaiah, Melur
• Application Number:
  US12/411,300
• Publication Number:
  US 20100245535A1
• Time in Patent Office: 1,126 Days
• Field of Search: 348 1401- 1416, 348/36, 348/48, 382/275, 382/276
• US Class Current: 348/14.08
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 2013/0092 Image segmentation from stereoscopic image signals
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

A telepresence apparatus, a method of operating a telepresence apparatus, and a tangible computer readable storage medium in a telepresence apparatus that is configured with instructions that when executed cause operating of the telepresence apparatus. The telepresence apparatus includes video cameras districted co-planar with a display wall and capturing camera views of a scene. The camera views from the video processor combined to form a video signal for transmission to one or more remote endpoints, the video signal corresponding to a synthetic view from a point that is substantially behind the wall and as if the wall was not there.

Claims

1. A telepresence apparatus comprising:

a display wall;

a plurality of synchronized video cameras substantially co-planar with the display wall providing respective camera views together covering a scene;

an interface to accept the camera views;

a camera view processor system to generate a synthetic particular view from a particular point on the opposite side of the display wall than the scene and at a substantial distance therefrom, the camera view processor system including;

a depth map creator module to create an integrated depth map;

a view synthesizer module to combine the camera views according to the integrated depth map, including warping of particular camera views to form the synthetic particular view at a viewpoint, including scaling the sizes of objects or participants to maintain perspective and the relationship of object sizes according to their respective distances to the display wall; and

a video forming module to form video for transmission to one or more remote endpoints.

2. A telepresence apparatus, as recited in claim 1, wherein the substantial distance for the synthetic particular view is sufficiently far away from the display wall such that there is approximately orthogonal perspective, such that the relative sizes of participants of the scene appear the same in the generated particular view as their actual relative sizes.

3. A telepresence apparatus, as recited in claim 1, further comprising a codec module configured to compress the formed video for transmission.

4. A telepresence apparatus, as recited in claim 3, wherein the codec is coupled to a network to which a remote endpoint is coupled, such that the remote endpoint can receive formed and compressed video from the codec of the telepresence apparatus.

5. A telepresence apparatus, as recited in claim 3, further comprising a plurality of microphones and an audio processing module coupled to and configured to accept signals from the microphones and to generate audio time synchronized to the formed video, wherein the codec module includes one or more audio codecs to compress the generated audio to combine with the formed video for transmission to one or more remote endpoints.

6. A telepresence apparatus as recited in claim 1, wherein camera view processor system includes one or more programmable processors and a memory subsystem, and wherein each of the disparity creator module, the integrator module, and the view synthesizer module includes at least a respective portion of a tangible computer readable storage medium configured with instructions that when executed by one or more of the processors carry out the respective functions of the respective module.

7. A telepresence apparatus as recited in claim 6, wherein the one or more processors include one or more graphic processing units.

8. A telepresence apparatus as recited in claim 1, further comprising one or more calibration targets, wherein the camera view processor includes a calibrator module to ascertain camera parameters of the cameras.

9. A telepresence apparatus as recited in claim 1, wherein the integrated depth map creator module comprises:

a disparity creator module configured to create disparity maps between individual cameras by operating on sets of views of pairs of cameras; and

an integrator module configured to create an integrated depth map of the scene.

10. A telepresence apparatus as recited in claim 9, wherein the disparity creator module is configured to compare views from a selected pair of closely spaced cameras using a search, the search limited based by the known geometry, and uses dynamic programming to form a disparity map for the pair of cameras representing the depth of objects and/or participants in the scene in the camera views from the cameras.

11. A telepresence apparatus as recited in claim 10, wherein the disparity creator module is configured also to determine occlusions.

12. A telepresence apparatus as recited in claim 1, wherein synthesizer module includes a z-buffer that is filled according to the integrated depth map in order from closest to the plane of the display wall to further away.

13. A telepresence apparatus as recited in claim 1, wherein synthesizer module is configured to carry out plane sweeping, including, for each plane of any determined objects/participants in the integrated depth map, for each ray from the particular viewpoint to the object, determining which camera view(s) to use to determine the rendering of the synthesized view, including determining occlusions.

14. A telepresence apparatus as recited in claim 1, wherein the plane sweeping includes examining multiple camera views and determining agreement or disagreement amongst the camera views.

15. A method of operating a videoconference apparatus, comprising:

accepting camera signals from a plurality of time synchronized cameras aimed at a scene;

processing the camera signals to create video signals representing a particular view of the scene as viewed from particular point at a particular distance from the scene;

compressing the created video signals to form a compressed video for transmission to one or more remote endpoints; and

transmitting the compressed video to the one or more remote endpoints;

wherein the cameras are distributed substantially co-planar with and along a substantially planar display wall that is made up of one or more flat screen displays or one or more projections displays,

wherein the particular distance of the particular point is at least the width of the display wall, and

wherein processing the camera signals includes scaling of parts of the scene to create the signal of the particular view, wherein objects in camera signals from a particular camera that are further away from the particular camera are enlarged relative to objects in camera signals from another camera that are close to the other camera, such that for two objects or participants that are in at least one camera view, the difference in size of the two objects or participants due to difference in their respective distance from the display wall is less in the particular view than in any single camera view in which the two objects or participants are present.

16. A method as recited in claim 15, wherein the videoconference apparatus includes an audio processor coupled to a set of one or more microphones and to at least one loudspeaker, the method further comprising:

accepting and processing audio from the set of one or more microphones via the audio processor to form audio that is synchronized with the created video;

compressing the processed audio for transmitting with the compressed video to form compressed audio;

transmitting the compressed audio to the one or more remote endpoints with the transmitting of the compressed video;

receiving a compressed remote audio signal and a synchronized remote audio signal from a first remote endpoint;

decompressing the compressed remote video to form a decompressed remote video signal and decompressing the compressed remote audio to form a decompressed remote audio signal;

displaying the decompressed remote video signal on the display wall;

playing back the decompressed remote audio on the at least one loudspeaker via the audio processor.

17. A method as recited in claim 15, wherein the processing the camera signals comprises:

creating a depth map of the scene using the camera views;

synthesizing the particular view from the camera views according to the integrated depth map, including warping of particular camera views.

18. A method as recited in claim 17, further comprising calibrating to ascertain camera parameters of the cameras.

19. A method as recited in claim 17, wherein creating a depth map of the scene comprises:

creating disparity maps between individual cameras by operating on sets of views of pairs of cameras; and

integrating the created disparity maps to generate an integrated depth map of the scene.

20. A method as recited in claim 19, wherein the creating disparity maps includes comparing views from a selected pair of closely spaced, cameras including searching using a dynamic programming method, wherein the searching is limited based by a known geometry of the scene to form a disparity map for each pair of cameras representing the depth of objects and/or participants in the scene in the camera views from the cameras.

21. A method as recited in claim 19, wherein creating disparity maps includes determining occlusions.

22. A method as recited in claim 17, wherein synthesizing the particular view uses a z-buffer that is filled according to the integrated depth map in order from closest to the plane of the display wall to further away.

23. A method as recited in claim 17, wherein synthesizing the particular view includes plane sweeping, including, for each plane of any determined objects/participants in the integrated depth map, for each ray from the particular viewpoint to the object, determining which camera view(s) to use to determine the rendering of the synthesized view, including determining occlusions.

24. A method as recited in claim 17, wherein the plane sweeping includes examining multiple camera views and determining agreement or disagreement amongst the camera views.

25. A tangible computer readable storage medium in a telepresence apparatus, the tangible computer readable storage medium including instructions that when executed by one or more processors in the telepresence apparatus, the instructions being configured to operate the telepresence apparatus, the storage medium including;

one or more portions storing code that when executed causes accepting camera signals from a plurality of time synchronized cameras aimed at a scene;

one or more portions storing code that when executed causes processing the camera signals to create video signals representing a particular view of the scene as viewed from particular point at a particular distance from the scene; and

one or more portions storing code that when executed causes compressing the created video signals to form a compressed video for transmission to one or more remote endpoints,

wherein the cameras are distributed substantially co-planar with and along a substantially planar display wall that is made up of one or more flat screen displays or one or more projections displays,

wherein the particular distance of the particular point is at least the width of the display wall, and

wherein processing the camera signals includes scaling of parts of the scene to create the signal of the particular view, wherein objects in camera signals from a particular camera that are further away from the particular camera are enlarged relative to objects in camera signals from another camera that are close to the other camera, such that for two objects or participants that are in at least one camera view, the difference in size of the two objects or participants due to difference in their respective distance from the display wall is less in the particular view than in any single camera view in which the two objects or participants are present.