Omni-directional camera design for video conferencing

• US 20040263611A1
• Filed: 06/26/2003
• Published: 12/30/2004
• Est. Priority Date: 06/26/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Cutler, Ross
• Granted Patent: US 7,298,392 B2
• Time in Patent Office: Days
• US Class Current: 348/36
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display

Abstract

An omni-directional camera array that is very small and unobtrusive and allows for higher pixel resolution for the face of meeting participants sitting or standing around a conference room table, as would be typical in a video conferencing situation. The camera system of the invention uses N cameras and an N sided mirror to capture images of the surrounding scene. The cameras are arranged to have a small distance between their virtual centers of projection. The images from the N cameras are stitched together to create a 360-degree panorama using a calibration surface. To stitch the images together in real-time, a static image stitcher is used, which stitches images together perfectly for objects on a calibration surface. The calibration surface is typically defined by the boundary of the table the camera is sitting on.

Claims

1. A camera system comprising:

an N-sided reflective surface that reflects its surroundings in 360 degrees;

N cameras each associated with a different side of said n-sided reflective surface, and aligned to have a small distance between virtual centers of projection relative to each other which provides minimal parallax error at a predefined distance from the cameras, each of n cameras aligned to capture a reflected image in its associated reflective surface; and

an image stitcher for stitching each of said reflected images taken by adjacent cameras together to create a panoramic image.

2. The camera system of claim 1 wherein said N cameras are tilted upward relative to the horizontal to capture an increased vertical field of view.

3. The camera system of claim 1 wherein said N cameras are tilted to capture a desired part of a scene.

4. The camera system of claim 3 wherein said N cameras each associated with a different side of said N-sided reflective surface are tilted upward in the range from six to eight degrees to capture a desired scene of objects that it is desired to capture while still being able to minimize the vertical field of view.

5. The camera system of claim 1 further comprising a mounting rod positioning said N-sided reflective surface above said N cameras each associated with a different side of said N-sided reflective surface.

6. The camera system of claim 5 wherein said mounting rod positioning said N-sided reflective surface above said N cameras increases said vertical field of view of said N cameras.

7. The camera system of claim 1 wherein said N cameras capture 360 degrees in the horizontal plane.

8. The camera system of claim 1 wherein said image stitcher uses a calibration surface of constant depth to stitch images captured by said N cameras.

9. The camera system of claim 8 wherein said calibration surface is predefined.

10. The camera system of claim 8 wherein said calibration surface is user defined.

11. The camera system of claim 8 wherein said calibration surface is defined by:

marking a point in each of said images;

determining a point to point correspondence in said images captured by each of N cameras to determine corresponding pixels in overlapping regions of each of said images;

generating a mapping table that defines how the pixels of the image captured by each of said N cameras is to be stitched into a panoramic image.

12. The camera system of claim 11 wherein said mapping table speeds up processing of generating subsequent panoramic images by using said mapping table each time new images are captured as the pixel correspondences do not have to be recomputed each time new images are captured in order to create a panorama.

13. A process of capturing video for teleconferencing and meeting recording, comprising the following process actions:

capturing images of an event in 360 degree with an omni-directional camera array, wherein said omni-directional camera comprises an N-sided mirror located above N cameras arranged to be equally spaced around the circumference of a circle in a circular fashion and tilted upward slightly from the horizontal plane, and positioned to have a small distance between their virtual centers of projection, each capturing an image reflected in a different side of said N-sided mirror; and

stitching together said images captured by each of N cameras using a calibration surface to create a composite image.

14. The process of claim 13 wherein each of said N cameras is a video sensor.

15. The process of claim 13 wherein said camera system is placed on a table and wherein edges of said table are used to define said calibration surface.

16. The process of claim 13 wherein said calibration surface is user-defined.

17. The process of claim 13 wherein said calibration surface is located at one or more prescribed depths from said camera system.

18. The process of claim 13 wherein said calibration surface is determined by the process actions of:

using a pattern created by a small array of LEDs placed at the prescribed distances from the cameras to determine the scale and translation between cameras by providing point-to-point correspondence between images captured by adjacent cameras;

using said point-to-point correspondences to generate a mapping table, mapping corresponding pixels between the images captured in one camera and the adjacent cameras in the overlapping regions; and

using said mapping table to determine pixel locations when stitching said images into a panoramic image.

19. The process of claim 13 wherein said N cameras are tilted upward to increase the vertical field of view for each camera.

20. The process of claim 13 wherein said N sided mirror is mounted on a rod above said cameras to increase each cameras vertical field of view.

21. The process of claim 19 wherein the tilt angle of said cameras is adjusted by tilt for each camera can be calibrated by adjusting the offset of the lens mount of the camera in the x and y directions.

22. A computer-readable medium having computer-executable instructions for viewing or recording a video-conference, said computer executable instructions comprising:

capturing images of persons in an event with an camera, wherein said camera comprises an N-sided mirror located above N cameras arranged to be equally spaced around at least a portion of a circle, such that said N cameras have a non-zero virtual center of projection, and tilted upward slightly from the horizontal plane, each capturing an image reflected in a different side of said N-sided mirror; and

stitching together said images captured by each of N cameras using a calibration surface to create a composite image.

23. The computer-readable medium of claim 22 wherein said N cameras in total cover 360 degrees in the horizontal plane.

24. The computer-readable medium of claim 22 wherein said N-cameras in total cover 180 degrees in the horizontal plane.

25. The computer-readable medium of claim 22 wherein said instruction for stitching together said images captured by each of N cameras using a calibration surface comprises sub-instructions to determine said calibration surface, said sub-instructions comprising:

using a pattern created by a small array of LEDs placed at the prescribed distances from the cameras to determine the scale and translation between cameras by providing point-to-point correspondence between images captured by adjacent cameras;

using said point-to-point correspondences to generate a mapping table, mapping corresponding pixels between the images captured in one camera and the adjacent cameras in the overlapping regions; and

using said mapping table to determine pixel locations when stitching said images into a panoramic image.