Foveated wide-angle imaging system and method for capturing and viewing wide-angle images in real time

• US 20040061787A1
• Filed: 09/30/2002
• Published: 04/01/2004
• Est. Priority Date: 09/30/2002
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Cohen, Michael, Liu, Zicheng
• Granted Patent: US 7,084,904 B2
• Time in Patent Office: Days
• US Class Current: 348/218.1
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display

Abstract

The present invention includes a foveated wide-angle imaging system and method for capturing a wide-angle image and for viewing the captured wide-angle image in real time. In general, the foveated wide-angle imaging system includes a foveated wide-angle camera system having multiple cameras for capturing a scene and outputting raw output images, a foveated wide-angle stitching system for generating a stitch table, and a real-time wide-angle image correction system that creates a composed warp table from the stitch table and processes the raw output images using the composed warp table to correct distortion and perception problems. The foveated wide-angle imaging method includes using a foveated wide-angle camera system to capture a plurality of raw output images, generating a composed warp table, and processing the plurality of raw output images using the composed warp table to generate a corrected wide-angle image for viewing.

Claims

1. A foveated wide-angle imaging system for capturing and viewing a wide-angle image of a scene, comprising:

a foveated wide-angle camera system having a plurality of cameras that outputs raw output images; and

a foveated wide-angle image stitching system that processes the raw output images to generate a stitch table,

wherein the stitch table includes pixel coordinates for a preliminary foveated wide-angle image produced by stitching together the raw output image.

2. The foveated wide-angle imaging system as set forth in claim 1, further comprising a real-time wide-angle image correction system that processes the stitch table to correct any distortion and perception problems and errors in the preliminary foveated wide-angle image.

3. The foveated wide-angle imaging system as set forth in claim 2, further comprising a corrected wide-angle image generated from the preliminary foveated wide-angle image after processing by the real-time wide-angle image correct system.

4. The foveated wide-angle imaging system as set forth in claim 1, wherein the plurality of cameras further comprises a center camera located at a center of a foveated wide-angle camera system having a first field-of-view.

5. The foveated wide-angle imaging system as set forth in claim 4, wherein the foveated wide-angle camera system is approximately a hemispherical shape.

6. The foveated wide-angle imaging system as set forth in claim 5, wherein the plurality of cameras further comprises:

a left set of N cameras located on a left of the center camera; and

a right set of N right cameras located on a right of the center camera.

7. The foveated wide-angle imaging system as set forth in claim 6, wherein the left set of N cameras and the right set of N cameras have an increasing field-of-view that is wider than the first field-of-view as referenced moving away from the center camera along the hemispherical shape.

8. The foveated wide-angle imaging system as set forth in claim 7, wherein N equals three.

9. The foveated wide-angle imaging system as set forth in claim 1, wherein the foveated wide-angle stitching system further comprises a pixel mapping module that selects a pixel and maps the pixel from the raw output images to the preliminary foveated wide-angle image.

10. The foveated wide-angle imaging system as set forth in claim 9, wherein the foveated wide-angle stitching system further comprises a weights generator that generates a weight for the pixel.

11. The foveated wide-angle imaging system as set forth in claim 3, wherein the real-time wide-angle image correction system further comprises a warp table generation module that generates a warp table.

12. The foveated wide-angle imaging system as set forth in claim 11, wherein the real-time wide-angle image correction system further comprises a composed warp table generated by composing the stitch table with the warp table.

13. The foveated wide-angle imaging system as set forth in claim 12, further comprising a real-time image warping system that applies the composed warp table to the raw output images to generate the corrected wide-angle image.

14. A foveated wide-angle camera system for capturing a wide-angle image of a scene, comprising:

a plurality of cameras arranged in a hemispherical arrangement, comprising;

a center camera located at a center of the hemispherical arrangement having a first field-of-view;

a set of left cameras located on a left side of the center camera;

a set of right cameras located on a right side of the center camera;

wherein the left and right sets of cameras have an increasingly wider field-of-view such that cameras nearest the center camera have narrower fields-of-view and camera furthest from the center camera have wider fields-of-view.

15. The foveated wide-angle camera system as set forth in claim 14, wherein the first field-of-view is narrower than any of the cameras in the left set of cameras or the right set of cameras.

16. The foveated wide-angle camera system as set forth in claim 15, wherein the first field-of-view is equal to approximately twenty degrees.

17. A foveated wide-angle camera system for generating a preliminary foveated wide-angle image, comprising:

a center camera disposed near a center of the system for capturing a center raw output image;

a second camera disposed on a left side of the center camera for capturing a first left raw output image;

a third camera disposed on a right side of the center camera for capturing a first right raw output image;

a fourth camera disposed on a left side of the second camera for capturing a second left raw output image; and

a fifth camera disposed on a right side of the third camera for capturing a second right raw output image.

18. The foveated wide-angle camera system as set forth in claim 17, wherein the preliminary foveated wide-angle image includes the center raw output image, the first left raw output image, the second left raw output image, the first right raw output image and the second right raw output image.

19. The foveated wide-angle camera system as set forth in claim 17, wherein the center raw output image has a higher pixel density than other raw output images.

20. The foveated wide-angle camera system as set forth in claim 17, wherein the center camera has a field-of view that is narrower than the second, third, fourth and fifth cameras.

21. The foveated wide-angle camera system as set forth in claim 20, wherein the center camera field-of-view is approximately twenty degrees.

22. The foveated wide-angle camera system as set forth in claim 20, wherein the second camera has a field-of-view that is narrower than the fourth camera and the third camera has a field-of-view that is narrower than the fifth camera.

23. The foveated wide-angle camera system as set forth in claim 22, wherein the second camera field-of-view is approximately identical to the third camera field-of view and the fourth camera field-of-view is approximately identical to the fifth camera field-of-view.

24. The foveated wide-angle camera system as set forth in claim 23, wherein the second and third camera field-of-views are approximately equal to forty-five degrees and the fourth and fifth camera fields-of-view are approximately equal to sixty-four degrees.

25. A real-time wide-angle image correction system for viewing wide-angle images, comprising:

a stitch table generated from raw output images captured by a foveated wide-angle camera;

a warp generation module for generating a warp table;

a composed warp table created by composing the stitch table with the warp table; and

a real-time image warping system that processes the raw output images to correct distortion and perception problems to produce a corrected wide-angle image.

26. The real-time wide-angle image correction system as set forth in claim 25, wherein the stitch table contains pixel coordinates of pixels in the raw output images along with a weight for each pixel.

27. The real-time wide-angle image correction system as set forth in claim 25, wherein the warp table generation module uses a warping function.

28. The real-time wide-angle image correction system as set forth in claim 27, wherein the warping function is a Spatially Varying Uniform Scaling function belonging to a parametric class of image warping functions.

29. A method for capturing and viewing wide-angle images, comprising:

using a foveated wide-angle camera to capture a plurality of raw output images;

generating a composed warp table using the plurality of raw output images and a warping function; and

processing the plurality of raw output images using the composed warp table to generate a corrected wide-angle image.

30. The method as set forth in claim 29, wherein generating a composed warp table further comprises:

generating a stitch table from the plurality of raw output images that maps pixels in the plurality of raw output images to a preliminary foveated wide-angle image; and

using the stitch table to generate the composed warp table.

31. The method as set forth in claim 30, further comprising:

generating a warp table; and

composing the warp table and the stitch table to generate the composed warp table.

32. The method as set forth in claim 29, wherein the stitch table contains pixel coordinates and weights.

33. A computer-readable medium having computer-executable instructions for performing the method recited in claim 29.

34. A computer-readable medium having computer-executable instructions for capturing a wide-angle image of a scene, comprising:

directing a foveated wide-angle camera at the scene, the foveated wide-angle camera containing a plurality of cameras with at least a portion of the plurality of cameras having different fields-of-view;

producing a raw image from each of the plurality of cameras of the foveated wide-angle camera to generate raw output images;

creating a stitch table from the raw output images, the stitch table mapping pixels in the raw output images to pixels in a preliminary foveated wide-angle image.

35. The computer-readable medium as set forth in claim 34, wherein a center camera of the plurality of cameras has a narrower field-of-view than the others of the plurality of cameras.

36. The computer-readable medium as set forth in claim 34, further comprising:

generating a warp table from the preliminary foveated wide-angle image; and

composing the stitch table and the warp table to generated a composed warp table.

37. The computer-readable medium as set forth in claim 36, further comprising processing the raw output images using the composed warp table to create a corrected wide-angle image.

38. A method for viewing wide-angle images of a scene, comprising:

inputting raw output images captured by a foveated wide-angle camera system;

using a stitch table to stitching the raw output images together to generate a preliminary foveated wide-angle image;

using the preliminary foveated wide-angle image to generate a warp table;

composing the stitch table and the warp table to produce a composed warp table; and

processing the raw output images in real time using the composed warp table to generate a corrected wide-angle image.