Systems and methods for panoramic imaging

• US 8,767,037 B2
• Filed: 03/17/2010
• Issued: 07/01/2014
• Est. Priority Date: 09/21/2007
• Status: Active Grant

• Current Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Original Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Inventors: Krishnan, Gurunandan, Nayar, Shree K.
• Primary Examiner(s): Rao, Andy
• Assistant Examiner(s): ELAHI, SHAN E
• Application Number:
  US12/726,029
• Publication Number:
  US 20110018964A1
• Time in Patent Office: 1,567 Days
• Field of Search: 348/36, 348 37- 39, 352/69, 359/725, 396/436
• US Class Current: 348/36
• CPC Class Codes: G02B 13/06 Panoramic objectives; So-called "sky lenses" including panoramic objectives having reflecting surfaces
• CPC Class Codes: G02B 3/0056 arranged along two different directions in a plane, e.g. honeycomb arrangement of lenses G02B3/0043 takes precedence; miniaturised objectives for electronic devices employing wafer level optics G02B13/0085
• CPC Class Codes: G02B 7/182 for mirrors
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: H04N 23/51 Housings
• CPC Class Codes: H04N 23/55 Optical parts specially adapted for electronic image sensors; Mounting thereof
• CPC Class Codes: H04N 23/58 Means for changing the camera field of view without moving the camera body, e.g. nutating or panning of optics or image sensors
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

Systems and techniques for panoramic imaging are disclosed herein. Systems for capturing a panoramic image of a panoramic field of view include a fisheye lens having a first field of view, a mirror optically coupled to the fisheye lens for reflecting an image of the a second field of view through the fisheye lens, the second field of view having a portion overlapping the first field of view, and a detector, optically coupled to the fisheye lens and the mirror, for capturing a first portion of the panoramic image corresponding to the first field of view and a second portion of the panoramic image corresponding to the second field of view.

Claims

1. A system for capturing a panoramic image of a panoramic field of view, comprising:

a fisheye lens having a direct field of view;

a mirror having a mirror field of view and being optically coupled to the fisheye lens for reflecting an image of the mirror field of view through the fisheye lens;

a detector, optically coupled to the fisheye lens and the mirror, for capturing a first portion of the panoramic image corresponding to the direct field of view and a second portion of the panoramic image corresponding to the mirror field of view, the first and second portions of the panoramic image each having a portion overlapping the other portion;

a processor operatively coupled to the detector; and

a memory operatively coupled to the processor, the memory storing program instructions that when executed by the processor, cause the processor to;

receive the captured first and second portions of the panoramic image from the detector,

combine the first and second portions of the panoramic image to form the panoramic image, and

smooth the overlapping portion of the first and second portions of the panoramic image by feathering.

2. The system of claim 1, wherein the mirror is a spherical mirror.

3. The system of claim 1, wherein the mirror is positioned coaxially with regard to the fisheye lens.

4. The system of claim 1,

wherein, an upper angle of the panoramic field of view is equal to an upper angle of the direct field of view;

a lower angle of the panoramic field of view is less than or equal to a lower angle of the mirror field of view; and

an angle of overlap between the direct field of view and the mirror field of view is equal to the difference between a lower angle of the direct field of view and an upper angle of the mirror field of view.

5. The system of claim 1, wherein the first portion of the panoramic image and the second portion of the panoramic image are combined using a scale-invariant feature transform algorithm.

6. The system of claim 1, wherein the memory stores program instructions that when executed by the processor, further cause the processor to calibrate the system.

7. The system of claim 1, wherein the memory stores program instructions that when executed by the processor, further cause the processor generate a table of correspondence between captured image pixel locations and panorama pixel locations.

8. The system of claim 1, wherein the memory stores program instructions that when executed by the processor, further cause the processor utilize the detector to capture the first portion of the panoramic image and capture the second portion of the panoramic image simultaneously.

9. A method for capturing a panoramic image of a panoramic field of view, comprising:

positioning a mirror having a mirror field of view a first distance from a fisheye lens having a direct field of view;

utilizing the fisheye lens to capture a first portion of the panoramic image corresponding to the direct field of view received directly through the fisheye lens;

utilizing the fisheye lens to capture a second portion of the panoramic image corresponding to the mirror field of view reflected from the mirror through the fisheye lens, the first and second portions of the panoramic image each having a portion overlapping the other portion;

combining the first and second portions of the panoramic image to compose the panoramic image; and

smoothing the overlapping portion of the first and second portions of the panoramic image by feathering.

10. The method of claim 9, further comprising:

locating one or more image pixels correspondingly found in both the first and second portions of the panoramic image;

determining calibration settings for the mirror;

determining calibration settings for the fisheye lens; and

generating a table of correspondence between a pixel location on the first portion of the panoramic image and a pixel location on the second portion of the panoramic image.

11. The method of claim 9, wherein the first and second portions of the panoramic image are captured utilizing a detector.

12. The method of claim 9, wherein the first and second portions of the panoramic image are captured simultaneously.

13. The method of claim 10, wherein the first and second portions of the panoramic image are combined utilizing a scale-invariant feature transform algorithm.

14. The method of claim 10, wherein determining the calibration settings for the mirror includes determining the center of curvature and the orientation of the mirror.

15. The method of claim 10, wherein determining the calibration settings for the fisheye lens includes determining a projection function of the fisheye lens.

16. The method of claim 10, wherein generating the table of correspondence further comprises:

determining an image location of one or more pixels;

calculating a corresponding panorama location for one or more pixels located only in the first portion of the panoramic image;

calculating a corresponding panorama location for one or more pixels located in only the second portion of the panoramic image; and

calculating a corresponding panorama location for one or more pixels located in both the first and second portions of the panoramic image.

17. The method of claim 10, wherein combining the first and second portions of the panoramic image further comprises using the correspondence table to map the first and second portions of the panoramic image into the panoramic image.

18. A system for capturing a panoramic image of a panoramic field of view, comprising:

a means for optically receiving a first portion of the panoramic image corresponding to a direct field of view;

a means for optically reflecting a second portion of the panoramic image corresponding to a mirror field of view through the means for optically receiving the first portion of the panoramic image;

a means for capturing the first and second portions of the panoramic image, the first and second portions of the panoramic image each having a portion overlapping the other portion;

a means for combining the first and second portions of the panoramic image to compose the panoramic image; and

a means for smoothing the overlapping portions of the first and second portions of the panoramic image by feathering.

19. The system of claim 18, wherein the means for optically receiving the first portion of the panoramic image is a fisheye lens.

20. The system of claim 18, wherein the means for optically reflecting the second portion of the panoramic image is a mirror.

21. The system of claim 18, wherein the means for capturing the first and second portions of the panoramic image is a detector.

22. The system of claim 18, wherein the means for combining the first and second portions of the panoramic image is a processor operatively coupled to a memory, the memory storing program instructions that when executed by the processor, cause the processor combine the first and second portions of the panoramic image to compose the panoramic image.