Stereo imaging using disparate imaging devices

• US 8,937,646 B1
• Filed: 10/05/2011
• Issued: 01/20/2015
• Est. Priority Date: 10/05/2011
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Baldwin, Leo B.
• Primary Examiner(s): Rao, Andy
• Assistant Examiner(s): ELAHI, SHAN E
• Application Number:
  US13/253,854
• Time in Patent Office: 1,203 Days
• Field of Search: 348/47
• US Class Current: 348/47
• CPC Class Codes: G02B 23/2415 Stereoscopic endoscopes
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: H04N 13/00 Stereoscopic video systems; Multi-view video systems; Details thereof
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 5/262 Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects

Abstract

Stereo imaging can be provided on an electronic device without the need for two matched, high resolution, large format cameras. In various embodiments, two or more cameras of disparate types can be used to capture stereoscopic images. Various processing algorithms can be used to adjust aspects such as the effective pixel size and depth of focus in order to provide for sufficient information to generate three-dimensional images without significant artifacts resulting from differences in the mismatched cameras.

Claims

1. A method of producing a stereoscopic image, comprising:

capturing a first image using a first camera having a first field of view and capturing a second image using a second camera having a second field of view, the second field of view at least partially overlapping the first field of view;

capturing a third image using a third camera positioned on the electronic device to have a third field of view at least partially overlapping the first field of view and the second field of view, the third camera having at least one different imaging characteristic than each of the first camera and the second camera;

generating, using at least one processor of the electronic device, a depth map including a depth value for each pixel location of the depth map using differences in a position of at least one object between the first image and the second image based at least in part on disparity information determined using at least one of edge matching, color matching, or texture matching of information in the first image and the second image;

locating pixels in the third image that correspond to pixel locations of the depth map; and

using, using the at least one processor of the electronic device, color information for each located pixel in the third image to determine a color value for a corresponding pixel location of the depth map,

wherein a stereoscopic image is able to be created using depth information from the first and second cameras and color information from the third camera.

2. The method of claim 1, wherein the third camera has a higher resolution than each of the first and second cameras.

3. The method of claim 1, wherein the third camera has a smaller depth of focus than each of the first and second cameras.

4. The method of claim 1, further comprising:

performing at least one of sub-sampling or binning pixels from the third image before using color information from the third image.

5. The method of claim 1, wherein the first camera and the second camera are configured to operate in a master/slave mode.

6. A method of producing a stereoscopic image, comprising:

determining a first depth of focus for a first camera having a first field of view based at least in part upon an aperture size of the first camera, the first camera being included in an electronic device;

calculating a second aperture size for a second camera of a different type than the first camera and having a second field of view such that a second depth of focus for the second camera is within a determined variance of the first depth of focus for the first camera;

capturing a first image using the first camera and a second image using the second camera, the first field of view at least partially overlapping the second field of view; and

generating, using a processor of the electronic device, a stereoscopic image by combining information in the first image and second image; and

displaying, after generating the stereoscopic image, the stereoscopic image on a display of the electronic device.

7. The method of claim 6, wherein generating the stereoscopic image includes combining the information in the first image and the second image to generate the stereoscopic image and storing the stereoscopic image for subsequent display.

8. The method of claim 6, further comprising:

storing the first image and the second image separately, the first image and the second image capable of being combined at a later time to generate the stereoscopic image.

9. The method of claim 6, further comprising:

performing at least one of matching a resolution of the first camera and the second camera or matching an effective pixel size of the first camera and the second camera.

10. The method of claim 9, further comprising:

determining an amount of sub-sampling to reduce a resolution of an image captured using the second camera to a resolution of an image captured using the first camera; and

determining an effective pitch of pixels of the second camera as a result of the sub-sampling,

wherein the aperture size of the second camera is further determined in part using the effective pitch of the pixels of the second camera.

11. The method of claim 6, further comprising:

selecting the aperture size of the first camera.

12. The method of claim 6, further comprising:

determining a target depth of focus value for the first and second cameras,

wherein the aperture size selected for the first camera is based at least in part upon the target depth of focus value.

13. The method of claim 6, further comprising:

adjusting the second field of view of the second camera to match the first field of view of the first camera at least a threshold amount.

14. A method of performing stereoscopic imaging using different types of camera, comprising:

capturing a first image using a first camera of an electronic device;

capturing a second image using a second camera separated a distance from the first camera on the electronic device, the second camera having a smaller depth of focus than the first camera and having at least one optical element positioned to separate image information captured by the second camera into at least two color components, the first and second cameras having at least partially overlapping fields of view;

analyzing a plurality of regions of the second image to determine a primary plane of focus for each region, the primary plane of focus being selected from a set of planes including a plane of focus for each of the at least two color components;

selecting intensity information from a first color component of the at least two color components corresponding to the determined primary plane of focus for each of the plurality of regions;

applying color information from a second color component of the at least two color components for each of the plurality of regions to the selected intensity information to generate a modified second image; and

generating one of a stereoscopic image by combining the first image and the modified second image for display using a stereoscopic image element or a stereoscopic image file for subsequent display.

15. The method of claim 14 further comprising:

storing the first image and modified image as separate image files capable of being subsequently combined to generate a stereoscopic image.

16. The method of claim 14, wherein the at least one optical element includes at least one lens operable to focus light of red, green, and blue components to different distances.

17. The method of claim 14, wherein the first camera further has at least one optical element positioned to separate image information captured by the first camera into at least two color components.

18. The method of claim 14, further comprising:

adjusting at least one of a field of view or a resolution of the modified second image before generating the stereoscopic image.

19. A computing device, comprising:

a device processor;

a first camera having a first field of view;

a second camera separated a distance on the computing device from the first camera and being of a different type than the first camera, the second camera having a second field of view that at least partially overlaps the first field of view; and

a memory device including instructions that, when executed by the device processor, enable the computing device to;

determine a first depth of focus for the first camera based at least in part upon a first aperture size of the first camera;

calculate a second aperture size for the second camera such that a second depth of focus for the second camera is within a determined variance of the first depth of focus for the first camera;

capture a first image using the first camera and a second image using the second camera; and

generate a stereoscopic image by combining information in the first image and second image; and

display, after generating the stereoscopic image, the stereoscopic image on a display of the computing device.

20. The computing device of claim 19, wherein generating the stereoscopic image includes combining the information in the first image and the second image to generate the stereoscopic image and storing the stereoscopic image for subsequent display.

21. The computing device of claim 19, wherein the instructions when executed further cause the computing device to:

store the first image and the second image separately, the first image and the second image capable of being combined at a later time to generate the stereoscopic image.

22. A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor of a computing device, cause the computing device to:

capture a first image using a first camera of an electronic device;

capture a second image using a second camera separated a distance from the first camera on the electronic device, the second camera having a smaller depth of focus than the first camera and having at least one optical element positioned to separate image information captured by the second camera into at least two color components, the first and second cameras having at least partially overlapping fields of view;

analyze a plurality of regions of the second image to determine a primary plane of focus for each region, the primary plane of focus being selected from a set of planes including a plane of focus for each of the at least two color components;

select intensity information from a first color component of the at least two color components corresponding to the determined primary plane of focus for each of the plurality of regions;

apply color information from a second color component of the at least two color components for each of the plurality of regions to the selected intensity information to generate a modified second image; and

generating a stereoscopic image for subsequent display on a display of the electronic device by combining the first image and the modified second image; and

displaying, after generating the stereoscopic image, the stereoscopic image on the display.

23. The non-transitory computer-readable storage medium of claim 22, wherein the instructions when executed further cause the computing device to:

store the first image and modified image as separate image files capable of being subsequently combined to generate a stereoscopic image.