Multi-Camera System Consisting Of Variably Calibrated Cameras

• US 20160173869A1
• Filed: 12/15/2014
• Published: 06/16/2016
• Est. Priority Date: 12/15/2014
• Status: Abandoned Application

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Corporation
• Inventors: Wang, Ting-Chun, Srikanth, Manohar
• Application Number:
  US14/570,090
• Publication Number:
  US 20160173869A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

An apparatus comprises a main camera configured to produce a high quality image; at least two auxiliary cameras configured to produce images of lower quality; and electronic circuitry linked to the main camera and the at least two auxiliary cameras, the electronic circuitry comprising a controller having a memory and a processor, the electronic circuitry configured to operate on data pertaining to the high quality image and pertaining to the images of lower quality to produce an enhanced high quality image as output data.

Claims

1. An apparatus, comprising:

a main camera configured to produce a high quality image;

at least two auxiliary cameras configured to produce images of lower quality; and

electronic circuitry linked to the main camera and the at least two auxiliary cameras, the electronic circuitry comprising a controller having a memory and a processor, the electronic circuitry configured to operate on data pertaining to the high quality image and pertaining to the images of lower quality to produce an enhanced high quality image as output data;

wherein the processor utilizes computational photography algorithms that utilize dense correspondence and best fit homography techniques, the dense correspondence being based on data from the high quality image from the main camera and the images of lower quality from the at least two auxiliary cameras.

4. The apparatus of claim 1, wherein the output data produced comprises at least one of a high quality image data, metadata, and combination thereof.

5. The apparatus of claim 4, wherein the metadata comprises one or more of disparity maps, depth maps, occlusion maps, defocus maps, and sparse light fields.

6. The apparatus of claim 1, wherein the main camera assumes varying parameters related to the operation of the main camera.

7. The apparatus of claim 1, wherein the at least two auxiliary cameras have intrinsic and extrinsic operating parameters that are known for operating conditions.

8. The apparatus of claim 1, wherein the apparatus comprises a point-and-shoot camera, a mobile camera, a professional camera, a medical imaging device, a camera for use in an automotive, aviation, marine application, or a security camera.

2. (canceled)

3. (canceled)

9. A method, comprising:

acquiring data from a main camera, the data pertaining to a high quality image;

acquiring data from at least two auxiliary cameras, the data pertaining to at least two images of lower quality;

combining the data pertaining to the high quality image and the data pertaining to the at least two images of lower quality;

producing metadata pertaining to the acquired data;

enhancing the high quality image with the metadata; and

outputting the high quality image as image data;

wherein producing metadata comprises using computational photography algorithms embodied in a controller comprising a processor and a memory;

wherein using computational photography algorithms comprises using a dense correspondence algorithm to generate dense correspondence between the acquired data pertaining to the high quality image and the acquired data pertaining to the at least two images of lower quality; and

wherein a best fit homography transform is computed from the dense correspondence generated based on data from the high quality image from the main camera and the images of lower quality from the at least two auxiliary cameras.

13. The method of claim 9, wherein enhancing the high quality image with the metadata is one of controlled by a processor and controlled by a user.

10. (canceled)

11. (canceled)

12. (canceled)

14. A method, comprising:

acquiring data pertaining to a high quality image and data pertaining to at least two images of lower quality;

using a dense correspondence algorithm to generate dense correspondence between the data pertaining to the high quality image and the data pertaining to the at least two images of lower quality, the dense correspondence being based on data from the high quality image and the at least two images of lower quality;

linking correspondence points from the dense correspondence generated to disparity values;

grouping the disparity values into levels;

computing a best fit homography transform of the disparity values for each level; and

transforming the disparity values for each level to a high quality image.

15. The method of claim 14, wherein transforming the disparity values for each level to a high quality image is an affine transformation.

16. The method of claim 14, wherein transforming the disparity values for each level to a high quality image comprises starting the dense correspondence algorithm from a level that corresponds to zero disparity and proceeds towards the level of highest disparity.

17. The method of claim 14, wherein using the dense correspondence algorithm to generate dense correspondence comprises using electronic circuitry comprising a controller having a memory and a processor.

18. The method of claim 14, wherein a dense correspondence map established by the data pertaining to a high quality image and the data pertaining to at least two images of lower quality is used to reduce errors in a disparity map obtained using only the data pertaining to at least two images of lower quality.

19. A non-transitory computer readable storage medium, comprising one or more sequences of one or more instructions which, when executed by one or more processors of an apparatus, causes the apparatus to at least:

use a dense correspondence algorithm to generate dense correspondence between data pertaining to a high quality image and data pertaining to at least two images of lower quality;

link correspondence points from the dense correspondence generated to disparity values;

group the disparity values into levels; and

compute a best fit homography transform of the disparity values for each level.

20. The non-transitory computer readable storage medium of claim 19, comprising one or more sequences of one or more instructions which, when executed by one or more processors of an apparatus, further causes the apparatus to at least:

transform the disparity values for each level to a high quality image.

21. An apparatus, comprising:

a first camera configured to produce a high quality image;

a second camera configured to produce images of lower quality; and

electronic circuitry linked to the first camera and the second camera, the electronic circuitry comprising a controller having a memory and a processor, the electronic circuitry configured to operate on data pertaining to the high quality image and pertaining to the images of lower quality to produce an enhanced high quality image as output data;

wherein the processor utilizes computational photography algorithms that utilize dense correspondence and best fit homography techniques, the dense correspondence being based on data from the high quality image from the first camera and the images of lower quality from the second camera.

22. The apparatus of claim 21, wherein one of the first camera and the second camera is strongly calibrated and the other of the first camera and the second camera is weakly calibrated.

23. The apparatus of claim 21, wherein the first camera and the second camera are strongly calibrated′

relative to each other.

24. The apparatus of claim 23, wherein defocus information in the first camera is used as an additional cue to disambiguate disparity values to further enhance a disparity map.