Compound camera and methods for implementing auto-focus, depth-of-field and high-resolution functions

• US 7,425,984 B2
• Filed: 04/04/2003
• Issued: 09/16/2008
• Est. Priority Date: 04/04/2003
• Status: Expired due to Fees

• Current Assignee: STMicroelectronics Incorporated (STMicroelectronics NV)
• Original Assignee: STMicroelectronics Incorporated (STMicroelectronics NV)
• Inventors: Hong, Li, McGuinness, Peter, Chen, George Q.
• Primary Examiner(s): VO, TUNG T
• Application Number:
  US10/407,490
• Publication Number:
  US 20040196379A1
• Time in Patent Office: 1,992 Days
• Field of Search: 348/159, 348/160, 348/161, 348/42, 348/49, 348/47, 348/218.1, 348/239, 382/154, 382/103, 382/294, 345/587
• US Class Current: 348/159
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 2200/21 involving computational photography
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio

Abstract

A compound camera system for generating an enhanced virtual image having a large depth-of-field. The compound camera system comprises a plurality of component cameras for generating image data of an object and a data processor for generating the enhanced virtual image from the image data. The data processor generates the enhanced virtual image by generating a first component virtual image at a first depth plane, generating a second component virtual image at a second depth plane, and inserting first selected pixels from the first component virtual image into enhanced the virtual image and inserting second selected pixels from the second component virtual image into the enhanced virtual image.

Claims

1. A compound camera system comprising:

a plurality of component cameras capable of generating image data of an object; and

a data processor capable of receiving first image data from a first of the component cameras and second image data from a second of the component cameras and generating a virtual image;

wherein the data processor is capable of generating the virtual image by

     1) back-projecting virtual pixel data to generate point data located at a depth associated with a first object plane of the object;

     2) projecting the point data to generate first pixel data located at an image plane of the first image data;

     3) projecting the point data to generate second pixel data located at an image plane of the second image data, wherein a pixel density of the virtual pixel data is greater than a pixel density of the first pixel data and the second pixel data; and

     4) combining first color of the first pixel data and second color of the second pixel data;

wherein the data processor is capable of combining the first color of the first pixel data and the second color of the second pixel data by multiplying the first color by a first weighting factor to form a first product, multiplying the second color by a second weighting factor to form a second product, and dividing a summation of the products by a summation of the weighting factors.

2. The compound camera system as set forth in claim 1 wherein each weighting factor is proportional to cos(φ

) where φ

is an angle between a virtual ray and a corresponding ray from one of the component cameras.

3. A compound camera system comprising:

a plurality of component cameras capable of generating image data of an object; and

a data processor capable of receiving first image data from a first of the component cameras and second image data from a second of the component cameras and generating a virtual image;

wherein the data processor is capable of generating the virtual image by

     1) back-projecting virtual pixel data to generate point data located at a depth associated with a first object plane of the object;

     2) projecting the point data to generate first pixel data located at an image plane of the first image data; and

     3) projecting the point data to generate second pixel data located at an image plane of the second image data, wherein a pixel density of the virtual pixel data is greater than a pixel density of the first pixel data and the second pixel data; and

wherein the data processor is capable of projecting the virtual pixel data to generate the point data using an inverse Plane Projection Matrix and is capable of projecting the point data to generate the first pixel data using a first Plane Projection Matrix.

4. The compound camera system as set forth in claim 3 wherein the data processor is capable of projecting the point data to generate the second pixel data using a second Plane Projection Matrix.

5. The compound camera system as set forth in claim 4 wherein the data processor is further capable of adjusting a focus of the compound camera system by back-projecting the virtual pixel data to generate the point data located at a second depth associated with a second object plane of the object and projecting the point data to generate the first pixel data located at the image plane of the first image.

6. The compound camera system as set forth in claim 5 wherein the data processor is further capable of adjusting the focus of the compound camera system by projecting the point data located at the second depth associated with the second object plane of the object to generate the second pixel data located at the image plane of the second image.

7. A method comprising:

receiving first image data of an object and second image data of the object; and

generating a virtual image of the object by;

back-projecting virtual pixel data to generate point data located at a depth associated with a first object plane of the object;

projecting the point data to generate first pixel data located at an image plane of the first image data;

projecting the point data to generate second pixel data located at an image plane of the second image data, wherein a pixel density of the virtual pixel data is greater than a pixel density of the first pixel data and the second pixel data; and

combining first color of the first pixel data and second color of the second pixel data, wherein combining the first color of the first pixel data and the second color of the second pixel data comprises multiplying the first color by a first weighting factor to form a first product, multiplying the second color by a second weighting factor to form a second product, and dividing a summation of the products by a summation of the weighting factors.

12. The method of claim 7, wherein each weighting factor is proportional to cos(φ

), where φ

is an angle between a virtual ray and a corresponding ray from one of the component cameras.

8. A method comprising:

receiving first image data of an object and second image data of the object; and

generating a virtual image of the object by;

back-projecting virtual pixel data to generate point data located at a depth associated with a first object plane of the object;

projecting the point data to generate first pixel data located at an image plane of the first image data; and

projecting the point data to generate second pixel data located at an image plane of the second image data, wherein a pixel density of the virtual pixel data is greater than a pixel density of the first pixel data and the second pixel data;

wherein projecting the virtual pixel data to generate the point data comprises using an inverse Plane Projection Matrix, and projecting the point data to generate the first pixel data comprises using a Plane Projection Matrix.

9. The method of claim 8, wherein projecting the point data to generate the second pixel data comprises using a second Plane Projection Matrix.

10. The method of claim 9, further comprising:

back-projecting the virtual pixel data to generate the point data located at a second depth associated with a second object plane of the object.

11. The method of claim 10, further comprising:

projecting the point data located at the second depth associated with the second object plane of the object to generate the second pixel data located at the image plane of the second image.