Compound camera and methods for implementing auto-focus, depth-of-field and high-resolution functions

• US 20040196379A1
• Filed: 04/04/2003
• Published: 10/07/2004
• Est. Priority Date: 04/04/2003
• Status: Active Grant

• Current Assignee: STMicroelectronics Incorporated (STMicroelectronics NV)
• Original Assignee: STMicroelectronics Incorporated (STMicroelectronics NV)
• Inventors: Hong, Li, McGuinness, Peter, Chen, George Q.
• Granted Patent: US 7,425,984 B2
• Time in Patent Office: Days
• US Class Current: 348/218.1
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 2200/21 involving computational photography
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio

Abstract

A compound camera system for generating an enhanced virtual image having a large depth-of-field. The compound camera system comprises a plurality of component cameras for generating image data of an object and a data processor for generating the enhanced virtual image from the image data. The data processor generates the enhanced virtual image by generating a first component virtual image at a first depth plane, generating a second component virtual image at a second depth plane, and inserting first selected pixels from the first component virtual image into enhanced the virtual image and inserting second selected pixels from the second component virtual image into the enhanced virtual image.

Claims

1. A compound camera system capable of generating an enhanced virtual image having a large depth-of-field, said compound camera system comprising:

a plurality of component cameras capable of generating image data of an object; and

a data processor capable of receiving image data from said plurality of component cameras and generating said enhanced virtual image therefrom, wherein said data processor generates said enhanced virtual image by generating a first component virtual image at a first depth plane, generating a second component virtual image at a second depth plane, and inserting first selected pixels from said first component virtual image into enhanced said virtual image and inserting second selected pixels from said second component virtual image into said enhanced virtual image to thereby generate said enhanced virtual image.

2. The compound camera system as set forth in claim 1 wherein said data processor selects said first selected pixels from said first component virtual image and said second selected pixels from said second component virtual image by comparing an amount of blur of corresponding pixels from said first and second component virtual images and selecting pixels that have the least amount of blur from each of said first and second component virtual images.

3. The compound camera system as set forth in claim 2 wherein said data processor selects said pixels that have the least amount of blur by selecting pixels from each of said first and second component virtual images that have the least amount of color variance.

4. The compound camera system as set forth in claim 3 wherein said data processor generates said first component virtual image by back-projecting virtual pixel data (u,v) to generate point data (x,y,z) located at said first depth plane of said object and then projecting said point data (x,y,z) to generate first pixel data (u_l,v._l) located at the image plane of a first component camera.

5. The compound camera system as set forth in claim 4 wherein said data processor generates said first component virtual image by projecting point data (x,y,z) located at said first depth plane to generate second pixel data (u₂,v₂) located at the image plane of a second component camera.

6. The compound camera system as set forth in claim 5 wherein said data processor generates said first component virtual image by combining color of said first pixel data (u₁,v₁) and color of said second pixel data (u₂,v₂).

7. The compound camera system as set forth in claim 3 wherein said data processor generates said second component virtual image by back-projecting virtual pixel data (u,v) to generate point data (x,y,z) located at said second depth plane of said object and then projecting said point data (x,y,z) to generate first pixel data (u₁,v₁) located at the image plane of a first component camera.

8. The compound camera system as set forth in claim 7 wherein said data processor generates said second component virtual image by projecting point data (x,y,z) located at said second depth plane to generate second pixel data (u₂,v₂) located at the image plane of a second component camera.

9. The compound camera system as set forth in claim 8 wherein said data processor generates said second component virtual image by combining color of said first pixel data (u₁,v₁) and color of said second pixel data (u₂,v₂).

10. For use in a compound camera system, a method of generating an enhanced virtual image having a large depth-of-field, the method comprising the steps of:

generating image data of an object in each of a plurality of component cameras;

receiving image data from the plurality of component cameras;

generating a first component virtual image at a first depth plane;

generating a second component virtual image at a second depth plane; and

inserting first selected pixels from the first component virtual image into the enhanced virtual image and inserting second selected pixels from the second component virtual image into the enhanced virtual image to thereby generate the enhanced virtual image.

11. The method of generating an enhanced virtual image as set forth in claim 10 further comprising the steps of:

selecting the first selected pixels from the first component virtual image and the second selected pixels from the second component virtual image by comparing an amount of blur of corresponding pixels from the first and second component virtual images; and

inserting into the enhanced virtual image pixels that have the least amount of blur from each of the first and second component virtual images.

12. The method of generating an enhanced virtual image as set forth in claim 11 wherein the step of inserting into the enhanced virtual image pixels that have the least amount of blur comprises the sub-step of inserting into the enhanced virtual image pixels from each of the first and second component virtual images that have the least amount of color variance.

13. The method of generating an enhanced virtual image as set forth in claim 12 wherein the step of generating the first component virtual image comprises the sub-steps of:

back-projecting virtual pixel data (u,v) to generate point data (x,y,z) located at the first depth plane of the object; and

projecting the point data (x,y,z) to generate first pixel data (u₁,v₁) located at the image plane of a first component camera.

14. The method of generating an enhanced virtual image as set forth in claim 13 wherein the step of generating the first component virtual image further comprises the sub-step of projecting point data (x,y,z) located at the first depth plane to generate second pixel data (u₂,v₂) located at the image plane of a second component camera.

15. The method of generating an enhanced virtual image as set forth in claim 14 wherein the step of generating the first component virtual image further comprises the step of combining color of the first pixel data (u₁,v₁) and color of the second pixel data (u₂,v₂).

16. The method of generating an enhanced virtual image as set forth in claim 12 wherein the step of generating the second component virtual image comprises the sub-steps of:

back-projecting virtual pixel data (u,v) to generate point data (x,y,z) located at the second depth plane of the object; and

projecting the point data (x,y,z) to generate first pixel data (u₁,v₁) located at the image plane of a first component camera.

17. The method of generating an enhanced virtual image as set forth in claim 16 wherein the step of generating the second component virtual image further comprises the sub-step of projecting point data (x,y,z) located at the second depth plane to generate second pixel data (u₂,v₂) located at the image plane of a second component camera.

18. The method of generating an enhanced virtual image as set forth in claim 17 wherein the step of generating the second component virtual image comprises the step of combining color of the first pixel data (u₁,v₁) and color of the second pixel data (u₂,v₂).

19. A compound camera system capable of generating a high-resolution virtual image, said compound camera system comprising:

a plurality of component cameras capable of generating image data of an object; and

a data processor capable of receiving first image data from a first component camera and second image data from a second component camera and generating said high-resolution virtual image by

     1) back-projecting virtual pixel data (u,v) to generate point data (x,y,z) located at a depth, z=Z1, associated with a first object plane of said object;

     2) projecting said point data (x,y,z) to generate first pixel data (u₁,v₁) located at the image plane of said first image data; and

     3) projecting said point data (x,y,z) to generate second pixel data (u₂,v₂) located at the image plane of said second image data, wherein a pixel density of said virtual pixel data (u,v) is greater than a pixel density of said first pixel data (u₁,v₁) and second pixel data (u₂,v₂).

20. The compound camera system as set forth in claim 19 wherein said data processor generates said virtual image by combining color of said first pixel data (u₁,v₁) and color of said second pixel data (u₂,v₂).

21. The compound camera system as set forth in claim 20 wherein said data processor combines color of said first pixel data (u₁,v₁) and color of said second pixel data (u₂,v₂) by multiplying said first color by a first weighting factor to form a first product, multiplying said second color by a second weighting factor to form a second product, adding said first and second products, and dividing the summation of colors by the summation of weights.

22. The compound camera system as set forth in claim 21 wherein said weight is proportional to cos(φ

) where φ

is the angle between the virtual ray and the corresponding ray from a component camera.

23. The compound camera system as set forth in claim 19 wherein said data processor projects said virtual pixel data (u,v) to generate said point data (x,y,z) using a inverse Plane Projection Matrix and projects said point data (x,y,z) to generate said first pixel data (u₁,v₁) using a first Plane Projection Matrix.

24. The compound camera system as set forth in claim 23 wherein said data processor projects said point data (x,y,z) to generate said second pixel data (u₂,v₂) using a second Plane Projection Matrix.

25. The compound camera system as set forth in claim 24 wherein said data processor is further capable of adjusting a focus of said compound camera system by back-projecting said virtual pixel data (u,v) to generate said point data (x,y,z) located at a depth, z=Z2, associated with a second object plane of said object and projecting said point data (x,y,z) to generate said first pixel data (u₁,v₁) located at said image plane of said first image.

26. The compound camera system as set forth in claim 25 wherein said data processor is further capable of adjusting said focus of said compound camera system by projecting said point data (x,y,z) located at a depth, z=Z2, associated with said second object plane of said object to generate second pixel data (u₂,v₂) located at said image plane of said second image.