IMAGE CAPTURE USING THREE-DIMENSIONAL RECONSTRUCTION

• US 20120075432A1
• Filed: 09/27/2011
• Published: 03/29/2012
• Est. Priority Date: 09/27/2010
• Status: Abandoned Application

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Bilbrey, Brett, Culbert, Michael F., Simon, David I., Gere, David S., DeVaul, Rich, Sarwar, Mushtaq
• Application Number:
  US13/246,821
• Publication Number:
  US 20120075432A1
• Time in Patent Office: Days
• US Class Current: 348/48
• CPC Class Codes: G01J 4/00 Measuring polarisation of light
• CPC Class Codes: G02B 30/25 using polarisation techniques
• CPC Class Codes: G06T 7/557 from light fields, e.g. from plenoptic cameras
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/596 from three or more stereo images
• CPC Class Codes: G06V 40/166 using acquisition arrangements
• CPC Class Codes: G07C 9/37 using biometric data, e.g. fingerprints, iris scans or voice recognition
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps

Abstract

Embodiments may take the form of three-dimensional image sensing devices configured to capture an image including one or more objects. In one embodiments, the three-dimensional image sensing device includes a first image device configured to capture a first image and extract depth information for the one or more objects. Additionally, the image sensing device includes a second imaging device configured to capture a second image and determine an orientation of a surface of the one or more objects.

Claims

1. A three-dimensional imaging apparatus configured to capture at least one image including one or more objects, comprising:

a first sensor for capturing a polarized image, the first sensor including a first imaging device and a polarized filter associated with the first imaging device;

a second sensor for capturing a first non-polarized image;

a third sensor for capturing a second non-polarized image; and

at least one processing module for deriving depth information for the one or more objects utilizing at least the first non-polarized image and the second non-polarized image, the processing module further operative to combine the polarized image, the first non-polarized image, and the second non-polarized image to form a composite three-dimensional image.

2. The three-dimensional imaging apparatus of claim 1, wherein the first sensor is positioned between the second and third sensors such that a blind region of the first sensor is between blind regions of the second and third sensors.

3. The three-dimensional imaging apparatus of claim 1, wherein:

the first sensor is a luminance sensor;

the second sensor is a first chrominance sensor; and

the third sensor is a second chrominance sensor.

4. The three-dimensional imaging apparatus of claim 3, wherein a field of view of the first sensor is offset from both a field of view of the second sensor and a field of view of the third sensor.

5. The three-dimensional imaging apparatus of claim 3, wherein:

the polarized image is a polarized luminance image;

the first non-polarized image is a first chrominance image;

the second non-polarized image is a second chrominance image;

the at least one processing module is configured to generate a stereo disparity map from at least the first and second chrominance images; and

the at least one processing module is configured to derive depth information at least partially from the stereo disparity map.

6. The three-dimensional imaging apparatus of claim 5, further comprising a fourth sensor configured to capture a second luminance image;

wherein

the at least one processing module is further configured to refine the stereo disparity map based on the second luminance image.

7. The three-dimensional imaging apparatus of claim 1, wherein the polarized filter comprises an array of polarizing subfilters.

8. The three-dimensional imaging apparatus of claim 7, wherein:

the first sensor comprises at least one pixel; and

a first polarized subfilter of the array of polarizing subfilters overlays the at least one pixel.

9. The three-dimensional imaging apparatus of claim 8, wherein the first polarized subfilter of the array of polarizing subfilters has a different type of polarization than a second polarized subfilter of the array of polarizing subfilters.

10. The three-dimensional imaging apparatus of claim 8, wherein:

the at least one pixel receives polarized light reflected from an imaged object, the polarized light corresponding to a polarization type of the first polarized subfilter; and

the first sensor determines a surface normal of the imaged object by measuring a polarization of the light received by the at least one pixel.

11. The three-dimensional imaging apparatus of claim 10, further comprising:

at least a second pixel adjacent to the at least one pixel;

wherein

the second polarized subfilter overlays the at least a second pixel; and

the first sensor determines a surface normal of the imaged object by measuring a polarization of the light received by the at least a second pixel and comparing it to the polarization of the light received by the at least one pixel.

12. The three-dimensional imaging apparatus of claim 8, wherein the luminance imaging device includes at least one additional pixel that corresponds to an unpolarized area of the polarized filter.

13. The three-dimensional imaging apparatus of claim 12, wherein the polarized luminance image is a high dynamic range image created from a first luminance image recorded at least by the at least one pixel and a second luminance image recorded at least by the at least one additional pixel.

14. The three-dimensional imaging apparatus of claim 8, wherein:

the first sensor includes a microlens array; and

at least one microlens of the microlens array corresponds to the at least one pixel and is configured to focus light onto the at least one pixel.

15. The three-dimensional imaging apparatus of claim 1, wherein the at least one processing module is further configured to identify at least one face in the composite three-dimensional image utilizing at least one of the surface information or the depth information.

16. A three-dimensional imaging apparatus configured to capture at least one image including one or more objects, comprising:

a first sensor for capturing a polarized chrominance image and determining surface information for the one or more objects, the first sensor including a color imaging device and a polarized filter associated with the color imaging device;

a second sensor for capturing a first luminance image;

a third sensor for capturing a second luminance image; and

at least one processing module for deriving depth information for the one or more objects utilizing at least the first luminance image and the second luminance image and combining the polarized chrominance image, the first luminance image, and the second luminance image to form a composite three-dimensional image utilizing the surface information and the depth information.

17. A method for capturing at least one image of an object, comprising:

capturing a polarized image of the object;

capturing a first non-polarized image of the object;

capturing a second non-polarized image of the object;

deriving depth information for the object from at least the first non-polarized image and the second non-polarized image;

determining a plurality of surface normals for the object, the plurality of surface normals derived from the polarized image;

creating a three-dimensional image from the depth information and the plurality of surface normals.

18. The method of claim 17, wherein the operation of deriving depth information for the object comprises creating a stereo disparity from the first non-polarized image and the second non-polarized image.

19. The method of claim 17, further comprising:

determining, based on the surface normals, a simulated lighting of the object; and

altering the three-dimensional image to insert the simulated lighting of the object.

20. The method of claim 17, wherein the operation of determining a plurality of surface normals for the object comprises:

grouping each pixel of a pixel array into a subarray with at least one other pixel;

evaluating the polarized light received by the subarray; and

based on the evaluation, assigning a surface normal to a portion of the image recorded by the subarray.

21. The method of claim 17, wherein the polarized image and first non-polarized image are captured by a single sensor.