Spatio-temporal light field cameras

• US 9,774,800 B2
• Filed: 10/01/2015
• Issued: 09/26/2017
• Est. Priority Date: 06/01/2012
• Status: Active Grant

• Current Assignee: Ostendo Technologies Inc.
• Original Assignee: Ostendo Technologies Inc.
• Inventors: El-Ghoroury, Hussein S., Alpaslan, Zahir Y., Cai, Jingbo, Maiers, Marty, Warner, Philip, McNeill, Dale A.
• Primary Examiner(s): Tissire, Abdelaaziz
• Application Number:
  US14/872,834
• Publication Number:
  US 20160182782A1
• Time in Patent Office: 726 Days
• CPC Class Codes: G02B 13/0085 employing wafer level optics
• CPC Class Codes: G02B 27/1066 for enhancing image performance, like resolution, pixel numbers, dual magnifications or dynamic range, by tiling, slicing or overlapping fields of view
• CPC Class Codes: G02B 27/1073 characterized by manufacturing or alignment methods
• CPC Class Codes: G02B 27/123 The splitting element being a lens or a system of lenses, including arrays and surfaces with refractive power
• CPC Class Codes: G02B 3/0012 characterised by the manufacturing method
• CPC Class Codes: G02B 3/0056 arranged along two different directions in a plane, e.g. honeycomb arrangement of lenses G02B3/0043 takes precedence; miniaturised objectives for electronic devices employing wafer level optics G02B13/0085
• CPC Class Codes: G02B 30/27 involving lenticular arrays
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 7/557 from light fields, e.g. from plenoptic cameras
• CPC Class Codes: H04N 13/207 using a single 2D image sensor
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 13/286 having separate monoscopic and stereoscopic modes
• CPC Class Codes: H04N 13/289 Switching between monoscopic and stereoscopic modes
• CPC Class Codes: H04N 23/00 Cameras or camera modules comprising electronic image sensors; Control thereof
• CPC Class Codes: H04N 23/16 Optical arrangements associated therewith, e.g. for beam-splitting or for colour correction
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/54 Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
• CPC Class Codes: H04N 23/57 Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
• CPC Class Codes: H04N 23/661 : Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/667 : Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/67 : Focus control based on electronic image sensor signals
• CPC Class Codes: H04N 23/68 : for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/80 : Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/815 : for controlling the resolution by using a single image
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 23/951 : by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 23/957 : Light-field or plenoptic cameras or camera modules
• CPC Class Codes: H04N 25/48 : Increasing resolution by shifting the sensor relative to the scene
• CPC Class Codes: H04N 5/04 : Synchronising for television systems using pulse code modulation H04N7/56

Abstract

Spatio-temporal light field cameras that can be used to capture the light field within its spatio temporally extended angular extent. Such cameras can be used to record 3D images, 2D images that can be computationally focused, or wide angle panoramic 2D images with relatively high spatial and directional resolutions. The light field cameras can be also be used as 2D/3D switchable cameras with extended angular extent. The spatio-temporal aspects of the novel light field cameras allow them to capture and digitally record the intensity and color from multiple directional views within a wide angle. The inherent volumetric compactness of the light field cameras make it possible to embed in small mobile devices to capture either 3D images or computationally focusable 2D images. The inherent versatility of these light field cameras makes them suitable for multiple perspective light field capture for 3D movies and video recording applications.

Claims

1. A light field camera comprising:

a micro photo-detector array device having a light detection surface and

a micro lens array of micro lens elements;

whereby the micro photo-detector array device and the micro lens array are assembled together and temporally angularly articulated as a single assembly around two orthogonal axes parallel to a plane of the light detection surface of the micro photo-detector array device;

wherein the temporal angular articulation is within a range of plus to minus a maximum angular articulation for each axis;

the periodicity of the temporal angular articulation being selected to enable temporal coverage of the maximum articulation angle within an image frame capture duration;

wherein the angular articulation is either temporally continuous or discrete and has a repetition rate that is proportional to and synchronized with an image capture frame rate whereby the maximum angular articulation around each of the two axes determines the full angular extent of the light field camera, the angular coverage shape it subtends, and its aspect ratio.

2. The light field camera of claim 1 wherein the micro photo-detector array device comprises an array of pixels whereby each of the pixels is a light detector that is individually addressable to output an electrical signal responsive to the color and intensity of light coupled into an aperture of the respective pixel.

3. The light field camera of claim 2 wherein the micro lens array is comprised of an array of lens elements that are precisely aligned relative to one another and relative to the micro photo-detector pixel array.

4. The light field camera of claim 3 wherein the light detection surface of the micro photo-detector array device is placed either at or a distance away from the focal plane of the micro lens elements, the placement either being fixed or variable using a z-axis actuator.

5. The light field camera of claim 4:

wherein each lens element of the micro lens array is associated and aligned with a plurality of pixels within a respective two dimensional pixel group array of the micro photo-detector array device, with each lens optically mapping light that impinges an aperture of the respective lens from a discrete set of directions within a light field defined by an angular extent of the respective lens onto the corresponding plurality of pixels within the two dimensional array pixel group of the micro photo-detector array device; and

wherein the light detection surface of the micro photo-detector array device is placed at a distance away from the focal plane of the micro lens elements causing the plurality of pixels comprising each of the pixel groups of the micro photo-detector array device to be shared collectively in recording the light field, thus enabling the light field camera to achieve angular and directional resolutions that are higher than if the micro photo-detector array device is placed at the focal plane of the micro lens elements.

6. The light field camera of claim 5:

wherein each lens element of the micro lens array is associated and aligned with a plurality of pixels within a respective two dimensional pixel group array of the micro photo-detector array device, with each lens optically mapping light that impinges an aperture of the respective lens from a discrete set of directions within a light field defined by an angular extent of the respective lens onto the corresponding plurality of pixels within the two dimensional array pixel group of the micro photo-detector array device; and

wherein each pixel within each of the pixel groups records a known weighted sum of the light field information from a multiplicity of directions, whereby the light field information can be resolved computationally to increase the angular and directional resolutions of the light field camera.

7. The light field camera of claim 5 wherein the recording of the pixels of the micro photo-detector array device together with the temporal angular articulation enables a desired angular and/or directional resolution using pixel groups comprising a fewer number of pixels than if there were no articulation, and providing more pixels for the formation of more pixel groups which consequently enables an increase in the achievable spatial resolution of the light field camera.