Spatio-temporal light field cameras

• US 9,681,069 B2
• Filed: 10/01/2015
• Issued: 06/13/2017
• Est. Priority Date: 06/01/2012
• Status: Active Grant

• Current Assignee: Ostendo Technologies Inc.
• Original Assignee: Ostendo Technologies Inc.
• Inventors: McNeill, Dale A., Alpaslan, Zahir Y., Maiers, Marty, Warner, Philip, El-Ghoroury, Hussein S., Cai, Jingbo
• Primary Examiner(s): Tissire, Abdelaaziz
• Application Number:
  US14/872,862
• Publication Number:
  US 20160191823A1
• Time in Patent Office: 621 Days
• CPC Class Codes: G02B 13/0085 employing wafer level optics
• CPC Class Codes: G02B 27/1066 for enhancing image performance, like resolution, pixel numbers, dual magnifications or dynamic range, by tiling, slicing or overlapping fields of view
• CPC Class Codes: G02B 27/1073 characterized by manufacturing or alignment methods
• CPC Class Codes: G02B 27/123 The splitting element being a lens or a system of lenses, including arrays and surfaces with refractive power
• CPC Class Codes: G02B 3/0012 characterised by the manufacturing method
• CPC Class Codes: G02B 3/0056 arranged along two different directions in a plane, e.g. honeycomb arrangement of lenses G02B3/0043 takes precedence; miniaturised objectives for electronic devices employing wafer level optics G02B13/0085
• CPC Class Codes: G02B 30/27 involving lenticular arrays
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 7/557 from light fields, e.g. from plenoptic cameras
• CPC Class Codes: H04N 13/207 using a single 2D image sensor
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 13/286 having separate monoscopic and stereoscopic modes
• CPC Class Codes: H04N 13/289 Switching between monoscopic and stereoscopic modes
• CPC Class Codes: H04N 23/00 Cameras or camera modules comprising electronic image sensors; Control thereof
• CPC Class Codes: H04N 23/16 Optical arrangements associated therewith, e.g. for beam-splitting or for colour correction
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/54 Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
• CPC Class Codes: H04N 23/57 Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
• CPC Class Codes: H04N 23/661 : Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/667 : Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/67 : Focus control based on electronic image sensor signals
• CPC Class Codes: H04N 23/68 : for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/80 : Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/815 : for controlling the resolution by using a single image
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 23/951 : by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 23/957 : Light-field or plenoptic cameras or camera modules
• CPC Class Codes: H04N 25/48 : Increasing resolution by shifting the sensor relative to the scene
• CPC Class Codes: H04N 5/04 : Synchronising for television systems using pulse code modulation H04N7/56

Abstract

Spatio-temporal light field cameras that can be used to capture the light field within its spatio temporally extended angular extent. Such cameras can be used to record 3D images, 2D images that can be computationally focused, or wide angle panoramic 2D images with relatively high spatial and directional resolutions. The light field cameras can be also be used as 2D/3D switchable cameras with extended angular extent. The spatio-temporal aspects of the novel light field cameras allow them to capture and digitally record the intensity and color from multiple directional views within a wide angle. The inherent volumetric compactness of the light field cameras make it possible to embed in small mobile devices to capture either 3D images or computationally focusable 2D images. The inherent versatility of these light field cameras makes them suitable for multiple perspective light field capture for 3D movies and video recording applications.

Claims

1. A light field camera comprising:

an micro photo-detector array device comprising a multiplicity of pixels;

a micro lens array aligned and physically bonded to a surface of the micro photo-detector array device, and

a two-axis gimbal supported on a gimbal base and having a bonding pad upon which the micro photo-detector array device with the micro lens array aligned and physically bonded to its surface is physically and electrically bonded;

two sets of electromechanical actuators aligned with the two axes of the gimbal to affect temporal angular articulation of said bonding pad around the two axes of the gimbal; and

a set of electrical contacts connecting said bonding pad to a set of micro photo-detector array device interface contacts.

2. The light field camera of claim 1 wherein the 2-axis gimbal is implemented using multiple silicon substrate layers for a 2-axis pivot of the gimbal, and a mechanical resistance spring that defines the neutral position of the gimbal relative to the gimbal base.

3. The light field camera of claim 2 wherein the mechanical resistance spring comprises the two sets of silicon bridges.

4. The light field camera of claim 2 wherein the 2-axis pivot is formed by the etching, using semiconductor lithography, of an inner ring and an outer ring around the periphery of the bonding pad, the rings being joined with a set of silicon bridges along each respective axis.

5. The light field camera of claim 4 wherein electrical connections between the bonding pad and the gimbal base are routed through the two sets of silicon bridges.

6. The light field camera of claim 1 wherein the two-axis gimbal comprises a spherical pivot on the backside of the bonding pad and a matched spherical socket on the topside of the gimbal base.

7. The light field camera of claim 6 wherein the electromechanical actuators are bonded to a backside of the bonding pad and to a topside of the gimbal base.

8. The light field camera of claim 6 wherein the electromechanical actuators are embedded in the spherical pivot and spherical socket.

9. The light field camera of claim 6 further comprising a mechanical resistance spring associated with the 2-axis gimbal in the form of a set of four spiral shaped springs defined by an etched spring layer and bonded at its inner end at the backside of the four corners of the bonding pad and at its outer end at the topside of the gimbal base.

10. The light field camera of claim 9 wherein electrical connections between the bonding pad and the set of the gimbal base are routed through the four spiral shaped springs.

11. The light field camera of claim 1 wherein the electromechanical actuators are driven by electrical drive signals to affect the temporal angular articulation, the electrical signals being either temporally continuous or discrete and having a repetition rate that is proportional to and synchronized with an image capture frame rate provided through the device interface contacts whereby said electrical drive signals include a correction value provided by a set of sensors bonded to a backside of the contact pad and a topside of the gimbal base.

12. The light field camera of claim 1 wherein the temporal angular articulation is affected by the electromechanical actuators having a maximum angular articulation value around each of the two axes of the gimbal.

13. The light field camera of claim 1 wherein the micro photo-detector array device comprises an array of pixels whereby each of the pixels is a few microns in size and is individually addressable through the device interface contacts to output an electrical signal that is responsive to color and intensity of light coupled into its aperture.

14. The light field camera of claim 13 wherein the micro lens array is comprised of an array of lens elements that is fabricated using ultraviolet imprint lithography at the wafer level to be precisely aligned relative to one another and relative to the micro photo-detector pixel arrays also fabricated at a wafer level using semiconductor wafer level alignment techniques.

15. The light field camera of claim 14 wherein the light detection surface of the micro photo-detector array device is placed either at or a distance away from the focal plane of the micro lens elements of the micro lens array, and the placement is either fixed by a design selection, or adjustable over a specified range using a z-axis actuator.

16. The light field camera of claim 15:

wherein each of the lens elements of the micro lens array is associated and precisely aligned with a plurality of pixels within a two dimensional pixel group array of the micro photo-detector array device, with each lens optically mapping the light that impinges its aperture from a discrete set of directions within a light field defined by an angular extent of said lens onto the corresponding plurality of pixels within the two dimensional pixel group array of the micro photo-detector array device, and

wherein the micro lens array is comprised of an array of lens elements that is fabricated using ultraviolet imprint lithography at the wafer level to be precisely aligned relative to one another and relative to the micro photo-detector pixel arrays also fabricated at a wafer level using semiconductor wafer level alignment techniques, and

wherein the placement of the light detection surface of the micro photo-detector array device at a distance away from the focal plane of the micro lens elements causes the plurality of pixels comprising each of the pixel groups of the micro photo-detector array device to be shared collectively in recording the light field, thus enabling the light field camera to achieve angular and directional resolutions that are higher than when the micro photo-detector array device is placed at the focal plane of the micro lens elements.

17. The light field camera of claim 16:

wherein each of the lens elements of the micro lens array is associated and precisely aligned with a plurality of pixels within a two dimensional pixel group array of the micro photo-detector array device, with each lens optically mapping the light that impinges its aperture from a discrete set of directions within a light field defined by an angular extent of said lens onto the corresponding plurality of pixels within the two dimensional pixel group array of the micro photo-detector array device, and

wherein the light field recording collective sharing of the pixels of the micro photo-detector array device causes each pixel within each of the pixel groups to record a known weighted sum of the light field information from a multiplicity of directions, whereby light field information can be resolved computationally to increase the angular and directional resolutions of the light field camera.

18. The light field camera of claim 16:

wherein each of the lens elements of the micro lens array is associated and precisely aligned with a plurality of pixels within a two dimensional pixel group array of the micro photo-detector array device, with each lens optically mapping the light that impinges its aperture from a discrete set of directions within a light field defined by an angular extent of said lens onto the corresponding plurality of pixels within the two dimensional pixel group array of the micro photo-detector array device, and

wherein the light field recording collective sharing of the pixels of the micro photo-detector array device together with the temporal angular articulation provides a desired angular and/or directional resolution using pixel groups of a fewer number of pixels, thus making more pixels for the formation of more pixel groups available, thereby enabling an increase in the achievable spatial resolution of the light field camera.

19. The light field camera of claim 1 wherein each of the lens elements of the micro lens array is associated and precisely aligned with a plurality of pixels within a two dimensional pixel group array of the micro photo-detector array device, with each lens optically mapping the light that impinges its aperture from a discrete set of directions within a light field defined by an angular extent of said lens onto the corresponding plurality of pixels within the two dimensional pixel group array of the micro photo-detector array device.

20. The light field camera of claim 19 wherein the spatial resolution and directional resolutions of the light field camera are determined by the size of the pixel group, the resolution and pixel pitch of the micro photo-detector array device, the lens pitch of the micro lens array, the angular extent and the maximum articulation angle.

21. The light field camera of claim 19:

wherein the micro lens array is comprised of an array of lens elements that is fabricated using ultraviolet imprint lithography at the wafer level to be precisely aligned relative to one another and relative to the micro photo-detector pixel arrays also fabricated at a wafer level using semiconductor wafer level alignment techniques, and

wherein the micro lens elements of the micro lens array are truncated to have their respective dimensional aspects match that of the two dimensional array of the pixels comprising the pixel groups and their optical center aligned with the center of their respective two dimensional array of the pixels comprising the pixel groups.

22. The light field camera of claim 19:

wherein the micro lens array is comprised of an array of lens elements that is fabricated using ultraviolet imprint lithography at the wafer level to be precisely aligned relative to one another and relative to the micro photo-detector pixel arrays also fabricated at a wafer level using semiconductor wafer level alignment techniques, and

wherein the micro lens elements of the micro lens array are truncated to have their respective size match dimensions of the two dimensional array of the pixels of the pixel groups and the respective optical center of the lens element at the center of the micro lens array aligned with the center of its respective pixel group and the lens elements away from the center of the micro lens array having their optical centers being offset from the center of their respective pixel group, with said offset gradually increasing for lens elements further away from the center of the micro lens array.

23. The light field camera of claim 19 wherein the temporal articulation causes any light directions within the expanded set of light directions to be detected during the time interval during which said light directions fall within the temporally articulated angular extent.

24. The light field camera of claim 19 wherein in the two dimensional array of the pixel groups, each of the individual pixel groups are individually addressable through the contact pad to detect the color, intensity and direction of the light that impinges the aperture of the light field camera, thereby enabling the light field camera to spatially detect the light that impinges its aperture.

25. The light field camera of claim 24:

wherein the micro photo-detector array device comprises an array of pixels whereby each of the pixels is a few microns in size and is individually addressable through the device interface contacts to output an electrical signal that is responsive to the color and intensity of light coupled into its aperture,

wherein the angular articulation temporally expands the angular extent by twice the maximum angular articulation and thereby temporally expands a detectable light field size and increases the number of light detection directions by the ratio of said angular extent expansion to the angular extent along each of the two articulation axes,

wherein the individual pixel color and intensity detection addressability and the temporal association of the pixels within a pixel group with the temporally expanded set of light directions enables the detection of the color and intensity of the light impinging from each of the temporally expanded set of directions, thereby enabling the light field camera to detect the color, intensity and direction of the light that impinges its aperture from the expanded light field defined by the discrete set of directions comprising the set of the temporally expanded set of light directions, and

wherein the directional, spatial, color and intensity detection addressability of the light field camera uses a multiple field data output from the light field camera whereby for each designated pixel group address within the spatial array of the pixel groups, at least one output data field is used to specify the direction of the light impinging the aperture of the light field camera and at least one output data field is used to specify the color and intensity of the light detected from that designated direction.

26. The light field camera of claim 24 wherein the angular articulation temporally expands the angular extent by twice the maximum angular articulation and thereby temporally expands a detectable light field size and increases the number of the light detection directions by the ratio of said angular extent expansion to the angular extent along each of the two articulation axes, the light field camera having spatial detection with a spatial resolution that is determined by the number of pixel groups along each axis of the two dimensional array of pixel groups and having a directional resolution that is determined by the number of individual light directions within the temporally expanded set of light directions.

27. The light field camera of claim 26 having an angular resolution which is defined as the angular separation between the individual light directions within the temporally expanded set of light directions and determined by the fractional value of the angular extent being addressed by each one pixel within the pixel group, and

wherein the angular articulation is either temporally continuous or discrete and has a repetition rate that is proportional to and synchronized with an image capture frame rate whereby the maximum angular articulation around each of the two axes determines the full angular extent of the light field camera, the angular coverage shape it subtends, and its aspect ratio, and

wherein the value of the maximum angular articulation is selected to affect an increase in the value of either the full angular extent or the spatial resolution for a given value of the angular resolution.

28. The light field camera of claim 26:

wherein the micro photo-detector array device comprises an array of pixels whereby each of the pixels is a few microns in size and is individually addressable through the device interface contacts to output an electrical signal that is responsive to the color and intensity of light coupled into its aperture,

wherein the micro lens array is comprised of an array of lens elements that is fabricated using ultraviolet imprint lithography at the wafer level to be precisely aligned relative to one another and relative to the micro photo-detector pixel arrays also fabricated at a wafer level using semiconductor wafer level alignment techniques,

wherein the angular articulation is either temporally continuous or discrete and has a repetition rate that is proportional to and synchronized with an image capture frame rate whereby the maximum angular articulation around each of the two axes determines the full angular extent of the light field camera, the angular coverage shape it subtends, and its aspect ratio, and

wherein the offset of the optical centers of the micro lens elements increases the directional resolution without sacrificing either the spatial resolution or the full angular extent of the light field camera.

29. The light field camera of claim 26:

wherein each of the lens elements of the micro lens array is associated and precisely aligned with a plurality of pixels within a two dimensional pixel group array of the micro photo-detector array device, with each lens optically mapping the light that impinges its aperture from a discrete set of directions within a light field defined by an angular extent of said lens onto the corresponding plurality of pixels within the two dimensional pixel group array of the micro photo-detector array device, and

wherein the angular articulation around each of the two axes is at least equal to the image capture frame rate multiplied by a factor that equals to ratio of the size in degrees of the full angular extent along each respective axis to the size in degrees of the angular extent, and

wherein in the two dimensional array of the pixel groups, each of the individual pixel groups are individually addressable through the contact pad to detect the color, intensity and direction of the light that impinges the aperture of the light field camera, thereby enabling the light field camera to spatially detect the light that impinges its aperture;

the light field camera being capable of recording a light field defined by spatial coordinates within the two dimensional array of the pixel groups, a set of angular directional coordinates defined by the two dimensional array pixel group and the temporal values of the articulation angle.

30. The light field camera of claim 19 wherein the angular articulation temporally expands the angular extent by twice the maximum angular articulation and thereby temporally expands a detectable light field size and increases the number of the light detection directions by the ratio of said angular extent expansion to the angular extent along each of the two articulation axes.

31. The light field camera of claim 30 wherein the association of the pixels within pixel groups with the temporally expanded set of directions and the individual pixel addressability enables the addressability of the temporally expanded set of directions, thereby enabling the light field camera to detect light that impinges its aperture from the expanded light field defined by the discrete set of directions comprising the set of the temporally expanded set of light directions.

32. The light field camera of claim 30 wherein the individual pixel color and intensity detection addressability and the temporal association of the pixels within a pixel group with the temporally expanded set of light directions enabling the detection of the color and the intensity of the light impinging from each of the temporally expanded set of directions, thereby enabling the light field camera to detect the color, intensity and direction of the light that impinges its aperture from the expanded light field defined by the discrete set of directions comprising the set of the temporally expanded set of light directions.

33. The light field camera of claim 30 having an angular resolution which is defined as the angular separation between the individual light directions within the temporally expanded set of light directions and determined by the fractional value of the angular extent being addressed by each one pixel within the pixel group.

34. The light field camera of claim 33:

wherein the micro photo-detector array device comprises an array of pixels whereby each of the pixels is a few microns in size and is individually addressable through the device interface contacts to output an electrical signal that is responsive to color and intensity of light coupled into its aperture,

wherein the micro lens array is comprised of an array of lens elements that is fabricated using ultraviolet imprint lithography at the wafer level to be precisely aligned relative to one another and relative to the micro photo-detector pixel arrays also fabricated at a wafer level using semiconductor wafer level alignment techniques, and

wherein the angular articulation is either temporally continuous or discrete and has a repetition rate that is proportional to and synchronized with an image capture frame rate whereby the maximum angular articulation around each of the two axes determines the full angular extent of the light field camera, the angular coverage shape it subtends, and its aspect ratio, and

wherein each of the micro lens elements is an optical system with at least one refracting surface, whereby the number of said refracting surfaces and their optical characteristics determine the angular extent and affect the value of the full angular extent and the angular resolution.

35. The light field camera of claim 34:

wherein the micro lens array is comprised of an array of lens elements that is fabricated using ultraviolet imprint lithography at the wafer level to be precisely aligned relative to one another and relative to the micro photo-detector pixel arrays also fabricated at a wafer level using semiconductor wafer level alignment techniques,

wherein each of the lens elements of the micro lens array is associated and precisely aligned with a plurality of pixels within a two dimensional pixel group array of the micro photo-detector array device, with each lens optically mapping the light that impinges its aperture from a discrete set of directions within a light field defined by an angular extent of said lens onto the corresponding plurality of pixels within the two dimensional pixel group array of the micro photo-detector array device, and

wherein the optical system of the micro lens elements is a Fourier optical system that is focused at infinity and maps the directional aspects of the light field that impinges its aperture within the angular extent onto the two dimensional arrays of the pixels.

36. The light field camera of claim 1 wherein the angular articulation is either temporally continuous or discrete and has a repetition rate that is proportional to and synchronized with an image capture frame rate whereby the maximum angular articulation around each of the two axes determines the full angular extent of the light field camera, the angular coverage shape it subtends, and its aspect ratio.

37. The light field camera of claim 36:

wherein each of the lens elements of the micro lens array is associated and precisely aligned with a plurality of pixels within a two dimensional pixel group array of the micro photo-detector array device, with each lens optically mapping the light that impinges its aperture from a discrete set of directions within a light field defined by an angular extent of said lens onto the corresponding plurality of pixels within the two dimensional pixel group array of the micro photo-detector array device, and

wherein the angular articulation around each of the two axes is at least equal to the image capture frame rate multiplied by a factor that equals the ratio of the size in degrees of the full angular extent along each respective axis to the size in degrees of the angular extent.

38. The light field camera of claim 1 wherein the light field camera is embedded in a mobile device.

39. The light field camera of claim 1 wherein the temporal angular articulation enables the light field camera to record a curved wavefront of the light field it captures, thus enabling the light field camera to be able to capture curved parallaxes of the light field.

40. The light field camera of claim 39 wherein the curved wavefront recording capability enables the light field camera to record information not only about the spatial and directional aspects of the light field but also about the phase of the wavefront of the light field it captures, thus enabling the light field camera to record a four dimensional spatio-directional space of the light field plus the phase of its wavefront.

41. The light field camera of claim 1, in plurality and in a tiled array.

42. The light field camera of claim 41 wherein each of the lens elements of the micro lens array is associated and aligned with a plurality of pixels within a two dimensional pixel group array of the micro photo-detector array device, with each lens optically mapping the light that impinges its aperture from a discrete set of directions within a light field defined by an angular extent of said lens onto the corresponding plurality of pixels within the two dimensional pixel group array of the micro photo-detector array device,

wherein the angular articulation temporally expands the angular extent by twice the maximum angular articulation and thereby temporally expands a detectable light field size and increases the number of light detection directions by the ratio of said angular extent expansion to the angular extent along each of the two articulation axes,

wherein the individual pixel color and intensity detection addressability and the temporal association of the pixels within a pixel group with the temporally expanded set of light directions enables the detection of the color and intensity of the light impinging from each of the temporally expanded set of directions, thereby enabling the light field camera to detect the color, intensity and direction of the light that impinges its aperture from the expanded light field defined by the discrete set of directions comprising the set of the temporally expanded set of light directions;

the light field camera with the collective set of the spatial arrays of the pixel groups, each with individual pixel color, intensity and direction detection addressability, providing an expanded spatial aperture to detect the light field that impinges across the expanded spatial aperture spanning the collective set of the spatial arrays of the tiled arrays.

43. The light field camera of claim 41 wherein the light field camera captures two dimensional light field images that are then computationally focused at a desired plane or curved surface or computationally focused within a desired depth volume having flat or curved surfaces.

44. The light field camera of claim 41 configured as a three dimensional light field camera with arbitrary spatial and directional resolutions.

45. The light field camera of claim 41 having operational modes for switching between operation as a three dimensional light field camera and operation as a two dimensional light field camera by adapting the desired operational mode.

46. The light field camera of claim 41 wherein the light field camera is embedded in a mobile device and networked to other mobile devices that also have a light field camera embedded therein, the mobile devices being interconnected either through wireless or wireline networks or through bulk data transfer using flash memory modules, thus making it possible to exchange light field camera output data with a multiplicity of such mobile devices that captured a different perspective of a viewing scene, and use such networked light field camera output data to create a collective light field of the viewed scene.

47. The light field camera of claim 46 wherein the networked mobile devices are either cell phones or computers, or alternatively the light field camera is integrated with its own network connectivity interface capabilities.

48. The light field camera of claim 41 configured to record either still or motion video images.

49. The light field camera of claim 1 wherein the light field camera captures two dimensional light field images that are then computationally focused at a desired plane or curved surface or computationally focused within a desired depth volume having flat or curved surfaces.

50. The light field camera of claim 1 configured as a three dimensional light field camera with arbitrary spatial and directional resolutions.

51. The light field camera of claim 1 having operational modes for switching between operation as a three dimensional light field camera and operation as a two dimensional light field camera by adapting the desired operational mode.

52. The light field camera of claim 1 wherein the light field camera is embedded in a mobile device and networked to other mobile devices that also have a light field camera embedded therein, the mobile devices being interconnected either through wireless or wireline networks or through bulk data transfer using flash memory modules, thus making it possible to exchange light field camera output data with a multiplicity of such mobile devices that captured a different perspective of a viewing scene, and use such networked light field camera output data to create a collective light field of the viewed scene.

53. The light field camera of claim 52 wherein the networked mobile devices are either cell phones or computers, or alternatively the light field camera is integrated with its own network connectivity interface capabilities.

54. The light field camera of claim 1 configured to record either still or motion video images.