Light Field Data Acquisition

• US 20120327222A1
• Filed: 09/04/2012
• Published: 12/27/2012
• Est. Priority Date: 12/08/2008
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Lytro Incorporated
• Inventors: Ng, Yi-Ren, Pitts, Colvin, Knight, Timothy
• Granted Patent: US 8,724,014 B2
• Time in Patent Office: Days
• US Class Current: 348/135
• CPC Class Codes: G02B 27/0075 with means for altering, e.g. increasing, the depth of field or depth of focus
• CPC Class Codes: G02B 7/282 Autofocusing of zoom lenses
• CPC Class Codes: G06T 2200/21 involving computational photography
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: H04N 23/00 Cameras or camera modules comprising electronic image sensors; Control thereof
• CPC Class Codes: H04N 23/95 Computational photography systems, e.g. light-field imaging systems

Abstract

A light field data acquisition device includes optics and a light field sensor to acquire light field image data of a scene. In at least one embodiment, the light field sensor is located at a substantially fixed, predetermined distance relative to the focal point of the optics. In response to user input, the light field acquires the light field image data of the scene, and a storage device stores the acquired data. Such acquired data can subsequently be used to generate a plurality of images of the scene using different virtual focus depths.

Claims

1. A light field imaging device for acquiring light field image data of a scene, the device comprising:

optics, wherein the optics includes an optical path and a focal point, wherein the focal point is associated with a focal length of the optics;

a light field sensor, located (i) in the optical path of the optics to acquire light field image data and (ii) at a substantially fixed, predetermined distance relative to the focal point of the optics, wherein the predetermined distance is substantially independent of the scene, and wherein an optical depth of field of the optics with respect to the light field sensor extends to a depth that is closer than optical infinity; and

a user interface configured to receive a user input, wherein, in response to the user input, the light field sensor acquires the light field image data of the scene; and

a storage device, coupled to the light field sensor, configured to store the acquired light field image data of the scene.

2. The light field imaging device of claim 1, wherein the optics is configurable to include a plurality of different focal lengths having associated focal points.

3. The light field imaging device of claim 2, wherein the substantially fixed, predetermined distance of the light field sensor, relative to the focal point of the optics, changes in accordance with the focal length of the optics.

4. The light field imaging device of claim 1, wherein the optics comprises a zoom lens system having a plurality of zoom positions.

5. The light field imaging device of claim 4, further comprising a mechanical system, coupled to the light field sensor, configured to maintain the light field sensor unit at the same fixed, predetermined distance relative to the focal point of the optics for the plurality of the zoom positions.

6. The light field imaging device of claim 4, wherein the zoom lens system comprises a continuous zoom providing the plurality of zoom positions.

7. The light field imaging device of claim 4, wherein the light field sensor is configured to maintain the predetermined distance relative to the focal point of the optics for the plurality of the zoom positions of the zoom lens system.

8. The light field imaging device of claim 4, further comprising a mechanical system, coupled to at least one of the light field sensor and the optics, configured to maintain the light field sensor unit at the predetermined distance relative to the focal point of the optics for the plurality of the zoom positions of the zoom lens system.

9. The light field imaging device of claim 8, further comprising:

a spatial adjustment unit, coupled to at least one of the light field sensor and the optics, configured to responsively move the light field sensor unit; and

control circuitry, coupled to the spatial adjustment unit, configured to control the spatial adjustment unit to maintain the light field sensor unit at the predetermined distance relative to the focal point of the optics.

10. The light field imaging device of claim 9, further comprising memory, coupled to the control circuitry, configured to store data representative of a plurality of locations of the light field sensor corresponding to a plurality of associated zoom positions.

11. The light field imaging device of claim 9, further comprising memory, coupled to the control circuitry, wherein the memory comprises at least one of a database and a lookup table correlating a plurality of locations of the light field sensor according to a plurality of associated zoom positions.

12. The light field imaging device of claim 11, wherein the control circuitry is configured to access the at least one of a database and a lookup table having data representative of the location based on the zoom position of the zoom lens system, and wherein the control circuitry is further configured to generate at least one control signal applied to the spatial adjustment unit to responsively move the light field sensor unit.

13. The light field imaging device of claim 1, further comprising a mechanical system, coupled to the light field sensor and/or the optics, configured to maintain the light field sensor unit at the predetermined distance relative to the focal point of the optics for a plurality of focuses of the optics.

14. The light field imaging device of claim 13, wherein the mechanical system is active.

15. The light field imaging device of claim 1, wherein the optics is configurable to include a plurality of zoom positions which provide a plurality of different focal lengths having associated focal points, and wherein the device further comprises a mechanical system, coupled to the light field sensor, configured to maintain the light field sensor unit at the same fixed, predetermined distance relative to the focal point of the optics for the plurality of focuses.

16. The light field imaging device of claim 15, wherein the mechanical system comprises at least one selected from the group consisting of an active mechanical system and a passive mechanical system.

17. The light field imaging device of claim 1, wherein the processing circuitry is configured to compute output images having an array of output pixels, and wherein the light field sensor comprises (i) a microlens array having a plurality of microlenses and (ii) a sensor having an array of sensor pixels, and wherein the predetermined distance is between (0.7*N_usable*W_MLA*F#_optics*m)/(W_output) and (3.0*N_usable*W_MLA*F#_optics*m)/(W_output),

where:

N_usable is equal to the number of pixels containing usable directional information across a microlens disk image that is formed, through each microlens, on the sensor, or are located under a microlens,

W_MLA is representative of a number of microlenses across a width of the microlens array,

W_output is representative of a number of pixels across a width of the array of output pixels,

m is equal to a distance between the centers of two neighboring microlenses in the microlens array, and

F#_optics is equal to an F-number of the optics.

18. The light field imaging device of claim 1, wherein the light field sensor comprises (i) a microlens array having a plurality of microlenses and (ii) a sensor having an array of pixels, and wherein the predetermined distance is greater than (m*F#_optics),

where:

m is equal to the distance between the centers of two neighboring microlenses in the microlens array, and

F#_optics is equal to an F-number of the optics.

19. The light field imaging device of claim 1, wherein the light field sensor comprises (i) a microlens array having a plurality of microlenses and (ii) a sensor having an array of pixels, and wherein the predetermined distance is less than (N_usable*m*F#_optics),

where:

N_usable is equal to the number of pixels containing usable directional information across a microlens disk image that is formed, through each microlens, on the sensor, or are located under a microlens,

m is equal to the distance between the centers of two neighboring microlenses in the microlens array, and

F#_optics is equal to an F-number of the optics.

20. A light field imaging device for acquiring light field image data of a subject in a scene, wherein the subject is in a focal plane, the device comprising:

optics, wherein the optics comprises an optical path, an optical depth of field, and an autofocus mechanism, and wherein the autofocus mechanism selects among a plurality of focuses including a first focus related to a focal plane of the subject;

a light field sensor, located (i) in the optical path of the optics to acquire light field image data and (ii) at a predetermined distance, relative to the optics, during acquisition of the light field data, wherein the predetermined distance is at a substantially fixed separation relative to the focal plane corresponding to the subject, and wherein the focal plane of the subject is outside of the optical depth of field of the optics with respect to the light field sensor; and

a storage device, coupled to the light field sensor, configured to store the acquired light field image data.

21. The light field imaging device of claim 20, further comprising:

a user interface configured to receive at least one user input, wherein, in response to the user input, the light field sensor acquires the light field image data of the scene.

22. The light field imaging device of claim 20, further comprising control circuitry and wherein:

the autofocus mechanism comprises an autofocus image sensor configured to acquire focus data,

the control circuitry is configured to determine focus metrics using the focus data, and

the optics comprises a second optics configured to provide at least a portion of the light in the optical path onto the autofocus image sensor.

23. A light field imaging device for acquiring light field image data of a subject in a scene, wherein the subject is in a focal plane, the device comprising:

optics, wherein the optics comprises an optical path, an optical depth of field and an autofocus mechanism, and wherein the autofocus is configured to select among a plurality of focuses including a first focus which is related to a focal plane of the subject;

a light field sensor, located (i) in the optical path of the optics to acquire light field image data and (ii) at a predetermined separation, relative to the optics, during acquisition of the light field data, wherein the focal plane of the subject is outside of the optical depth of field of the optics with respect to the light field sensor; and

control circuitry configured to adjust the separation between the light field sensor and the optics among a plurality of discrete, fixed-focus separations, wherein each discrete separation provides a refocusing range of the light field image data; and

a storage device, coupled to the light field sensor, configured to store the acquired light field image data.

24. The light field imaging device of claim 23, wherein the discrete, fixed-focus separations overlap in refocusing range of the light field image data.

25. The light field imaging device of claim 23, wherein the control circuitry is further configured to select a discrete, fixed-focus separation based on which of the discrete, fixed-focus separations provides a range of refocusing which is the furthest towards optical infinity and the in-focus plane of the subject is within the range of refocusing of the selected discrete, fixed-focus separation.

26. The light field imaging device of claim 25, wherein:

the autofocus mechanism comprises an autofocus image sensor configured to acquire focus data;

the control circuitry is configured to determine focus metrics using the focus data, and

the optics comprises a second optics configured to divert at least a portion of the light in the optical path onto the autofocus image sensor.

27. The light field imaging device of claim 23, further comprising:

a user interface configured to receive at least one user input, wherein, in response to the user input, the light field sensor acquires the light field image data of the scene; and

wherein the processing circuitry outputs the data which is representative of the image of the scene, and, after outputting the data which is representative of the image of the scene and in response to the second user input,

determines a final virtual focus depth of the light field image data using data which is representative of the second user input, wherein the second user input is indicative of the final virtual focus depth, and

generates data which is representative of the final image of the scene using the light field image data wherein the final image includes a focus which corresponds to the final virtual focus depth.

28. The light field imaging device of claim 23, further comprising control circuitry and wherein:

the autofocus mechanism comprises an autofocus image sensor to acquire focus data,

the control circuitry determines focus metrics using the focus data, and

the optics comprises a second optics to provide at least a portion of the light in the optical path onto the autofocus image sensor.

29. A light field imaging device for acquiring light field image data of a scene, wherein the light field image device comprises a maximum output image resolution, the device comprising:

optics, wherein the optics comprises an optical path and a focal point;

a light field sensor, located (i) in the optical path of the optics to acquire light field image data and (ii) at a substantially fixed distance relative to the focal point of the optics, wherein the substantially fixed distance;

(a) is substantially independent of the scene and (b) creates an optical depth of field of the optics with respect to the light field sensor that extends to a depth that is closer than optical infinity; and

a storage device, coupled to the light field sensor, configured to store the acquired light field image data;

wherein the light field sensor is configured to acquire light field image data corresponding to an output image of the scene which includes a virtual focus of optical infinity and a resolution that is at least 0.5 times the maximum output image resolution of the light field imaging device.

30. The light field imaging device of claim 29, wherein the maximum output image resolution of the device substantially equals a number of rows of pixels in an image output by the device.

31. The light field imaging device of claim 29, wherein maximum output image resolution of the device substantially equals a maximum resolution of any image output by the device virtually refocused to any subject depth.

32. The light field imaging device of claim 29, wherein the light field sensor is further configured to acquire light field image data corresponding to an output image of the scene which includes a focus of optical infinity and a resolution that is less than 0.8 times the maximum output image resolution of the light field imaging device.