Light field data acquisition devices, and methods of using and manufacturing same

• US 8,289,440 B2
• Filed: 12/08/2009
• Issued: 10/16/2012
• Est. Priority Date: 12/08/2008
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Lytro Incorporated
• Inventors: Knight, Timothy J., Ng, Yi-Ren, Pitts, Colvin
• Primary Examiner(s): Misleh, Justin P
• Application Number:
  US12/632,979
• Publication Number:
  US 20100141802A1
• Time in Patent Office: 1,043 Days
• Field of Search: None
• US Class Current: 348/349
• CPC Class Codes: G02B 27/0075 with means for altering, e.g. increasing, the depth of field or depth of focus
• CPC Class Codes: G02B 7/282 Autofocusing of zoom lenses
• CPC Class Codes: G06T 2200/21 involving computational photography
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: H04N 23/00 Cameras or camera modules comprising electronic image sensors; Control thereof

Abstract

A light field data acquisition device includes optics and a sensor to acquire light field image data of a scene. In at least one embodiment, the sensor is located at a substantially fixed, predetermined location relative to the focal point of the optics. The device determines a first virtual focus depth different from an optical focus depth of the light field image data, and automatically generates and outputs data representative of a first image of the scene at the first virtual focus depth. In response to user input, the device determines a second virtual focus depth and generates data representative of a second image of the scene at the second virtual focus depth.

Claims

1. A light field imaging device for acquiring light field image data of a scene, the device comprising:

optics, wherein the optics includes an optical path and a focal point, wherein the focal point is associated with a focal length of the optics;

a light field sensor, located (i) in the optical path of the optics to acquire light field image data and (ii) at a substantially fixed, predetermined location relative to the focal point of the optics, wherein the predetermined location is substantially independent of the scene, and wherein an optical depth of field of the optics with respect to the light field sensor extends to a depth that is closer than optical infinity;

a user interface to receive a plurality of user inputs, including first and second user inputs, wherein, in response to the first user input, the light field sensor acquires the light field image data of the scene; and

processing circuitry, coupled the user interface, to generate and output image data which is representative of an image, the processing circuitry to;

determine a first virtual focus depth of the light field image data, wherein the first virtual focus depth is different from the optical focus depth of the light field image data,

automatically generate data which is representative of a first image of the scene using the light field image data, wherein the first image includes a focus which corresponds to the first virtual focus depth,

output the data which is representative of the first image, and, after outputting the data which is representative of the first image and in response to the second user input,

determine a second virtual focus depth of the light field image data using data which is representative of the second user input, wherein the second user input is indicative of the second virtual focus depth and the second virtual focus depth is different from the optical focus depth of the light field image data, and

generate data which is representative of a second image of the scene using the light field image data wherein the second image includes a focus which corresponds to the second virtual focus depth.

2. The light field imaging device of claim 1 wherein the optics is configurable to include a plurality of different focal lengths having associated focal points.

3. The light field imaging device of claim 2 wherein the substantially fixed, predetermined location of the light field sensor, relative to the focal point of the optics, changes in accordance with the focal length of the optics.

4. The light field imaging device of claim 1 wherein the optics includes a zoom lens system having a plurality of zoom positions.

5. The light field imaging device of claim 4 further including a mechanical system, coupled to the light field sensor, to maintain the light field sensor unit at the same fixed, predetermined location relative to the focal point of the optics for the plurality of the zoom positions.

6. The light field imaging device of claim 4 wherein the zoom lens system includes a continuous zoom providing the plurality of zoom positions.

7. The light field imaging device of claim 4 wherein the light field sensor maintains the predetermined location relative to the focal point of the optics for the plurality of the zoom positions of the zoom lens system.

8. The light field imaging device of claim 4 further including a mechanical system, coupled to the light field sensor and/or the optics, to maintain the light field sensor unit at the predetermined location relative to the focal point of the optics for the plurality of the zoom positions of the zoom lens system.

9. The light field imaging device of claim 8 further including:

a spatial adjustment unit, coupled to the light field sensor and/or the optics, to responsively move the light field sensor unit; and

control circuitry, coupled to the spatial adjustment unit, to control the spatial adjustment unit to maintain the light field sensor unit at the predetermined location relative to the focal point of the optics.

10. The light field imaging device of claim 9 further including memory, coupled to the control circuitry, to store data which is representative of a plurality of locations of the light field sensor corresponding to a plurality of associated zoom positions.

11. The light field imaging device of claim 9 further including memory, coupled to the control circuitry, wherein the memory includes a database or lookup table which correlates a plurality of locations of the light field sensor according to a plurality of associated zoom positions.

12. The light field imaging device of claim 11 wherein the control circuitry accesses the database or lookup table having data which is representative of the location based on the zoom position of the zoom lens system and generates control signal which are applied to the spatial adjustment unit to responsively move the light field sensor unit.

13. The light field imaging device of claim 8 wherein, in response, to outputting the first image, the user interface receives the second user input which is indicative of the second virtual focus depth.

14. The light field imaging device of claim 1 further including a mechanical system, coupled to the light field sensor and/or the optics, to maintain the light field sensor unit at the predetermined location relative to the focal point of the optics for a plurality of focuses of the optics.

15. The light field imaging device of claim 14 wherein the mechanical system is active.

16. The light field imaging device of claim 1 wherein the optics is configurable to include a plurality of zoom positions which provide a plurality of different focal lengths having associated focal points and the device further includes a mechanical system, coupled to the light field sensor, to maintain the light field sensor unit at the same fixed, predetermined location relative to the focal point of the optics for the plurality of focuses.

17. The light field imaging device of claim 16 wherein the mechanical system is active or passive.

18. The light field imaging device of claim 1 wherein the user interface includes a display to (i) receive the data which is representative of the first image and (ii) output the first image.

19. The light field imaging device of claim 1 further including memory, coupled to the processing circuitry, to store the data which is representative of the second image.

20. The light field imaging device of claim 1 wherein the processing circuitry computes output images having an array of output pixels, and wherein the light field sensor includes (i) a microlens array having a plurality of microlenses and (ii) a sensor having an array of sensor pixels, and wherein the predetermined distance is between (0.7*N_usable*W_MLA*F#_optics*m)/(W_output) and (3.0*N_usable*W_MLA*F#_optics*m)/(W_output),

where:

N_usable is equal to the number of pixels containing usable directional information across a microlens disk image that is formed, through each microlens, on the sensor, or are located under a microlens,

W_MLA is representative of a number of microlenses across a width of the microlens array,

W_output is representative of a number of pixels across a width of the array of output pixels,

m is equal to a distance between the centers of two neighboring microlenses in the microlens array, and

F#_optics is equal to an F-number of the optics.

21. The light field imaging device of claim 1 wherein the light field sensor includes (i) a microlens array having a plurality of microlenses and (ii) a sensor having an array of pixels, and wherein the predetermined distance is greater than (m*F#_optics),

where:

m is equal to the distance between the centers of two neighboring microlenses in the microlens array, and

F#_optics is equal to an F-number of the optics.

22. The light field imaging device of claim 1 wherein the light field sensor includes (i) a microlens array having a plurality of microlenses and (ii) a sensor having an array of pixels, and wherein the predetermined distance is less than (N_usable*m*F#_optics),

where:

N_usable is equal to the number of pixels containing usable directional information across a microlens disk image that is formed, through each microlens, on the sensor, or are located under a microlens,

m is equal to the distance between the centers of two neighboring microlenses in the microlens array, and

F#_optics is equal to an F-number of the optics.

23. A light field imaging device for acquiring light field image data of a subject in a scene, wherein the subject is in a focal plane, the device comprising:

optics, wherein the optics includes an optical path, an optical depth of field and an autofocus mechanism, wherein the autofocus selects between a plurality of focuses including a first focus which is related to a focal plane of the subject;

a light field sensor, located (i) in the optical path of the optics to acquire light field image data and (ii) at a predetermined location, relative to the optics, during acquisition of the light field data, wherein the predetermined location is at a substantially fixed separation relative to the focal plane corresponding to the subject, and wherein the focal plane of the subject is outside of the optical depth of field of the optics with respect to the light field sensor; and

processing circuitry to generate an output image data which is representative of an image including the subject in the scene, the processing circuitry to;

determine a virtual focus depth of the light field image data, wherein the virtual focus depth is different from the optical focus depth of the light field image data, and

automatically generate data which is representative of the image of the scene using the light field image data and data which is representative of the virtual focus depth, wherein the image includes a focus which corresponds to the virtual focus depth.

24. The light field imaging device of claim 23 further including memory, coupled to the processing circuitry, to store the data which is representative of the image of the scene.

25. The light field imaging device of claim 23 further including:

a user interface to receive a plurality of user inputs, including first and second user inputs, wherein, in response to the first user input, the light field sensor acquires the light field image data of the scene; and

wherein the processing circuitry outputs the data which is representative of the image of the scene, and, after outputting the data which is representative of the image of the scene and in response to the second user input,

determines a final virtual focus depth of the light field image data using data which is representative of the second user input, wherein the second user input is indicative of the final virtual focus depth, and

generates data which is representative of the final image of the scene using the light field image data wherein the final image includes a focus which corresponds to the final virtual focus depth.

26. The light field imaging device of claim 23 further including control circuitry and wherein:

the autofocus mechanism includes an autofocus image sensor to acquire focus data,

the control circuitry determines focus metrics using the focus data, and

the optics includes a second optics to provide at least a portion of the light in the optical path onto the autofocus image sensor.

27. A light field imaging device for acquiring light field image data of a subject in a scene, wherein the subject is in a focal plane, the device comprising:

optics, wherein the optics includes an optical path, an optical depth of field and an autofocus mechanism, wherein the autofocus selects between a plurality of focuses including a first focus which is related to a focal plane of the subject;

a light field sensor, located (i) in the optical path of the optics to acquire light field image data and (ii) at a predetermined separation, relative to the optics, during acquisition of the light field data, wherein the focal plane of the subject is outside of the optical depth of field of the optics with respect to the light field sensor;

control circuitry to adjust the separation between the light field sensor and the optics between a plurality of discrete, fixed-focus separations, wherein each discrete separation provides a refocusing range of the light field image data; and

processing circuitry to generate an output image data which is representative of an image including the subject in the scene, the processing circuitry to;

determine a virtual focus depth of the light field image data, wherein the virtual focus depth is different from the optical focus depth of the light field image data, and

automatically generate data which is representative of the image of the scene using the light field image data and data which is representative of the virtual focus depth, wherein the image includes a focus which corresponds to the virtual focus depth.

28. The light field imaging device of claim 27 wherein the discrete, fixed-focus separations overlap in refocusing range of the light field image data.

29. The light field imaging device of claim 27 wherein the control circuitry selects a discrete, fixed-focus separation based on which of the discrete, fixed-focus separations provides a range of refocusing which is the furthest towards optical infinity and the in-focus plane of the subject is within the range of refocusing of the selected discrete, fixed-focus separation.

30. The light field imaging device of claim 29, wherein:

the autofocus mechanism includes an autofocus image sensor to acquire focus data, wherein the control circuitry determines focus metrics using the focus data, and

the optics includes a second optics to divert at least a portion of the light in the optical path onto the autofocus image sensor.

31. The light field imaging device of claim 27 further including:

a user interface to receive a plurality of user inputs, including first and second user inputs, wherein, in response to the first user input, the light field sensor acquires the light field image data of the scene; and

wherein the processing circuitry outputs the data which is representative of the image of the scene, and, after outputting the data which is representative of the image of the scene and in response to the second user input,

determines a final virtual focus depth of the light field image data using data which is representative of the second user input, wherein the second user input is indicative of the final virtual focus depth, and

generates data which is representative of the final image of the scene using the light field image data wherein the final image includes a focus which corresponds to the final virtual focus depth.

32. The light field imaging device of claim 27 further including control circuitry and wherein:

the autofocus mechanism includes an autofocus image sensor to acquire focus data,

the control circuitry determines focus metrics using the focus data, and

the optics includes a second optics to provide at least a portion of the light in the optical path onto the autofocus image sensor.

33. A light field imaging device for acquiring light field image data of a scene, wherein the light field image device includes a maximum output image resolution, the device comprising:

optics, wherein the optics includes an optical path and a focal point;

a light field sensor, located (i) in the optical path of the optics to acquire light field image data and (ii) at a substantially fixed location relative to the focal point of the optics, wherein the substantially fixed location;

(a) is substantially independent of the scene and (b) creates an optical depth of field of the optics with respect to the light field sensor that extends to a depth that is closer than optical infinity; and

processing circuitry to generate and output image data which is representative of an output image of the scene, the processing circuitry to;

determine a virtual focus depth of the light field image data, wherein the virtual focus depth is different from the optical focus depth of the light field image data, and

automatically generate data which is representative of the output image of the scene using the light field image data and data which is representative of the virtual focus depth, wherein the output image includes a focus which corresponds to the virtual focus depth; and

wherein the light field sensor, which is located at the substantially fixed location relative to the focal point of the optics, acquires light field image data which corresponds to an output image of the scene which includes a virtual focus of optical infinity and a resolution in lines per picture height (LPH) that is at least 0.5 times the maximum output image resolution of the light field imaging device.

34. The light field imaging device of claim 33 wherein the maximum output image resolution of the device is a number of rows of pixels in an image output by the device.

35. The light field imaging device of claim 33 wherein maximum output image resolution of the device is the maximum resolution in LPH of any image output by the device virtually refocused to any subject depth.

36. The light field imaging device of claim 33 wherein the light field sensor, which is located at the substantially fixed location relative to the focal point of the optics, acquires light field image data which corresponds to an output image of the scene which includes a focus of optical infinity and a resolution in lines per picture height (LPH) that is less than 0.8 times the maximum output image resolution of the light field imaging device.