LIGHT FIELD CAMERA IMAGE, FILE AND CONFIGURATION DATA, AND METHODS OF USING, STORING AND COMMUNICATING SAME

• US 20130113981A1
• Filed: 10/31/2012
• Published: 05/09/2013
• Est. Priority Date: 12/01/2006
• Status: Abandoned Application

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Lytro Incorporated
• Inventors: Knight, Timothy J., Pitts, Colvin, Fishman, Alex, Ng, Yi-Ren
• Application Number:
  US13/664,938
• Publication Number:
  US 20130113981A1
• Time in Patent Office: Days
• US Class Current: 348/345
• CPC Class Codes: G02B 27/0075 with means for altering, e.g. increasing, the depth of field or depth of focus
• CPC Class Codes: G02B 3/0056 arranged along two different directions in a plane, e.g. honeycomb arrangement of lenses G02B3/0043 takes precedence; miniaturised objectives for electronic devices employing wafer level optics G02B13/0085
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof

Abstract

A method for acquiring, generating, and/or outputting image data comprises (i) acquiring light field data representative of a scene, (ii) acquiring configuration data representative of how light rays optically propagate through a device, (iii) generating image data using the light field data and the configuration data, wherein the image data includes a focus depth different from that of the light field data, (iv) generating an electronic data file including the image data, the light field data, and the configuration data, and (v) outputting the electronic data file. In one aspect, a light field acquisition device comprises optics, a light field sensor, and processing circuitry to: (i) determine configuration data representative of how light rays optically propagate through the optics, and (ii) generate the electronic data file, wherein the electronic data file includes image data, light field data representative of a light field from the scene, and configuration data.

Claims

1. A method of generating and outputting image data corresponding to a scene, the method comprising:

acquiring light field data which is representative of a light field from the scene, wherein the light field data is acquired using a data acquisition device;

acquiring configuration data which is representative of a characteristic of the data acquisition device;

generating first image data by using at least a portion of the acquired configuration data to interpret at least a portion of the light field data, wherein the first image data comprises a focus or focus depth that is different from a focus or focus depth of the light field data;

generating a first electronic data file comprising (i) the first image data, (ii) the light field data, and (iii) the configuration data; and

outputting the first electronic data file;

wherein the acquired configuration data is representative of at least one of an optical model and a geometric model representing optical propagation of light rays through the acquisition device.

2. The method of claim 1, wherein generating the first electronic data file further comprises arranging the first image data of the first electronic data file in a standard image format.

3. The method of claim 1, wherein generating the first electronic data file further comprises arranging the first image data of the first electronic data file in a JPEG format.

4. The method of claim 2, wherein generating the first electronic data file further comprises at least one of interleaving, threading, watermarking, encoding, multiplexing and meshing the first image data and the light field data.

5. The method of claim 2, wherein generating a first electronic data file further comprises generating a header of the first electronic data file, wherein the header includes the configuration data.

6. The method of claim 2, further comprising:

reading the first electronic data file;

displaying the first image data;

receiving a user input;

generating second image data, in response to the user input, using (i) the light field data of the electronic data file and (ii) the configuration data, wherein the second image data is different from the first image data;

generating a second electronic data file comprising (i) the second image data, (ii) the light field data, and (iii) the configuration data; and

outputting the second electronic data file.

7. The method of claim 6, wherein the second image data comprises a focus or focus depth that is different from the focus or focus depth of the first image data.

8. The method of claim 7, wherein generating the second electronic data file further comprises arranging the second image data of the second electronic data file in a standard image format.

9. The method of claim 6, wherein generating a second electronic data file further comprises at least one of interleaving, threading, watermarking, encoding, multiplexing and meshing the second image data and the light field data.

10. The method of claim 2, further comprising compressing the light field data to generate compressed light field data, and wherein the light field data of the first electronic data file comprises the compressed light field data.

11. The method of claim 2, wherein:

acquiring configuration data comprises acquiring an N-bit key; and

the method further comprises determining optical model data by correlating the N-bit key to predetermined optical model data and wherein generating first image data comprises generating first image data using the light field data and the optical model data.

12. The method of claim 2, further comprising:

reading the first electronic data file;

displaying the first image data;

receiving a user input;

generating second image data, in response to the user input, using (i) the light field data of the electronic data file and (ii) the configuration data, wherein the second image data is different from the first image data;

generating a second electronic data file comprising the second image data; and

outputting the second electronic data file.

13. The method of claim 12, wherein the second image data comprises a focus or focus depth that is different from the focus or focus depth of the first image data.

14. The method of claim 13, wherein generating the second electronic data file further comprises arranging the second image data of the second electronic file in a standard image format.

15. The method of claim 1, wherein the configuration data comprises data which is representative of at least one of an aperture function and an exit pupil associated with the acquisition of the light field data.

16. The method of claim 1, wherein the at least one of an optical model and a geometric model comprises data which is representative of a mapping from a two-dimensional position on a captured 2D array of pixel values of the data acquisition device to a four-dimensional parameterization of the light field from the scene.

17. The method of claim 1, wherein the configuration data of the electronic data file further comprises data representative of a parameter of the data acquisition device.

18. The method of claim 1, wherein the configuration data of the electronic data file further comprises data representative of a configuration of the data acquisition device.

19. The method of claim 1, wherein the acquired light field data comprises directional information for light rays.

20. The method of claim 1, further comprising:

acquiring metadata describing the light field data;

wherein generating the first image data comprises generating the first image data using the light field data, the configuration data, and the metadata describing the light field data.

21. A system comprising:

read circuitry configured to read a first electronic data file which is stored in a memory, wherein the first electronic data file comprises (i) first image data, (ii) light field data which is representative of a light field from a scene, and (iii) configuration data which is representative of a characteristic of a light field data acquisition device;

a display configured to visually output an image of the scene using the first image data;

a user interface configured to receive a user input;

processing circuitry, coupled to the read circuitry, display and user interface, configured to;

generate second image data, in response to the user input, by using at least a portion of the configuration data to interpret at least a portion of the light field data, wherein the second image data comprises a focus or focus depth that is different from a focus or focus depth of the first image data, and

generate a second electronic data file comprising the second image data; and

write circuitry, coupled to the processing circuitry, to write the second electronic data file to the memory;

wherein the configuration data is representative of at least one of an optical model and a geometric model representing optical propagation of light rays through the acquisition device.

22. The system of claim 21, wherein the second electronic data file further comprises (i) the light field data which is representative of a light field from the scene, and (ii) the configuration data.

23. The system of claim 22, wherein the configuration data comprises data which is representative of an aperture function or an exit pupil which is associated with the light field data acquisition device that acquired the light field data.

24. The system of claim 22, wherein the processing circuitry is configured to generate the second electronic data file by performing at least one of interleaving, threading, watermarking, encoding, multiplexing and meshing the second image data and the light field data.

25. The system of claim 22, wherein the second electronic data file comprises a header, wherein the header comprises the configuration data.

26. The system of claim 22, wherein the processing circuitry is configured to generate the first electronic data file by compressing the light field data to generate compressed light field data, and wherein the light field data of the second electronic data file comprises the compressed light field data.

27. The system of claim 21, wherein the processing circuitry is configured to arrange the second image data of the second electronic data file in a standard image format.

28. The system of claim 21, wherein the processing circuitry is configured to arrange the second image data of the second electronic data file in a JPEG format.

29. The system of claim 21, wherein the configuration data of the first electronic data file comprises an N-bit key, and wherein the processing circuitry determines optical model data by correlating the N-bit key to a plurality of different, predetermined optical model data.

30. The system of claim 21, wherein the configuration data of the first electronic data file further comprises data representative of a parameter of the data acquisition device.

31. The system of claim 21, wherein the configuration data of the first electronic data file further comprises data representative of a configuration of the data acquisition device.

32. The system of claim 21, wherein the light field data comprises directional information for light rays.

33. The system of claim 21, wherein:

the processing circuitry is further configured to acquire metadata describing the light field data; and

the processing circuitry is configured to generate the second image data using the light field data, the configuration data, and the metadata describing the light field data.

34. A light field acquisition device for acquiring light field image data of a scene, the device comprising:

optics, wherein the optics includes an optical path;

a light field sensor, located in the optical path of the optics, configured to acquire light field image data;

a user interface configured to receive a user input, wherein, in response to the user input, the light field sensor acquires the light field image data of the scene; and

processing circuitry, coupled the light field sensor and the user interface, configured to generate and output an electronic data file, the processing circuitry configured to;

determine configuration data which is representative of a characteristic of the optics and light field sensor; and

generate and output the electronic data file, wherein the electronic data file comprises (i) image data, (ii) light field data which is representative of a light field from the scene, and (iii) configuration data; and

memory, coupled to the processing circuitry, configured to store the electronic data file therein;

wherein;

the configuration data is representative of at least one of an optical model and a geometric model representing optical propagation of light rays through the acquisition device; and

generating the electronic data file comprises using at least a portion of the configuration data to interpret at least a portion of the light field data.

35. The device of claim 34, wherein the configuration data comprises data which is representative of an aperture function or exit pupil of the light field acquisition device.

36. The device of claim 34, wherein the configuration data comprises data which is representative of a mapping from a two-dimensional position on a captured 2D array of pixel values to a four-dimensional parameterization of a light field from the scene.

37. The device of claim 34, wherein the processing circuitry is configured to generate the electronic data file by performing at least one of interleaving, threading, watermarking, encoding, multiplexing and meshing the image data and the light field data.

38. The device of claim 34, wherein the processing circuitry is configured to generate a header which comprises the configuration data, wherein the electronic data file includes the header.

39. The device of claim 34, wherein the processing circuitry is configured to generate the electronic data file by compressing the light field data to generate compressed light field data, and wherein the light field data of the electronic data file comprises the compressed light field data.

40. The device of claim 34, wherein the processing circuitry is configured to arrange the image data of the electronic data file in a standard image format.

41. The device of claim 34, wherein the processing circuitry is configured to arrange the image data of the electronic data file in a JPEG format.

42. The device of claim 34, wherein the configuration data of the electronic data file comprises an N-bit key which is representative of predetermined optical model data.