LIGHT FIELD PROCESSING METHOD

• US 20150146032A1
• Filed: 11/22/2013
• Published: 05/28/2015
• Est. Priority Date: 11/22/2013
• Status: Abandoned Application

• Current Assignee: Ecole polytechnique fãdãrale de lausanne epfl (Schweizerische Eidgenossenschaft)
• Original Assignee: Vidinoti SA (BigTinCan Holdings Limited)
• Inventors: RIME, Laurent, ARULRAJ, Bernard M., NISHIDA, Keishi
• Application Number:
  US14/087,616
• Publication Number:
  US 20150146032A1
• Time in Patent Office: Days
• US Class Current: 348/222.1
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/21 involving computational photography
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/232 using fly-eye lenses, e.g. arrangements of circular lenses
• CPC Class Codes: H04N 13/30 Image reproducers optical systems for producing stereoscopic or other three-dimensional effects G02B30/00
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/957 Light-field or plenoptic cameras or camera modules
• CPC Class Codes: H04N 7/01 Conversion of standards , e.g. involving analogue television standards or digital television standards processed at pixel level

Abstract

A light field processing method for processing data corresponding to a light field, comprising:

• • capturing with a plenoptic camera initial data representing a light field in a format dependent from said plenoptic camera;
  • converting said initial data into converted data representing said light field in a camera independent format;
  • processing said converted data so as to generate processed data representing a different light field.

Claims

1. A light field processing method for processing data corresponding to a light field, comprising:

capturing with a plenoptic camera initial data representing a light field in a format dependent from said plenoptic camera;

converting said initial data into converted data representing said light field in a camera independent format;

processing said converted data so as to generate processed data representing a different light field.

2. The method of claim 1, wherein the direction of each light ray in said light field is parametrized in said camera independent format by exactly four parameters.

3. The method of claim 2, wherein the direction of each light ray in said light field is parametrized in said camera independent format with the positions where said light ray intersects two pre-defined planes.

4. The method of claim 2, wherein the direction of each light ray in said light field is parametrized in said camera independent format with the positions where it crosses two spheres tangent to each other, one of said positions being the tangent point.

5. The method of claim 2, wherein the direction of each light ray in said light field is parametrized in said camera independent format with the two positions where it crosses one single sphere.

6. The method of claim 2, wherein the direction of each light ray in said light field is parametrized in said camera independent format with the intersection of one plane with one sphere.

7. The method of claim 2, wherein each light ray in said light field is parametrized in said camera independent format with the distance of the closest point on the ray from the center of a sphere, the polar coordinates of this closest point with respect to the spherical coordinate, and the angle between the ray and a plane tangent to the sphere at said closest point.

8. The method of claim 1, comprising a step of converting each light ray of said light field from one said camera-independent representation to a different said camera-independent representation.

9. The method of claim 1, wherein said step of converting said initial data uses a conversion function depending on the model of the camera.

10. The method of claim 9, comprising a step of determining said conversion function using the plenoptic representation of a known scene.

11. The method of claim 1, comprising a step of processing a representation of a light field in said camera independent format to produce a scene-centered representation of the same light field.

12. The method of claim 11, comprising a step of modifying light rays of said scene-centered representation so as to modify the visual appearance of the object at the center of the representation.

13. The method of claim 11, further comprising determining the depth of a point by identifying a plurality of rays emitted by this point, and computing the distance to this point by triangulation.

14. The method of claim 11, further comprising determining the depth of a point by extracting epipolar lines.

15. The method of claim 1, said step of processing comprising applying a filter to the converted data representing said light field in a camera independent format.

16. The method of claim 1, said step of processing comprising recognising at least one image feature in said converted data, and adding to said converted data representing at least one augmented reality element depending on said feature, said data representing at least one augmented reality element being in said camera independent format.

17. The method of claim 1, said step of processing comprising selecting an object of the scene based on his position in space as determined from said converted data.

18. The method of claim 17, said step of processing further comprising replacing the rays emitted from the selected object.

19. The method of claim 11, said step of processing comprising performing an object segmentation by starting from said scene-centered representation, and identifying which rays actually belong to the same object.

20. The method of claim 1, said step of processing comprising resizing an object.

21. The method of claim 1, said step of processing comprising refocusing by moving the Rx-Ry plane.

22. The method of claim 1, said step of processing comprising:

retrieving second data corresponding to a different light field into a second camera independent format different from the format used for representing said converted data,

converting said second data into said camera independent data;

fusing said data and said second data,

23. The method of claim 1, further comprising a step of visualizing as single or multiple 2D or 3D images a light field stored in said camera independent format

24. The method of claim 23, said camera independent format comprising a representation of each ray by the position where each ray crosses two predefined planes, said step of rendering comprising a step of generating a 2D image in which each pixel corresponds to the brightness of the ray that crosses one of said plane at a given position.

25. The method of one of the claims 23 to 24, said step of rendering comprising applying a perspective shift in order to generate a view of said light field from an arbitrary position.

27. A computer carrier storing data processed with the method of claim 1.

28. A computer carrier storing program data causing a processor to carry out the method of claim 1 when program data is executed.

30. The method of claim 28, comprising processing said converted data by merging with another light field representing data in said second format.

26. A light field processing apparatus for processing data corresponding to a light field, comprising:

a plenoptic camera arranged for capturing initial data representing a light field in a format dependent from said plenoptic camera;

a data converter for converting said initial data into converted data representing said light field in a camera independent format;

a light field processor for processing said converted data so as to generate processed data representing a different light field.

29. A method for processing data corresponding to a light field, comprising:

converting data representing said light field in first, camera dependent format, into converted data representing said light field in a second format in which the direction of each light ray in said light field is parametrized with the positions where said light ray intersects two planes or two spheres or one sphere and one plane;

processing said converted data so as to generate processed data representing a different light field.

31. A computer carrier storing data representing a light field in a format in which the direction of each light ray in said light field is parametrized with the positions where said light ray intersects two planes or two spheres or one sphere and one plane.