Statistical representation and coding of light field data

• US 20040114807A1
• Filed: 12/13/2002
• Published: 06/17/2004
• Est. Priority Date: 12/13/2002
• Status: Abandoned Application

• Current Assignee: NTT Docomo Incorporated (Nippon Telegraph and Telephone Corporation)
• Original Assignee: NTT Docomo Incorporated (Nippon Telegraph and Telephone Corporation)
• Inventors: Lelescu, Dan, Bossen, Frank Jan
• Application Number:
  US10/318,837
• Publication Number:
  US 20040114807A1
• Time in Patent Office: Days
• US Class Current: 382/229
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G06T 9/00 Image coding bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00

Abstract

A method of representing light field data by capturing a set of images of at least one object in a passive manner at a virtual surface where a center of projection of an acquisition device that captures the set of images lies and generating a representation of the captured set of images using a statistical analysis transformation based on a parameterization that involves the virtual surface.

Claims

1. A method of representing light field data, the method comprising:

capturing a set of images of at least one object in a passive manner at a virtual surface where a center of projection of an acquisition device that captures said set of images lies; and

generating a representation of said captured set of images using a statistical analysis transformation based on a parameterization that involves said virtual surface.

2. The method of claim 1, wherein said statistical analysis transformation is a principal component analysis.

3. The method of claim 1, wherein said statistical analysis transformation is an independent component analysis.

4. The method of claim 1, wherein said virtual surface is a plane.

5. The method of claim 1, wherein said parameterization involves a second virtual surface spaced from said virtual surface.

6. The method of claim 4, wherein said parameterization involves a second virtual surface that is parallel to said virtual surface.

7. The method of claim 1, wherein said representation is generated by a single global principal component analysis applied to said set of images captured at said virtual surface.

8. The method of claim 1, further comprising:

ordering pixels of each image of said sets of images; and

creating a corresponding set of vectors that are used to generate said representation.

9. The method of claim 1, further comprising determining dimensionality of a PCA representation subspace associated with said representation.

10. The method of claim 9, wherein said dimensionality is pre-determined.

11. The method of claim 9, wherein said determining is based on visual characteristics of said set of images.

12. The method of claim 1, wherein said statistical analysis transformation is a direct principal component analysis.

13. The method of claim 1, wherein said statistical analysis transformation is a training sample principal component analysis.

14. The method of claim 1, wherein said statistical analysis transformation is a training sample independent component analysis.

15. The method of claim 13, wherein said determining comprises selecting a uniformly distributed sample of said set of images to be used by said training sample principal component analysis.

16. The method of claim 13, wherein said determining comprises selecting a nonuniformly distributed sample of said set of images to be used by said training sample principal component analysis.

17. The method of claim 13, wherein said determining comprises:

initially selecting J vectors that are used for said training sample principal component analysis;

determining a PCA representation based on said training sample principal component analysis;

generating at most J eigenvectors;

retaining M eigenvectors of said J eigenvectors, wherein M J; and

applying said M eigenvectors to generate said representation.

18. The method of claim 1, wherein said statistical analysis transformation is an iterative principal component analysis.

19. The method of claim 18, wherein said determining comprises selecting a uniformly distributed sample of said set of images to be used by said iterative principal component analysis.

20. The method of claim 18, wherein said determining comprises selecting a nonuniformly distributed sample of said set of images to be used by said iterative principal component analysis.

21. The method of claim 18, wherein said determining comprises:

a) determining an initial PCA representation based on an initial sample set of eigenvectors of said set of images;

b) generating an initial set of M eigenvectors;

c) performing an iteration with all of said M eigenvectors and an original vector from said set of images excluding said sample set and generating a new set of eigenvectors;

d) repeat step c) until all original vectors have been used during said iteration step c) so as to generate a final set of M eigenvectors; and

e) applying said final set of M eigenvectors to generate said representation.

22. The method of claim 1, wherein said representation is generated by a set of local PCA representation subspaces that correspond to a set of local areas of said virtual surface.

23. The method of claim 22, further comprising determining dimensionality of each one of said local PCA representation subspaces.

24. The method of claim 23, wherein said determining is made subject to a constraint imposed on a total dimensionality of said virtual surface.

25. The method of claim 22, wherein said set of local PCA representation subspaces are direct PCA representation subspaces.

26. The method of claim 22, wherein said set of local PCA representation subspaces are training sample PCA representation subspaces.

27. The method of claim 22, wherein set of local PCA representation subspaces are iterative PCA representation subspaces.

28. The method of claim 22, wherein said local areas each have the same area.

29. The method of claim 22, wherein said local areas are selected based on geometry of an imaging device at said virtual plane.

30. The method of claim 22, wherein said local areas are selected based on a linear discriminating analysis applied to images associated with said virtual surface.

31. The method of claim 22, wherein said set of local PCA representation subspaces have variable dimensionality.

32. The method of claim 31, wherein said variable dimensionality is selected based on rate-distortion measures.

33. The method of claim 1, wherein said representation is generated by a set of local ICA representation subspaces that correspond to a set of local areas of said virtual surface.

34. The method of claim 1, further comprising coding eigenvector data associated with images in said virtual surface.

35. The method of claim 34, wherein said coding comprises using inverse lexicographic ordering of said eigenvector data to generate corresponding eigenimages.

36. The method of claim 35, further comprising adjusting coding of said eigenimages based on rankings of said eigenimages.

37. The method of claim 36, wherein said adjusting comprises using a predetermined adjustment.

38. The method of claim 36, wherein said adjusting comprises using an eigenvalue magnitude-driven analytic function.

39. The method of claim 1, further comprising coding PCA or ICA transformed image vectors associated with each image of said set of images in said virtual surface.

40. The method of claim 39, further comprising gathering said transformed image vectors as columns of a matrix S.

41. The method of claim 40, further comprising mapping each row of said matrix into a two dimensional matrix through inverse lexicographic ordering.

42. The method of claim 41, further comprising:

a) coding said two dimensional matrix corresponding to a first row of said matrix S, and denoted by A; and

b) coding a matrix B formed by concatenating said two dimensional matrices corresponding to all rows of matrix S except said first row of matrix S.

43. The method of claim 34, further comprising controlling scalability by coding a limited number of said eigenvectors and correspondingly truncated transformed image vectors corresponding to said set of images.

44. The method of claim 3, further comprising coding ICA basis vector data.

45. The method of claim 3, further comprising coding ICA transformed image vectors associated with each image of said set of images in said virtual surface.

46. The method of claim 34, further comprising transmitting coded eigenvector data based on said coding.

47. The method of claim 42, further comprising transmitting coded transformed vector data based on said coding.

48. The method of claim 34, further comprising decoding eigenvector data based on said coding.

49. The method of claim 42, further comprising decoding transformed vector data based on said coding.

50. The method of claim 48, further comprising reconstructing an image from decoded transformed vector data and said decoded eigenvector data using an inverse PCA transformation

51. The method of claim 50, further comprising randomly accessing and reconstructing any image associated with said virtual surface.

52. The method of claim 50, wherein said reconstructing involves using a subset of said decoded eigenvector data for scalability.

53. The method of claim 44, further comprising transmitting coded basis vector data based on said coding.

54. The method of claim 42, further comprising transmitting coded transformed image vector data based on said coding.

55. The method of claim 44, further comprising decoding basis vector data based on said coding.

56. The method of claim 42, further comprising decoding transformed image vector data based on said coding.

57. The method of claim 55, further comprising reconstructing an image from decoded transformed vector data and said decoded basis vector data using an inverse ICA transformation.

58. The method of claim 56, further comprising reconstructing an image from said decoded transformed vector data using an inverse ICA transformation.

59. The method of claim 58, further comprising randomly accessing and reconstructing any image associated with said virtual surface.