Processing of light fields by transforming to scale and depth space

• US 9,460,515 B2
• Filed: 10/25/2013
• Issued: 10/04/2016
• Est. Priority Date: 10/25/2013
• Status: Expired due to Fees

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Toi, Ivana, Berkner, Kathrin
• Primary Examiner(s): Desire, Gregory M
• Application Number:
  US14/064,090
• Publication Number:
  US 20150117756A1
• Time in Patent Office: 1,075 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 17/14 Fourier, Walsh or analogous domain transformations , e.g. Laplace, Hilbert, Karhunen-Loeve, transforms for correlation function computation G06F17/156; spectrum analysers G01R23/16
• CPC Class Codes: G06T 2200/21 involving computational photography
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 7/557 from light fields, e.g. from plenoptic cameras
• CPC Class Codes: H04N 13/232 using fly-eye lenses, e.g. arrangements of circular lenses
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

Light field images of a three-dimensional scene are transformed from an (image,view) domain to an (image,scale,depth) domain. Processing then occurs in the (image,scale,depth) domain.

Claims

1. A method for processing light field images of a three-dimensional scene, the method implemented on a computer system and comprising:

accessing an (image,view) domain representation of the light field images of the three-dimensional scene, wherein the (image,view) domain representation of the light field images is a representation of the light field images as a function of (image) and (view) dimensions;

applying a scale-depth transform to transform the (image,view) domain representation to an (image, scale, depth) domain representation, wherein the (image,scale,depth) domain representation is a representation of the light field images as a function of (image), (scale) and (depth) dimensions; and

processing the (image,scale,depth) domain representation of the three-dimensional scene, wherein processing the (image,scale,depth) domain representation comprises at least one of (a) estimating depth in the three-dimensional scene based on processing the (image,scale,depth) domain representation or (b) extracting three-dimensional features in the three-dimensional scene based on processing the (image, scale, depth) domain representation.

2. The method of claim 1 wherein the (scale) portion of the scale-depth transform is based on a Gaussian kernel or one of its derivatives.

3. The method of claim 1 wherein the (depth) portion of the scale-depth transform is based on points at different depths in the three-dimensional scene creating different curves in the (image,view) domain.

4. The method of claim 3 wherein the (depth) portion of the scale-depth transform is based on points at different depths in the three-dimensional scene creating rays at different angles in the (image,view) domain.

5. The method of claim 4 further comprising:

selecting rays in the (image,view) domain based on whether the rays have sharp edges on both sides of the ray;

wherein processing the (image,scale,depth) domain representation of the three-dimensional scene is based on processing the selected rays.

6. The method of claim 1 wherein the scale-depth transform is based on a Ray-Gaussian kernel

7. The method of claim 6 wherein applying the scale-depth transform comprises:

convolving the (image,view) domain representation with the Ray-Gaussian kernel or its derivative for σ

ε

{σ

₁, . . . , σ

_n} and for φ

ε

{(φ

₁, . . . , φ

_m}; and

repeating (k−

1) times the step of downsampling the (image,view) domain representation by p and convolving with the Ray-Gaussian kernel or its derivative for σ

ε

{σ

₁, . . . , σ

_n} and for φ

ε

{φ

₁, . . . , φ

_m};

where n is the number of samples per downsampling range of scale, m is the number of samples in the depth domain, and p is the downsampling factor.

8. The method of claim 6 wherein applying the scale-depth transform comprises:

convolving the (image,view) domain representation with the Ray-Gaussian kernel or its derivative for σ

ε

{σ

₁, . . . , σ

_n} and for φ

ε

{(φ

₁, . . . , φ

_m};

repeating (k−

1) times the step of downsampling an image portion of the (image,view) domain representation by p and convolving with the Ray-Gaussian kernel or its derivative for σ

ε

{σ

₁, . . . , σ

_n} and for {φ

′

₁, . . . , φ

′

_m};

where n is the number of samples per downsampling range of scale, m is the number of samples in the depth domain, and p is the downsampling factor.

9. The method of claim 6 wherein applying the scale-depth transform comprises:

convolving the (image,view) domain representation with the Ray-Gaussian kernel or its derivative for σ

ε

{σ

₁, . . . , σ

_n, pσ

₁, . . . , pσ

_n, . . . , p^kσ

₁, . . . , p^kσ

_n} and for φ

ε

{φ

₁, . . . , φ

_m};

where p is a scaling ratio, n is the number of samples per range of scale, m is the number of samples in the depth domain, and (k+1) is the number of scales.

10. The method of claim 6 wherein p=2.

11. The method of claim 1 wherein processing the (image,scale,depth) domain representation of the three-dimensional scene comprises estimating depth in the three-dimensional scene based on processing the (image,scale,depth) domain representation.

12. The method of claim 11 wherein the scale-depth transform is a normalized second derivative Ray-Gaussian transform, and estimating depth in the three-dimensional scene comprises finding extrema of the normalized second derivative Ray-Gaussian transform and estimating depth based on said extrema.

13. The method of claim 12 wherein estimating depth in the three-dimensional scene further comprises detecting occlusions based on said extrema.

14. The method of claim 12 further comprising:

additionally applying a normalized first derivative Ray-Gaussian transform to the (image,view) domain representation;

finding extrema of the normalized first derivative Ray-Gaussian transform;

estimating edges in the three-dimensional scene based on the extrema of the normalized first derivative Ray-Gaussian transform; and

improving the depth estimates based on the estimated edges.

15. The method of claim 1 wherein processing the (image,scale,depth) domain representation of the three-dimensional scene comprises extracting three-dimensional features in the three-dimensional scene based on processing the (image,scale,depth) domain representation.

16. The method of claim 15 wherein the scale-depth transform is a normalized first derivative Ray-Gaussian transform;

and extracting three-dimensional features in the three-dimensional scene comprises finding extrema of the normalized first derivative Ray-Gaussian transform and extracting three-dimensional features based on said extrema.

17. The method of claim 1 wherein the light field images of the three-dimensional scene are taken from regularly spaced viewpoints.

18. The method of claim 1 wherein the light field images of the three-dimensional scene are captured by a plenoptic imaging system.

19. The method of claim 1 wherein the (scale) and (depth) domains are each one-dimensional.

20. The method of claim 1 wherein the light field images have a deterministic relation between the (scale) and (depth) domains, and the step of processing the (image,scale,depth) domain representation makes use of the deterministic relation.

21. The method of claim 20 wherein the light field images of the three-dimensional scene are captured by a plenoptic imaging system, and the deterministic relation between the (scale) and (depth) domains are determined by the plenoptic imaging system.

22. The method of claim 20 wherein the step of processing the (image,scale,depth) domain representation includes searching over the (image,scale,depth) domain, and the deterministic relation between the (scale) and (depth) domains reduces a computational complexity of the search.

23. A non-transitory tangible computer readable medium containing computer program code for instructing a computer system to implement a method for processing light field images of a three-dimensional scene, the method comprising:

accessing an (image,view) domain representation of the light field images of the three-dimensional scene, wherein the (image,view) domain representation of the light field images is a representation of the light field images as a function of (image) and (view) dimensions;

applying a scale-depth transform to transform the (image,view) domain representation to an (image, scale, depth) domain representation, wherein the (image,scale,depth) domain representation is a representation of the light field images as a function of (image), (scale) and (depth) dimensions; and

processing the (image,scale,depth) domain representation of the three-dimensional scene, wherein processing the (image,scale,depth) domain representation comprises at least one of (a) estimating depth in the three-dimensional scene based on processing the (image,scale,depth) domain representation or (b) extracting three-dimensional features in the three-dimensional scene based on processing the (image,scale,depth) domain representation.

24. The non-transitory tangible computer readable medium of claim 23 wherein the (scale) portion of the scale-depth transform is based on a Gaussian kernel or one of its derivatives.

25. The non-transitory tangible computer readable medium of claim 23 wherein the (depth) portion of the scale-depth transform is based on points at different depths in the three-dimensional scene creating different curves in the (image,view) domain.

26. The non-transitory tangible computer readable medium of claim 25 wherein the (depth) portion of the scale-depth transform is based on points at different depths in the three-dimensional scene creating rays at different angles in the (image,view) domain.

27. The non-transitory tangible computer readable medium of claim 26 wherein the method further comprises:

selecting rays in the (image,view) domain based on whether the rays have sharp edges on both sides of the ray;

wherein processing the (image,scale,depth) domain representation of the three-dimensional scene is based on processing the selected rays.

28. The non-transitory tangible computer readable medium of claim 23 wherein the scale-depth transform is based on a Ray-Gaussian kernel

29. The non-transitory tangible computer readable medium of claim 28 wherein applying the scale-depth transform comprises:

convolving the (image,view) domain representation with the Ray-Gaussian kernel or its derivative for σ

ε

{σ

₁, . . . , σ

_n, pσ

₁, . . . , pσ

_n, . . . , p^kσ

₁, . . . , p^kσ

_n} and for φ

ε

{φ

₁, . . . , φ

_m};

where p is a scaling ratio, n is the number of samples per range of scale, m is the number of samples in the depth domain, and (k+1) is the number of scales.

30. The non-transitory tangible computer readable medium of claim 28 wherein p=2.

31. The non-transitory tangible computer readable medium of claim 28 wherein applying the scale-depth transform comprises:

convolving the (image,view) domain representation with the Ray-Gaussian kernel or its derivative for σ

ε

{σ

₁, . . . , σ

_n} and for φ

ε

{σ

₁, . . . , σ

_m}; and

repeating (k−

1) times the step of downsampling the (image,view) domain representation by p and convolving with the Ray-Gaussian kernel or its derivative for σ

ε

{₁, . . . , σ

_n} and for φ

ε

{φ

₁, . . . , φ

_m};

where n is the number of samples per downsampling range of scale, m is the number of samples in the depth domain, and p is the downsampling factor.

32. The non-transitory tangible computer readable medium of claim 28 wherein applying the scale-depth transform comprises:

convolving the (image,view) domain representation with the Ray-Gaussian kernel or its derivative for σ

ε

{σ

₁, . . . , σ

_n} and for φ

ε

{φ

₁, . . . , φ

_m};

repeating (k−

1) times the step of downsampling an image portion of the (image,view) domain representation by p and convolving with the Ray-Gaussian kernel or its derivative for σ

ε

{σ

₁, . . . , σ

_n} and for {φ

^′

₁, . . . , φ

^′

_m};

where n is the number of samples per downsampling range of scale, m is the number of samples in the depth domain, and p is the downsampling factor.

33. The non-transitory tangible computer readable medium of claim 23 wherein processing the (image,scale,depth) domain representation of the three-dimensional scene comprises estimating depth in the three-dimensional scene based on processing the (image,scale,depth) domain representation.

34. The non-transitory tangible computer readable medium of claim 33 wherein the scale-depth transform is a normalized second derivative Ray-Gaussian transform, and estimating depth in the three-dimensional scene comprises finding extrema of the normalized second derivative Ray-Gaussian transform and estimating depth based on said extrema.

35. The non-transitory tangible computer readable medium of claim 34 wherein estimating depth in the three-dimensional scene further comprises detecting occlusions based on said extrema.

36. The non-transitory tangible computer readable medium of claim 34 wherein the method further comprises:

additionally applying a normalized first derivative Ray-Gaussian transform to the (image,view) domain representation;

finding extrema of the normalized first derivative Ray-Gaussian transform;

estimating edges in the three-dimensional scene based on the extrema of the normalized first derivative Ray-Gaussian transform; and

improving the depth estimates based on the estimated edges.

37. The non-transitory tangible computer readable medium of claim 23 wherein processing the (image,scale,depth) domain representation of the three-dimensional scene comprises extracting three-dimensional features in the three-dimensional scene based on processing the (image,scale,depth) domain representation.

38. The non-transitory tangible computer readable medium of claim 37 wherein the scale-depth transform is a normalized first derivative Ray-Gaussian transform;

and extracting three-dimensional features in the three-dimensional scene comprises finding extrema of the normalized first derivative Ray-Gaussian transform and extracting three-dimensional features based on said extrema.

39. The non-transitory tangible computer readable medium of claim 23 wherein the light field images of the three-dimensional scene are taken from regularly spaced viewpoints.

40. The non-transitory tangible computer readable medium of claim 23 wherein the light field images of the three-dimensional scene are captured by a plenoptic imaging system.

41. The non-transitory tangible computer readable medium of claim 23 wherein the (scale) and (depth) domains are each one-dimensional.

42. The non-transitory tangible computer readable medium of claim 23 wherein the light field images have a deterministic relation between the (scale) and (depth) domains, and the step of processing the (image,scale,depth) domain representation makes use of the deterministic relation.

43. The non-transitory tangible computer readable medium of claim 42 wherein the light field images of the three-dimensional scene are captured by a plenoptic imaging system, and the deterministic relation between the (scale) and (depth) domains are determined by the plenoptic imaging system.

44. The non-transitory tangible computer readable medium of claim 42 wherein the step of processing the (image,scale,depth) domain representation includes searching over the (image,scale,depth) domain, and the deterministic relation between the (scale) and (depth) domains reduces a computational complexity of the search.