Techniques for spatial displacement estimation and multi-resolution operations on light fields

• US 6,476,805 B1
• Filed: 12/23/1999
• Issued: 11/05/2002
• Est. Priority Date: 12/23/1999
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Chan, Shing-Chow, Shum, Heung-Yeung
• Primary Examiner(s): Vo, Cliff N.
• Application Number:
  US09/471,678
• Time in Patent Office: 1,048 Days
• Field of Search: 345/418, 345/419, 345/420, 345/421, 345/422, 345/423, 345/424, 345/426
• US Class Current: 345/420
• CPC Class Codes: G06T 2207/20228 Disparity calculation for image-based rendering
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85

Abstract

Selective quality light field operations efficiently manipulate a multi-resolution representation of a light field. These operations include intra-image and inter-image decomposition and compression of a light field to a multi-resolution representation. These operations also include intra-image and inter-image decompression and reconstruction of a light field at selective quality. These selective quality operations also apply to storage, rendering, and transmission. Various techniques improve spatial displacement estimation of a prediction light field image from a reference light field image. These techniques includes constraining the placement and size of a search window based upon a geometrical relationship between prediction and reference light field images, hierarchical spatial displacement estimation, edge extension of a reference light field image, differential coding of displacement vectors, and multi-predictor spatial displacement estimation. Configuring reference and prediction light field images in view of geometrical relationships between light field images also improves spatial displacement estimation.

Claims

1. A method for performing a selective granularity operation involving a multi-resolution representation of one Or more light field images of a light field, the method comprising:

providing one or more light field images of a light field; and

performing a selective granularity operation involving a multi-resolution representation of the one or more light field images, wherein the multi-Tesolution representation of the one or more light field images comprises a set of plural layers, wherein the set of plural layers includes a low granularity component layer and one or more higher granularity component layers, and wherein each of the one or more higher granularity component layers represents less significant information about the one or more light field images than the low granularity component layer.

2. The method of claim 1 wherein granularity corresponds to spatial frequency, and wherein the performing comprises:

separating a low frequency component layer from a light field image;

separating one or more higher frequency component layers from the light field image; and

sub-sampling one or more of the frequency component layers to reduce representation.

3. A computer-readable medium having computer-executable instructions for performing the method of claim 2.

4. The method of claim 2 wherein one or more integer filter banks that use integer wavelet transforms separate the frequency component layers of the light field image.

5. The method of claim 2 wherein the separating limits the spatial frequency range of a frequency component layer by a factor of n, and wherein sub-sampling reduces representation of a frequency component layer by a factor of n.

6. The method of claim 2 wherein the separating a low frequency component layer comprises bandpass filtering the light field image to remove a high frequency component.

7. The method of claim 6 wherein the bandpass filtering occurs in the horizontal and vertical directions.

8. The method of claim 2 wherein the separating one or more higher frequency component layers comprises, for a frequency component layer, bandpass filtering the light field image to remove at least a low frequency component in at least one direction.

9. The method of claim 8 wherein the bandpass filtering occurs in the horizontal direction and/or vertical direction.

10. The method of claim 2 further comprising:

frequency decomposing the low frequency component layer into a multi-resolution representation of the low frequency component layer;

forming plural wavelet blocks from the multi-resolution representation of the low frequency component layer; and

compressing the wavelet blocks by zero-tree coding using successive approximation quantization and arithmetic coding.

11. The method of claim 2 further comprising:

for a section of spatial domain coefficients of a frequency component layer,

transform coding the section into transform coefficients;

quantizing the transform coefficients; and

entropy coding the quantized transform coefficients.

12. The method of claim 1 wherein granularity corresponds to degree of spatial displacement estimation refinement for compressing representation of a light field by reducing spatial redundancy between light field images, and wherein the performing comprises:

for a prediction light field image,

for each of plural groups of pieces of the prediction light field image,

determining a corresponding group of pieces predictor of a reference light field image;

determining a displacement vector based upon the corresponding group of pieces predictor, wherein the displacement vector indicates a spatial transformation; and

representing the prediction light field image based upon displacement vectors determined for the plural groups of pieces of the prediction light field image.

13. A computer-readable medium having computer-executable instructions for performing the method of claim 12.

14. The method of claim 12 wherein the determining a corresponding group of pieces predictor of a reference light field image comprises:

establishing a search window within the reference light field image; and

within the search window, finding a corresponding group of pieces predictor that minimizes a deviation measure for the group of pieces of the prediction light field image.

15. The method of claim 12 further comprising:

for each piece of a group of pieces of the prediction light field image,

determining a corresponding piece predictor in the reference light field image; and

determining a displacement difference vector based upon the corresponding piece predictor and the displacement vector for the group of pieces.

16. The method of claim 15 wherein the determining a corresponding piece predictor comprises:

establishing a piece-level search window within the reference light field image; and

within the piecelevel search window, finding a corresponding piece predictor that minimizes a deviation measure for the piece of the prediction light field image.

17. The method of claim 16 wherein the position of the piece within the group of pieces constrains placement of the piece-level search window around a corresponding position within the reference light field image.

18. The method of claim 15 wherein pieces of the prediction light field image are macroblocks.

19. The method of claim 15 wherein pieces of the prediction light field image are constituent blocks of macroblocks.

20. The method of claim 12 further comprising:

for each piece of a group of pieces of the prediction light field image,

determining a corresponding piece predictor in the reference light field image;

determining a residual based upon the difference between the piece of the prediction light field image and the corresponding piece predictor.

21. The method of claim 20 further comprising:

comparing a deviation measure for the residual to a failure threshold; and

if the deviation measure exceeds the failure threshold, intra-coding the piece of the prediction light field image.

22. The method of claim 20 further comprising:

comparing a deviation measure for the residual to a minimum residual threshold;

if the deviation measure is equal to or below the minimum residual threshold,

discarding the residual; and

designating the piece of the prediction light field image as lacking a residual.

23. The method of claim 12 further comprising:

for each of the plural groups of pieces of the prediction light field image,

determining a corresponding group of pieces predictor in each of plural reference light field images; and

for each corresponding group of pieces predictor, determining a displacement vector based upon the corresponding group of pieces predictor, wherein a first piece of a group of pieces estimates spatial displacement based upon a first selective displacement vector for the group of pieces, and wherein a second piece of the group of pieces estimates spatial displacement based upon a second selective displacement vector for the group of pieces.

24. The method of claim 1 wherein granularity corresponds to spatial frequency, and wherein the performing comprises:

for each of one or more light field images, decompressing a low frequency component layer;

for a point of a novel perspective light field image,

determining at least one match point in the one or more light field images; and

for a match point, selectively decompressing higher frequency component layer information in the light field image that includes the match point.

25. A computer-readable medium having computer-executable instructions for performing the method of claim 24.

26. The method of claim 24 wherein a match area includes the match point, and wherein higher frequency component layer information for the match area is selectively decompressed.

27. The method of claim 24 further comprising:

conditionally bypassing the selectively decompressing higher frequency component layer information for a match point.

28. The method of claim 27 wherein the conditionally bypassing comprises:

checking a flag value that indicates availability of the higher frequency component layer information for the match point; and

if the higher frequency component layer information for the match point is unavailable, bypassing the selectively decompressing.

29. The method of claim 27 wherein the conditionally bypassing depends upon a processor, in memory, or other system constraint.

30. The method of claim 24 further comprising:

after the selectively decompressing, calculating a reproduction value based upon decompressed low frequency component layer information for the match point and decompressed higher frequency component layer information for the match point; and

assigning the reproduction value to the point of the novel perspective view.

31. The method of claim 1 wherein granularity corresponds to degree of spatial displacement estimation refinement for compressing representation of a light field by reducing spatial redundancy between light field images, and wherein the performing comprises:

for each of one or more light field images, decompressing rough spatial displacement estimation information;

for a point of a novel perspective light field image,

determining at least one match point in the one or more light field images; and

for a match point, selectively decompressing spatial displacement estimation refinement information.

32. A computer-readable medium having computer-executable instructions for performing the method of claim 31.

33. The method of claim 31 wherein the rough spatial displacement estimation information comprises one or more displacement vectors for each of one or more sections of a light field image.

34. The method of claim 33 wherein the spatial displacement estimation refinement information comprises one or more displacement difference vectors for each of one or more sub-sections of a section.

35. The method of claim 33 wherein the spatial displacement estimation refinement information comprises a residual for each of one or more sub-sections of a section.

36. The method of claim 33 wherein the spatial displacement estimation refinement information comprises a displacement vector selector for a sub-section of a season with plural displacement vectors.

37. The method of claim 31 wherein a match area includes the match point, and wherein spatial displacement estimation refinement information for the match area is selectively decompressed.

38. The method of claim 31 further comprising:

conditionally bypassing the selectively decompressing spatial displacement estimation refinement information for a match point.

39. The method of claim 38 wherein the conditionally bypassing comprises:

checking a flag value that indicates availability of the spatial displacement estimation refinement information for the match point; and

if the spatial displacement estimation refinement information for the match point is unavailable, bypassing the selectively decompressing.

40. The method of claim 38 wherein the conditionally bypassing depends upon a processor, memory, or other system constraint.

41. The method of claim 1 wherein granularity for a prediction light field image corresponds to degree of spatial displacement estimation refinement for compressing representation of a light field by reducing spatial redundancy between light field images, wherein granularity for a reference light field image corresponds to spatial frequency, and wherein the performing comprises:

for each of one or more reference light field images of the one or more light field images, decompressing a low frequency component layer;

for one or more prediction light field images of the one or more light field images, decompressing rough spatial displacement estimation information; and

selectively decompressing enhancement information.

42. A computer-readable medium having computer-executable instructions for performing the method of claim 41.

43. The method of claim 41 wherein the enhancement information includes information for one or more higher frequency component layers.

44. The method of claim 41 wherein the rough spatial displacement estimation information comprises one or more displacement vectors for each of one or more sections of a prediction light field image, and wherein the enhancement information includes residuals, one or more displacement difference vectors, and/or selectors of displacement vectors.

45. The method of claim 41 further comprising:

conditionally bypassing the selectively decompressing enhancement information based upon a flag value.

46. The method of claim 41 further comprising:

conditionally bypassing the selectively decompressing enhancement information based upon processor capacity, availability of computer memory, or available transmission capacity.

47. The method of claim 41 further comprising:

for a point of a novel perspective light field image,

determining at least one match point in the light field; and

for a match point, performing the act of selectively decompressing enhancement information.

48. The method of claim 47 wherein a match area includes the match point, and wherein enhancement information for the match area is selectively decompressed.

49. The method of claim 47 further comprising:

conditionally bypassing the selectively decompressing enhancement information for the match point;

if a reference light field image includes the match point, calculating a reproduction value based upon decompressed low frequency component layer information for the match point,

otherwise, calculating the reproduction value based upon decompressed low frequency component layer information and decompressed rough spatial displacement estimation information for the match point; and

assigning the reproduction value to the point of the novel perspective light field image.

50. A method for selectively accessing enhancement information for a set of plural spatially-related views of an object or static scene, wherein a data structure stores a spatially-related view, the data structure comprising a base field and an enhancement field, the method comprising:

for a point of a novel perspective view, determining at least one match point in the view stored in the data structure;

for a match point of the at least one match point in the view,

accessing information in the base field; and

selectively accessing information in the enhancement field.

51. A computer-readable medium having computer-executable instructions for performing the method of claim 50.

52. The method of claim 50 wherein the information in the base field is compressed and the information in the enhancement field is compressed, the method further comprising:

before the determining the at least one match point, decompressing the information in the base field; and

after the selectively accessing information, decompressing the enhancement information.

53. The method of claim 50 wherein the view stored in the data structure is a reference view, wherein the base field information includes a low frequency component layer information, and wherein the enhancement field information includes information for higher frequency component layers.

54. The method of claim 50 wherein the view stored in the data structure is a prediction view, wherein the base field information includes rough spatial displacement estimation information, and wherein the enhancement field information includes spatial displacement estimation refinement information.

55. The method of claim 50 wherein a match area includes the match point, wherein the enhancement field is organized by match area, and wherein enhancement field information for the match area that includes the match point is selectively accessed.

56. The method of claim 50 wherein the data structure further comprises an array of flag values, the method further comprising:

conditionally bypassing the selectively accessing information in the enhancement field for a match point based upon a flag value in the array of flag values.

57. A computer-readable medium having stored thereon computer-executable instructions for selective transmission of enhancement information for a set of plural spatially-related views of an object or static scene over a network, the computer-executable instructions for performing the acts of:

for each of one or more reference views from a set of plural spatially-related views of an object or static scene, transmitting low frequency component layer information;

for each of one or more prediction views from the set of spatially-related views, transmitting rough spatial displacement estimation information;

selectively transmitting enhancement information based upon a network condition.

58. A computer-readable medium having stored thereon computer-executable instructions for causing a computer system programmed thereby to perform a method of processing data in a data structure for storing a multi-resolution representation of a light field image, the data structure comprising a base field and an enhancement field, the method comprising:

processing base field data representing low granularity information for a light field image; and

processing enhancement field data representing higher granularity information for the light field image, wherein the higher granularity information represents less significant detail of the light field image On the low granularity information.

59. The computer-readable medium of claim 58 wherein the method further comprises:

processing an array of plural flag values that indicate the presence or absence of information in the enhancement field.

60. The computer-readable medium of claim 58 wherein the base field data is decompressed.

83. The computer-readable medium of claim 58 wherein the processing the base field data comprises compressing the base field data, and wherein the processing the enhancement field data comprises compressing the enhancement field data.

84. The computer-readable medium of claim 58 wherein the processing the base field data comprises decompressing the base field data, and wherein the processing the enhancement field data comprises decompressing the enhancement field data.

85. The computer-readable medium of claim 58 wherein the processing the base field data comprises accessing the base field data, and wherein the processing the enhancement field data comprises accessing the enhancement field data.

86. The computer-readable medium of claim 58 wherein the processing the base field data comprises transmitting the base field data, and wherein the processing the enhancement field data comprises transmitting the enhancement field data.

87. The computer-readable medium of claim 58 wherein the processing the base field data comprises storing the base field data, and wherein the processing the enhancement field data comprises storing the enhancement field data.

61. A method for compressing a representation of a set of plural spatially-related views by reducing spatial redundancy between the spatially-related views, the method comprising:

for a prediction view from a set of plural spatially-related views,

for each of plural sections of the prediction view,

determining a corresponding section predictor in a reference view; and

determining a displacement vector based upon the corresponding section predictor, wherein the displacement vector indicates a spatial transformation; and

representing the prediction view based upon displacement vectors determined for the plural sections of the prediction view.

62. A computer-readable medium having computer-executable instructions for performing the method of claim 61.

63. The method of claim 61 wherein a section of the prediction view comprises a group of plural pieces.

64. The method of claim 61 wherein the determining a corresponding section predictor in a reference view comprises:

establishing a search window within the reference view; and

within the search window, finding a corresponding section predictor that minimizes a deviation measure for the section of the prediction view.

65. The method of claim 64 further comprising:

constraining placement of the search window based upon a geometrical relationship between the reference view and the prediction view.

66. The method of claim 64 further comprising:

reducing the size of the search window based upon a geometrical relationship between the reference view and the prediction view.

67. The method of claim 64 further comprising:

extending an edge of the reference view, whereby a corresponding section predictor at least partially lies outside the reference view.

68. The method of claim 61 further comprising:

after the determining a displacement vector, determining a residual based upon the difference between the section of the prediction view and the corresponding section predictor;

comparing a deviation measure for the residual to a failure threshold; and

if the deviation measure exceeds the failure threshold, intra-coding the section of the prediction view.

69. The method of claim 61 further comprising:

after the determining a displacement vector, determining a residual based upon the difference between the section of the prediction view and the corresponding section predictor;

comparing a deviation measure for the residual to a minimum residual threshold;

if the deviation measure is equal to or below the minimum residual threshold,

discarding the residual; and

designating the section of the prediction view as lacking a residual.

70. The method of claim 61 further comprising:

differentially coding the displacement vectors.

71. The method of claim 61 wherein the set of plural spatially-related views includes plural reference views, the method further comprising:

for each of the plural sections of the prediction view,

determining a corresponding section predictor in each of the plural reference views; and

for each corresponding section predictor, determining a displacement vector based upon the corresponding section predictor;

wherein a prediction view section estimates spatial displacement based upon a selective one of the displacement vectors.

72. A method for rendering a novel perspective view from a set of plural compressed spatially-related views of an object or static scene, the method comprising:

decompressing information for one or more reference views of the set of spatially-related views;

decompressing information for a prediction view of the set of spatially-related views, wherein the prediction view estimates spatial displacement from at least one reference view;

for a point of a novel perspective view,

determining at least one match point in the decompressed views;

calculating a reproduction value based upon decompressed information for the at least one match points; and

assigning the reproduction value to the point of the novel perspective view.

73. A computer-readable medium having computer-executable instructions for performing the method of claim 72.

74. The method of claim 72 wherein the decompressed information for the one or more reference views comprises low frequency component layers, wherein the decompressed information for the prediction view comprises rough spatial displacement estimation, the method further comprising;

for the point of the novel perspective view, selectively decompressing enhancement information to improve the quality of the reproduction value.

75. A method for configuring a set of spatially-related views, the method comprising:

arranging a set of spatially-related views in a configuration that reflects the geometrical relationships between the spatially-related views;

designating as reference views one or more views from the set of spatially-related views;

designating as prediction views one or more views from the set of spatially-related views that are not designated as reference views;

representing each prediction view in terms of plural displacement vectors from one or more of the reference views.

76. A computer-readable medium having computer-executable instructions for performing the method of claim 75.

77. The method of claim 75 wherein plural spatially-related views are designated as reference views, and wherein a prediction view is represented in terms of displacement from multiple reference views.

78. The method of claim 75 wherein the configuration is two-dimensional.

79. The method of claim 78 wherein the configuration is rectangular, and wherein the views at the corners of the rectangular configuration are reference views.

80. The method of claim 79 wherein a prediction view in the same row or column of the rectangular configuration as two reference views is represented in terms of displacement from up to both of the two reference views.

81. The method of claim 79 wherein a prediction view in the interior of the rectangular configuration is represented in terms of displacement from up to four of the corner reference views.

82. The method of claim 75 wherein reference views appear at regular intervals in the configuration of spatially-related views.

88. A computer system for performing a selective granularity operation involving a multi-resolution representation of one or more light field images of a light field, the computer system comprising:

memory storing computer-executable instructions for causing the computer system to perform a selective granularity operation involving a multi-resolution representation of the one or more light field images, wherein the multi-resolution representation of the one or more light field images comprises a set of plural layers, wherein the set of plural layers includes a low granularity component layer and one or more higher granularity component layers, and wherein each of the one or more higher granularity component layers represents less significant information about the one or more light field images than the low granularity component layer; and

one or more processors for performing the selective granularity operation.

89. The computer system of claim 88 wherein the selective granularity operation includes decompressing the low granularity component layer and selectively decompressing the one or more higher granularity component layers.

90. The computer system of claim 88 wherein the selective granularity operation includes compressing the one or more light field images into the multi-resolution representation.

91. The computer system of claim 88 wherein the selective granularity operation includes directing transmission of the low granularity component layer and directing selective transmission of the one or more higher granularity component layers.

92. The computer system of claim 88 wherein the selective granularity operation includes accessing the low granularity component layer and selectively accessing the one or more higher.