Parameterized animation compression methods and arrangements

• US 6,593,925 B1
• Filed: 06/22/2000
• Issued: 07/15/2003
• Est. Priority Date: 06/22/2000
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Hakura, Ziyad S., Snyder, John M., Lengyel, Jerome E.
• Primary Examiner(s): Vo, Cliff N.
• Application Number:
  US09/602,376
• Time in Patent Office: 1,118 Days
• Field of Search: 345/418, 345/419, 345/420, 345/423, 345/424, 345/425, 345/427, 345/473
• US Class Current: 345/426
• CPC Class Codes: G06T 13/20 3D [Three Dimensional] animation
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 9/001 Model-based coding, e.g. wire frame

Abstract

Methods and arrangements are provided for real-time rendering of scenes having various light sources and objects having differing specular surfaces. An offline encoder is employed to parameterize images by two or more arbitrary variables allowing view, lighting, and object changes. The parameterized images are encoded as a set of per-object parameterized textures based on shading models, camera parameters, and the scene'"'"'s geometry. Texture maps are inferred from a ray-tracer'"'"'s segmented imagery to provide the best match when applied to specific graphics hardware. The parameterized textures are encoded as a multidimensional Laplacian pyramid on fixed size blocks of parameter space. This technique captures the coherence in parameterized animations and decodes directly into texture maps that are easy to load into conventional graphics hardware.

Claims

1. A method comprising:

generating image data associated with each point in a modeled parameter space of a computer-generated animation; and

selectively inferring texture data for each defined object within the parameter space.

2. The method as recited in claim 1, wherein the texture data includes data selected from a group comprising texture resolution data, texture domain mapping data, and texture sample data.

3. The method as recited in claim 1, wherein the parameter space is n-dimensional, wherein n is an integer greater than or equal to two.

4. The method as recited in claim 1, wherein the image data includes at least one type of image data selected from a group comprising synthetic renderer image data, synthetic renderer, and ray-traced image data.

5. The method as recited in claim 4, wherein the image data includes sampled data having image information from a camera and a geometry based on at least one of the following vision algorithms and range scanners.

6. The method as recited in claim 4, wherein the ray-traced image data is selectively segmented based on each defined object within the parameter space.

7. The method as recited in claim 1, further comprising:

generating scene geometry data associated with the parameter space; and

wherein selectively inferring texture data for each defined object within the parameter space further includes producing a plurality of per-object texture maps using at least the scene geometry data and the image data.

8. The method as recited in claim 1, further comprising:

generating scene lighting data associated with the parameter space; and

wherein selectively inferring texture data for each defined object within the parameter space further includes producing a plurality of per-object texture maps using at least the scene lighting data and the image data.

9. The method as recited in claim 1, further comprising:

generating scene viewing data associated with the parameter space; and

wherein selectively inferring texture data for each defined object within the parameter space further includes producing a plurality of per-object texture maps using at least the scene viewing data and the image data.

10. The method as recited in claim 1, further comprising:

generating scene geometry data, scene lighting data and scene viewing data associated with the parameter space; and

wherein selectively inferring texture data for each defined object within the parameter space further includes producing a plurality of per-object texture maps using the scene geometry data, the scene lighting data, the scene viewing data, and the image data.

11. The method as recited in claim 1, wherein selectively inferring texture data for each defined object within the parameter space further includes producing a plurality of parameter-dependent texture maps for each of the defined objects.

12. The method as recited in claim 11, wherein the parameter-dependent texture maps include texture information based on at least two parameters.

13. The method as recited in claim 12, wherein the texture information based on the at least two parameters includes radiance information.

14. The method as recited in claim 12, wherein at least one of the two parameters is associated with a parameter selected from a group comprising a time parameter, a light source position parameter, a viewpoint parameter, a surface reflectance parameter, and an object position parameter.

15. The method as recited in claim 12, wherein at least one of the two parameters is associated with a modeled parameter that is configured to provide an arbitrary-dimensional parameterized animation over a sequence of generated images.

16. The method as recited in claim 11, further comprising:

compressing at least a portion of the plurality of parameter-dependent texture maps.

17. The method as recited in claim 16, wherein compressing at least a portion of the plurality of parameter-dependent texture maps further includes:

selectively encoding the portion of the plurality of parameter-dependent texture maps as a multidimensional Laplacian pyramid based on blocks of the parameter space.

18. The method as recited in claim 17, wherein selectively encoding the portion of the plurality of parameter-dependent texture maps as a multidimensional Laplacian pyramid further includes adaptively splitting the parameter space.

19. The method as recited in claim 18, wherein the parameter space is adaptively split based on differences in coherence across different parameter dimensions.

20. The method as recited in claim 19, wherein the parameter space is adaptively split based on separate diffuse and specular lighting layers.

21. The method as recited in claim 11, further comprising:

transporting at least a portion of the plurality of parameter-dependent texture maps.

22. The method as recited in claim 11, further comprising:

selectively rendering a two-dimensional image of at least a portion of the parameter space using the plurality of parameter-dependent texture maps.

23. The method as recited in claim 22, wherein selectively rendering a two-dimensional image of at least a portion of the parameter space using the plurality of parameter-dependent texture maps further includes rendering one frame at a time at one point of the parameter space, such that a sequence of images can be generated for user navigation through the parameter space.

24. A computer-readable medium having computer-executable instructions for causing at least one processing unit to perform steps comprising:

generating image data associated with each point in a modeled parameter space of a computer-generated animation; and

selectively inferring texture data for each defined object within the parameter space.

25. The computer-readable medium as recited in claim 24, wherein the texture data includes data selected from a group comprising texture resolution data, texture domain mapping data, and texture sample data.

26. The computer-readable medium as recited in claim 24, further comprising computer-executable instructions for causing the at least one processing unit to perform steps comprising:

generating scene geometry data associated with the parameter space; and

wherein selectively inferring texture data for each defined object within the parameter space further includes producing a plurality of per-object texture maps using at least the scene geometry data and the image data.

27. The computer-readable medium as recited in claim 24, further comprising computer-executable instructions for causing the at least one processing unit to perform steps comprising:

generating scene lighting data associated with the parameter space; and

wherein selectively inferring texture data for each defined object within the parameter space further includes producing a plurality of per-object texture maps using at least the scene lighting data and the image data.

28. The computer-readable medium as recited in claim 24, further comprising computer-executable instructions for causing the at least one processing unit to perform steps comprising:

generating scene viewing data associated with the parameter space; and

wherein selectively inferring texture data for each defined object within the parameter space further includes producing a plurality of per-object texture maps using at least the scene viewing data and the image data.

29. The computer-readable medium as recited in claim 24, further comprising computer-executable instructions for causing the at least one processing unit to perform steps comprising:

generating scene geometry data, scene lighting data and scene viewing data associated with the parameter space; and

wherein selectively inferring texture data for each defined object within the parameter space further includes producing a plurality of per-object texture maps using the scene geometry data, the scene lighting data, the scene viewing data, and the image data.

30. The computer-readable medium as recited in claim 24, wherein selectively inferring texture data for each defined object within the parameter space further includes producing a plurality of parameter-dependent texture maps for each of the defined objects.

31. The computer-readable medium as recited in claim 30, further comprising computer-executable instructions for causing the at least one processing unit to perform steps comprising:

compressing at least a portion of the plurality of parameter-dependent texture maps.

32. The computer-readable medium as recited in claim 31, wherein compressing at least a portion of the plurality of parameter-dependent texture maps further includes:

selectively encoding the portion of the plurality of parameter-dependent texture maps as a multidimensional Laplacian pyramid based on blocks of the parameter space.

33. The computer-readable medium as recited in claim 30, further comprising computer-executable instructions for causing the at least one processing unit to perform steps comprising:

transporting at least a portion of the plurality of parameter-dependent texture maps.

34. The computer-readable medium as recited in claim 30, further comprising computer-executable instructions for causing the at least one processing unit to perform steps comprising:

selectively rendering a two-dimensional image of at least a portion of the parameter space using the plurality of parameter-dependent texture maps.

35. An apparatus comprising:

a first renderer configured to generate image data associated with each point in a modeled parameter space of a computer-generated animation; and

a compiler operatively coupled to the first renderer and configured to selectively infer texture data for each defined object within the parameter space.

36. The apparatus as recited in claim 35, wherein the texture data includes data selected from a group comprising texture resolution data, texture domain mapping data, and texture sample data.

37. The apparatus as recited in claim 35, wherein the parameter space is n-dimensional, wherein n is an integer greater than or equal to two.

38. The apparatus as recited in claim 35, wherein the first renderer generates synthetic renderer image data.

39. The apparatus as recited in claim 38, wherein the synthetic renderer image data includes ray-traced image data.

40. The apparatus as recited in claim 38, wherein compiler selectively segments the ray-traced image data based on each defined object within the parameter space.

41. The apparatus as recited in claim 35, wherein:

the first renderer is further configured to generate scene geometry data associated with the parameter space; and

the compiler is further configured to produce a plurality of per-object texture maps using at least the scene geometry data and the image data.

42. The apparatus as recited in claim 35, wherein:

the first renderer is further configured to generate scene lighting data associated with the parameter space; and

the compiler is further configured to produce a plurality of per-object texture maps using at least the scene lighting data and the image data.

43. The apparatus as recited in claim 35, wherein:

the first renderer is further configured to generate scene viewing data associated with the parameter space; and

the compiler is further configured to produce a plurality of per-object texture maps using at least the scene viewing data and the image data.

44. The apparatus as recited in claim 35, wherein:

the first renderer is further configured to generate scene geometry data, scene lighting data and scene viewing data associated with the parameter space; and

the compiler is further configured to produce a plurality of per-object texture maps using the scene geometry data, the scene lighting data, the scene viewing data, and the image data.

45. The apparatus as recited in claim 35, wherein the compiler is further configured to produce a plurality of parameter-dependent texture maps for each of the defined objects.

46. The apparatus as recited in claim 45, further comprising:

an encoder operatively coupled to the compiler and configured to compress at least a portion of the plurality of parameter-dependent texture maps.

47. The apparatus as recited in claim 46, wherein the encoder is further configured to selectively encode the portion of the plurality of parameter-dependent texture maps as a multidimensional Laplacian pyramid based on blocks of the parameter space.

48. The apparatus as recited in claim 45, further comprising:

a communication media operatively coupled to the encoder and configured to transport at least a portion of the plurality of parameter-dependent texture maps.

49. The apparatus as recited in claim 48, further comprising:

a second renderer operatively coupled to the communication media and configured to selectively render a two-dimensional image of at least a portion of the parameter space using the plurality of transported parameter-dependent texture maps.

50. The apparatus as recited in claim 49, wherein the second renderer selectively renders one frame at a time at one point of the parameter space, such that a sequence of images can be generated for user navigation through the parameter space.

51. The apparatus as recited in claim 49, wherein the second renderer is further configured to decode and decompress the plurality of transported parameter-dependent texture maps, when applicable.

52. A method for rendering an arbitrary-dimensional parameterized animation, the method comprising:

for at least one object within a scene, parameterizing a radiance field based on at least one parameter selected from a group comprising time, lighting, viewpoint, reflectance, object positions, and degrees of freedom in a scene, resulting in an arbitrary-dimensional parameterized animation; and

encoding image data associated with the parameterized animation.

53. The method as recited in claim 52, further comprising transporting the encoded image data.

54. The method as recited in claim 53, further comprising:

selectively decoding at least a portion of the transported encoded image data; and

rendering a visually explorable image based on the decoded image data.

55. The method as recited in claim 54, wherein parameterizing the radiance field further includes selectively inferring parameter-dependent texture maps for individual objects.

56. A method for encoding ray-traced images for each point in a parameter space associated with an n-dimensional frame sequence as generated by a high quality renderer, the method comprising:

providing image data to a compiler along with related scene geometry information, lighting model information, and viewing parameter information; and

using a compression engine that is configured to implement a multi-dimensional compression scheme to encode the complied image data.

57. The method as recited in claim 56, wherein the compiler is operatively configured for use with a specific runtime decoder.

58. The method as recited in claim 57, wherein the compiler is configured to infer texture resolution information, texture domain mapping information, and texture sample information for each object within a parameter space represented by the image data.

59. The method as recited in claim 58, further comprising:

providing at least a portion of the encoded image data to a decoder;

with the decoder, decoding the portion of the encoded image data using a texture decompression engine, and rendering decoded image data using a rendering engine.

60. A method for inferring, for each geometric object, a parameterized texture based on ray-traced images, the method comprising:

segmenting ray-traced images into per-object portions by generating a per-object mask image as well as a combined image, each at supersampled resolutions;

for each object, filtering a relevant portion of the combined image as indicated by the object'"'"'s respective mask and dividing by a fractional coverage computed by applying a filter to the object'"'"'s mask.

61. The method as recited in claim 60, wherein the filter includes a gaussian filter.

62. A method for use with a computer, the method comprising the steps of:

developing a matrix A that is an n_s×

n_t matrix, where n_s is an integer equal to a number of screen pixels in which an object is visible with in a graphically depicted scene, and n_t is an integer that is equal to a number of texels in a corresponding texture MIPMAP pyramid for the object; and

solving for a texture represented by a vector x by minimizing a function f(x) defined as

63. The method as recited in claim 62, further comprising selectively scaling up the solving step until a desired texture resolution is reached as part of a multiple resolution solution and wherein the multiple resolution solution is extendable to an n-dimensional parameter space, wherein n is an integer greater than or equal to two.

64. The method as recited in claim 63, wherein the multiple resolution solution further includes selectively segmenting ray-traced images into view-dependent and view-independent layers so as to compute a single diffuse texture as follows:

$\stackrel{\stackrel{A^{\mathrm{\prime <br><br>}}}{\mathrm{<br><br>}}}{\left[\begin{array}{c}{A}_{v_0}\\ A_{v_1}\\ \mathrm{⋮<br><br>}\\ \text{\hspace{1em}<br><br>}\\ A_{v_{n-1}}\end{array}\right]}<br><br>\text{\hspace{1em}<br><br>}<br><br>\stackrel{\stackrel{x}{\mathrm{<br><br>}}}{\left[\begin{array}{c}\text{\hspace{1em}<br><br>}\\ \text{\hspace{1em}<br><br>}\\ x\\ \text{\hspace{1em}<br><br>}\\ \text{\hspace{1em}<br><br>}\end{array}\right]}=\stackrel{\stackrel{b^{\mathrm{\prime <br><br>}}}{\mathrm{<br><br>}}}{\left[\begin{array}{c}{b}_{v_0}\\ b_{v_1}\\ \mathrm{⋮<br><br>}\\ \text{\hspace{1em}<br><br>}\\ b_{v_{n-1}}\end{array}\right]}$

wherein matrix A′

concatenates A matrices for individual viewpoints v₀ through v<sub>n−

1</sub>, vector b′

concatenates ray-traced images at corresponding viewpoints, and vector x represents a diffuse texture to be solved.

65. The method as recited in claim 63, further comprising:

providing a pyramidal regularization term that support a controlled degradation away from original parameter samples and to further improve spatial coherence.

66. The method as recited in claim 65, wherein the pyramidal regularization term has the form of:

$f_{\mathrm{reg}-\mathrm{pyramid}}<br><br>\left(x\right)=f_{\mathrm{reg}-01}<br><br>\left(x\right)+E_f<br><br>\left(\frac{n_s}{n_l}\right)<br><br>\mathrm{\Gamma <br><br>}<br><br>\left(x\right)$

where Γ

(x) includes a difference between the texels at each level of the MIPMAP with an interpolated version of a next coarser level, factor n_s/n_l, provides a magnitude roughly equal with f, and E_f is a definable constant.

67. The method as recited in claim 66, wherein the pyramidal regularization term is computed by taking the sum of squared differences between texels at each level of the MIPMAP with the interpolated image of the next higher level.

68. The method as recited in claim 62, further comprising selectively optimizing texture coordinates and resolutions.

69. The method as recited in claim 68, wherein optimizing texture coordinates and resolutions further includes:

identifying texture coordinates that minimize the texture area actually needed to render an object within a block of the parameter space; and

encoding a global affine transformation on any associated texture coordinates.

70. The method as recited in claim 69, wherein the encoding further includes:

(1) repositioning branch cut in texture dimensions that have wrapping enabled;

(2) calculating a least-squares solution for the most isometric affine transformation;

(3) computing a maximum singular value of a Jacobian of texture to screen space mapping and scale transformation along a direction of maximal stretch

(4) repeating step (3) until the maximum singular value is below a defined threshold;

(6) identifying a bounding rectangle with minimum area; and

(7) determining a texture resolution.

71. A parameterized texture compression method comprising:

generating at least one block of parameterized texture for at least one object within a multidimensional parameter space, wherein the texture is parameterized by a plurality of spatial parameters; and

encoding the parameter space using a block-based compression scheme configured to exploit spatial coherence within each of the parameterized textures.

72. The method as recited in claim 71, wherein the block-based compression scheme further includes an adaptive Laplacian pyramid configured to encode the parameterized texture blocks using a Laplacian pyramid, wherein each level of the pyramid represents detail that should be added to the sum of higher levels in order to reconstruct an image of the object during subsequent rendering.

73. The method as recited in claim 72, wherein each of the levels of the Laplacian pyramid include a series of encoded two-dimensional images.

74. The method as recited in claim 73, further comprising:

providing automatic storage allocation, whereby during the encoding of the Laplacian pyramid, storage of information is assigned to the various levels of the pyramid so as to minimize the sum of mean squared errors (MSEs) since a texture image at a given point in parameter space can be reconstructed as a sum of images from each level.

75. The method as recited in claim 74, further comprising:

for objects with both specular and diffuse reflectance information, encoding separate lighting layers for which storage is allocated.

76. A method of compensation for gamma correction in image-based rendering, the method comprising:

selectively splitting an object'"'"'s lighting layers into a sum of two terms L₁ and L₂ such that the two terms conflict with a gamma correction, since γ

(L₁+L₂)≠

γ

(L₁)+γ

(L₂) wherein γ

(x)=x^1/g is a nonlinear gamma correction function; and

selectively encoding corresponding object image data based on a gamma corrected signals, γ

(L_l), by controlling compression errors associated with dark regions in the object'"'"'s image.

77. The method as recited in claim 76, further comprising:

selectively decoding the encoding corresponding image data using an inverse gamma correction function γ

<sup>−

1</sup>(x)=x^g.

78. An apparatus configured to cache encoded texture images selectively decode the cached texture images and apply encoded affine transformations to vertex texture coordinates.

79. The apparatus as recited in claim 78, further configurable to generate at least one call to a graphics system capable of rendering an image based on decoded image information.

80. The apparatus as recited in claim 78, further configured to selectively store decoded image information based on an adaptive caching strategy.

81. The apparatus as recited in claim 80, wherein the adaptive caching strategy is based at least in part on a lifetime associated with the texture image information.

82. The apparatus as recited in claim 80, wherein the adaptive caching strategy is based on texture image information having different levels of priority.

83. The apparatus as recited in claim 78, wherein the cached texture images have an associated Laplacian pyramidal encoding scheme.

84. The apparatus as recited in claim 83, wherein the Laplacian pyramidal encoding scheme includes a plurality of blocks that are selectively staggered.

85. The apparatus as recited in claim 84, wherein the plurality of blocks that are selectively staggered by having different block origins for different objects within a parameter space.