Method for measuring the fidelity of warped image layer approximations in a real-time graphics rendering pipeline

• US 6,064,393 A
• Filed: 07/30/1997
• Issued: 05/16/2000
• Est. Priority Date: 08/04/1995
• Status: Expired due to Term

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Snyder, John, Kajiya, James T., Lengyel, Jerome E.
• Primary Examiner(s): Powell, Mark R.
• Assistant Examiner(s): GOOD JOHNSON, MOTILEWA
• Application Number:
  US08/904,245
• Time in Patent Office: 1,021 Days
• Field of Search: 345/421, 345/423, 345/426, 345/427, 345/433, 345/435, 345/437-439, 345/473
• US Class Current: 345/427
• CPC Class Codes: G06T 15/005 General purpose rendering architectures
• CPC Class Codes: G06T 15/04 Texture mapping

Abstract

A layered graphics rendering pipeline measures image fidelity ("fiducials") to determine how accurately a transformed image layer approximates a rendering of a 3D object. The graphics rendering pipeline approximates the change in position or color of 3D object by transforming a rendering of the 3D object from a previous frame. The pipeline uses the fiducials to control rendering of factored scene elements to independent image layers. The pipeline then combines the layers to compute frames of animation. The types of fiducials include sampling, visibility, and photometric fiducials. The sampling fiducial measures the distortion of an image sample when warped to screen coordinates. The visibility fiducial measures the change in visibility of a scene element since a previous rendering of the scene element. The photometric fiducial measures either the change in lighting from the time of the previous rendering to the current time, or it measures the difference between warped color samples and actual color samples of the scene element for a current frame.

Claims

1. In a real time graphics rendering pipeline, a method for measuring fidelity of a transformed image layer that approximates motion of 3D object geometry, comprising:

factoring a scene into scene elements, where the scene elements include a 3D object or set of 3D objects in the scene;

independently rendering the scene elements to separate image layers;

computing a 2D transform that approximates motion of a scene element from an initial frame to a current frame in an animation sequence;

computing sampling distortion of the 2D transform;

using the sampling distortion to determine whether to re-render the scene element for the current frame;

approximating motion of the scene element by using the sprite transform to transform an initial rendering of the scene element at the initial frame to a new position in the current frame;

compositing the image layers, including at least one transformed layer, into an output image in output device coordinates to construct each frame in the animation sequence.

2. The method of claim 1 wherein the scene elements are rendered to separate image layers, each in a corresponding sprite coordinate space, and each at a non-fixed location in output device coordinates.

3. The method of claim 2 wherein at least one corresponding sprite space has a different spatial resolution from the output device coordinates.

4. The method of claim 2 wherein at least one corresponding sprite space has a different orientation from the output device coordinates.

5. The method of claim 1 wherein the step of computing the sampling distortion includes computing singular values for the 2D transform.

6. The method of claim 1 wherein the 2D transform is an affine transform that is capable of computing translation, rotation, scaling and shear of an image layer to approximate motion of object geometry depicted in the image layer.

7. The method of claim 6 including:

computing sampling distortion for image samples in a variety of locations across the initial rendering of the first scene element;

using the sampling distortion for the image samples across the initial rendering to determine whether to re-render the first scene element.

8. A computer readable medium including computer-executable instructions for performing the steps of claim 1.

9. In a real time graphics rendering pipeline, a method for measuring fidelity of a transformed image layer comprising:

factoring a scene into scene elements, where the scene elements include a 3D object or set of 3D objects in the scene;

independently rendering the scene elements to separate image layers;

computing a 2D transform that approximates motion of a scene element from an initial frame to a current frame in an animation sequence;

computing visibility distortion caused by approximating motion of the scene element with the 2D transform by determining transitions between back-facing to front-facing surfaces of characteristic geometry of the scene element from the initial frame to the current frame;

using the visibility distortion to determine whether to re-render the scene element for the current frame;

approximating motion of the scene element by using the 2D transform to transform an initial rendering of the scene element at the initial frame to a new position in the current frame;

compositing the image layers, including one or more transformed layers, into an output image in output device coordinates to construct each frame in the animation sequence.

10. The method of claim 9 wherein the image layers correspond to non-fixed portions of the output image.

11. The method of claim 9 including:

computing 2D transforms for each of the scene elements to approximate the motion of the scene elements from a previous frame to the current frame;

computing visibility distortion for each of the scene elements;

determining whether to re-render the scene elements based on the visibility distortion of each of the scene elements.

12. The method of claim 9 further including:

clipping characteristic geometry of the scene elements to an expanded viewing frustum that extends beyond screen boundaries of the output image;

tracking position of clipped points of the characteristic geometry to determine whether the clipped points, when projected to the viewing frustum, are located within the screen boundaries;

determining whether to re-render the image layers when the clipped points corresponding to the image layers are within the screen boundaries.

13. A computer readable medium having computer-executable instructions for performing the steps recited in claim 9.

14. In a real time graphics rendering pipeline, a method for measuring fidelity of a transformed image layer comprising:

factoring a scene into scene elements, where the scene elements include a 3D object or set of 3D objects in the scene;

independently rendering the scene elements to separate image layers;

computing a 2D transform that approximates motion of a scene element from an initial frame to a current frame in an animation sequence by transforming an image layer representing an initial rendering of the scene element;

measuring change in lighting of the scene element to compute photometric distortion of the 2D transform;

using the photometric distortion to determine whether to re-render the first scene element;

approximating motion of the scene element by using the 2D transform to transform an initial rendering of the scene element at the initial frame to a new position in the current frame;

to construct frames in the animation sequence, compositing the image layers, including at least one transformed layer, into an output image in output device coordinates.

15. The method of claim 14 wherein the step of computing the photometric distortion includes sampling lighting at characteristic points of the scene element at the initial frame and the current frame, and determining the change in color values at the characteristic points.

16. The method of claim 14 wherein the step of computing the photometric distortion includes determining change in position of a light source relative to the scene element from the initial frame to the current frame.

17. The method of claim 16 wherein the step of computing the photometric distortion includes determining angular change in position of a light source relative to the scene element from the initial frame to the current frame.

18. The method of claim 16 wherein the step of computing the photometric distortion includes determining change in distance of a light source relative to the scene element from the initial frame to the current frame.

19. The method of claim 16 wherein the step of computing the photometric distortion includes determining change in position of a viewpoint of the scene relative to the scene element from the initial frame to the current frame.

20. The method of claim 14 further including performing a color warp to approximate lighting change from the initial rendering to the current frame.

21. A computer-readable medium having computer executable instructions for performing the steps recited in claim 14.

22. In a real time graphics rendering pipeline, a method for measuring fidelity of a transformed image layer comprising:

factoring a scene into scene elements, where each scene element includes a 3D object or set of 3D objects in the scene;

independently rendering the scene elements to separate image layers;

computing a color warp that approximates change in lighting of a scene element from an initial frame to a current frame in an animation sequence by modifying color values in an image layer representing an initial rendering of the scene element;

measuring a difference between warped color values computed by performing the color warp on the initial rendering and color values of the scene element for the current frame to compute photometric distortion;

using the photometric distortion to determine whether to re-render the scene element;

approximating the change in lighting of the scene element by using the color warp to modify the color values in the initial rendering of the scene element at the initial frame to new color values for the current frame;

compositing the image layers, including at least one layer modified using a color warp, into an output image in output device coordinates to construct each frame in the animation sequence.

23. The method of claim 22 wherein the step of computing the photometric distortion includes sampling lighting at characteristic points of the scene element and determining how the sampled lighting differs from lighting at warped samples corresponding to the characteristic points.

24. The method of claim 22 wherein the step of computing the photometric distortion includes determining the change in position of a light source relative to the first scene element from the initial frame to the current frame.

25. The method of claim 22 wherein the color warp comprises a single color multiplier applied to samples in the initial rendering.

26. The method of claim 23 wherein the color warp comprises an array of multipliers applied to samples in the initial rendering, where the values of the multipliers in the array vary with location and are based on lighting sampled at the characteristic points of the scene element.

27. The method of claim 23 wherein the color warp comprises at least one multiplier and at least one offset applied to samples of the initial rendering to approximate lighting changes.

28. A computer-readable medium having computer executable instructions for performing the steps recited in claim 22.

29. In a real time graphics rendering pipeline, a method for measuring fidelity of a transformed image layer in an output image comprising:

factoring a scene into scene elements, where the scene elements include a 3D object or set of 3D objects in the scene;

independently rendering the scene elements to separate image layers, including rendering a scene element to a 2D sprite space at a different resolution than a resolution of the output image;

computing a 2D transform that transforms the scene element from the 2D sprite space to the screen space of a display device;

computing sampling distortion of the 2D transform;

using the sampling distortion to determine whether to render the scene element at a different spatial resolution;

using the 2D transform to transform a rendering of the scene element from the 2D sprite space to the screen space;

compositing the image layers, including at least one transformed layer, into an output image in screen space to construct each frame in the animation sequence.

30. In a real time graphics rendering pipeline, a method for measuring fidelity of a transformed image layer comprising:

factoring a scene into scene elements;

independently rendering the scene elements to separate image layers;

computing a 2D transform that approximates motion of a scene element from an initial frame to a current frame in an animation sequence;

computing sampling distortion, photometric distortion, geometric distortion and visibility distortion of a transformed image layer representing the scene element;

using the sampling distortion, photometric distortion, geometric distortion and visibility distortion to determine whether to re-render the scene element;

approximating motion of the scene element by transforming an initial rendering of the scene element at the initial frame to a new position in the current frame;

compositing the image layers into a output image in output device coordinates to construct frames in the animation sequence.

31. A method for measuring fidelity of a transformed image layer, comprising:

computing a 2D transform that approximates motion of a scene element from an initial frame to a current frame in an animation sequence;

computing a sampling distortion measure of the 2D transform; and

using the sampling distortion measure to determine accuracy of approximating motion of the scene element with the 2D transform.

32. The method of claim 31 further including:

approximating motion of the scene element by using the 2D transform to transform the initial rendering of the scene element at the initial frame to a new position in the current frame; and

compositing image layers, including at least one transformed layer, into an output image in output device coordinates to construct each frame in the animation sequence.

33. The method of claim 31 wherein the step of computing the sampling distortion measure includes computing singular values for the 2D transform.

34. The method of claim 31 wherein the 2D transform is an affine transform that is capable of computing translation, rotation, scaling and shear of an image layer to approximate motion of object geometry depicted in the image layer.

35. The method of claim 31 including:

computing sampling distortion for image samples in a variety of locations across the initial rendering of the first scene element.

36. A computer readable medium including computer-executable instructions for performing the steps of claim 31.

37. A method for measuring fidelity between instances of a scene element comprising:

computing a visibility distortion measure of the scene element; and

using the visibility distortion measure to determine distortion of an approximation of the scene element.

38. The method of claim 37 wherein the step of computing the visibility distortion measure includes:

determining transitions between back-facing to front-facing surfaces or between front-facing to back-facing surfaces of characteristic geometry of the instances of the scene element.

39. The method of claim 37 wherein the step of computing the visibility distortion measure includes:

determining whether a clipped point of the scene element intersects with screen boundaries of a display device on which a rendering of the scene element is displayed.

40. The method of claim 37 wherein the step of computing the visibility distortion measure includes:

determining transitions between front-facing to back-facing surfaces or between back-facing to front-facing surfaces of characteristic polygons of the instances of the scene element and weighting the transitions by screen area occupied by the characteristic polygons.

41. The method of claim 37 further including:

approximating motion of the scene element by using a 2D transform to transform an initial rendering of a first instance of the scene element to a new position.

42. The method of claim 37 further including:

clipping characteristic geometry of the scene element to an expanded viewing frustum that extends beyond screen boundaries of an output image;

tracking position of a clipped point of the characteristic geometry to determine whether the clipped point, when projected to the viewing frustum, is located within the screen boundaries;

determining whether to re-render the scene element when the clipped point corresponding to the scene element is within the screen boundaries.

43. A computer readable medium having computer-executable instructions for performing the steps recited in claim 37.

44. A method for measuring fidelity between instances of a scene element comprising:

computing a photometric distortion measure of the scene element; and

using the photometric distortion measure to determine distortion of an approximation of the scene element.

45. The method of claim 44 further including:

using the photometric distortion measure to determine whether to re-render the first scene element.

46. The method of claim 44 further including:

to construct frames in the animation sequence, compositing image layers, including at least one transformed layer, into an output image in output device coordinates.

47. The method of claim 44 wherein the step of computing the photometric distortion measure includes sampling lighting at characteristic points of a first and second instance of the scene element, and determining the change in color values at the characteristic points between the first and second instances.

48. The method of claim 44 wherein the step of computing the photometric distortion measure includes determining change in position of a light source relative to the scene element between a first and second instance of the scene element.

49. The method of claim 44 further including:

approximating motion of the scene element by using a 2D transform to transform an initial rendering of the scene element to a new position;

using the photometric distortion measure to determine distortion of approximating motion of the scene element with the 2D transform.

50. The method of claim 44 further including performing a color warp to approximate lighting change between first and second instances of the scene element.

51. The method of claim 44 including:

computing a color warp that approximates change in lighting of the scene element between first and second instances of the scene element by modifying color values in an image layer representing an initial rendering of the scene element;

measuring a difference between warped color values computed by performing the color warp on the initial rendering and color values of the second instance of the scene element to compute the photometric distortion measure; and

using the photometric distortion measure to determine whether to re-render the scene element.

52. The method of claim 51 further including:

approximating the change in lighting of the scene element by using the color warp to modify the color values in the initial rendering of the scene element to new color values.

53. A computer-readable medium having computer executable instructions for performing the steps recited in claim 44.

54. A method for measuring fidelity of a transformed image layer comprising:

computing a color warp that approximates change in lighting of a scene element from an initial frame to a current frame in an animation sequence by modifying color values in an image layer representing an initial rendering of the scene element; and

measuring a difference between warped color values computed by performing the color warp on the initial rendering and color values of the scene element for the current frame to compute photometric distortion.

55. The method of claim 54 further including:

approximating the change in lighting of the scene element by using the color warp to modify the color values in the initial rendering of the scene element at the initial frame to new color values for the current frame.

56. A graphics rendering system comprising:

a preprocessor for computing a sprite transform used to approximate motion of a scene element and for computing the fidelity of a transformed image layer by measuring sampling, lighting, or visibility distortion of the transformed image layer as a result of the sprite transform being applied to a rendering of the scene element;

a rendering sub-system for rendering a scene element to an image layer, and for transforming the image layer by applying the sprite transform to the image layer to approximate motion of the scene element in frames of animation.

57. The graphics rendering system of claim 56 wherein the rendering system is operable to render more than one scene element to separate image layers and is operable to composite the image layers into a final output image.

58. A method for measuring the fidelity of an image layer that is re-used in an animation sequence comprising:

computing a visibility or photometric distortion measure of a scene element of the image layer; and

using the visibility or the photometric distortion measure to determine the accuracy of re-using the image layer rather than re-rendering the scene element for the image layer.