Real-Time Rendering of Realistic Rain

• US 20080068386A1
• Filed: 09/14/2006
• Published: 03/20/2008
• Est. Priority Date: 09/14/2006
• Status: Active Grant

• Current Assignee: Zhigu Holdings Limited
• Original Assignee: Microsoft Corporation
• Inventors: Yang, Xu, Wang, Jian, Tang, Xiaoou, Lin, Zhouchen, Wang, Lifeng, Fang, Tian, Yu, Xuan
• Granted Patent: US 7,692,647 B2
• Time in Patent Office: Days
• US Class Current: 345/473
• CPC Class Codes: G06T 13/60 of natural phenomena, e.g. rain, snow, water or plants
• CPC Class Codes: G06T 2210/56 Particle system, point based geometry or rendering

Abstract

Real-time rendering of realistic rain is described. In one aspect, image samples of real rain and associated information are automatically modeled in real-time to generate synthetic rain particles in view of respective scene radiances of target video content frames. The synthetic rain particles are rendered in real-time using pre-computed radiance transfer with uniform random distribution across respective frames of the target video content.

Claims

1. A computer-implemented method comprising:

automatically modeling, in real-time, images of real rain and associated information to generate synthetic rain particles; and

rendering, in real-time, the synthetic rain particles across respective frames of video content.

2. The method of claim 1, wherein the images and associated information is automatically modeled for blending with multiple frames of image content, each frame depicting a static scene, and wherein automatically modeling further comprises:

for each frame of the multiple frames and for each image of at least a subset of the images and associated information;

determining current velocity of a raindrop represented by the image based on a previous velocity of the raindrop;

computing shape of the raindrop based on the current velocity and camera exposure time; and

calculating position and orientation attributes of the raindrop based on 3-D coordinate space of the frame, current velocity, and a uniform random distribution criteria.

3. The method of claim 1, wherein the images and associated information are automatically modeled for blending with respective ones of multiple frames of image content, each frame depicting a static scene, and wherein automatically modeling further comprises:

for each frame of the multiple frames and for each sample raindrop represented by at least a subset of the images;

generating a rain mask for the sample raindrop, the rain mask identifying portion(s) of the frame that will be shaded by the sample raindrop, the rain mask being generated from alpha values defining opacity of the sample raindrop, the alpha values having been modeled to 3-D coordinate space of the frame; and

creating a shaded rain mask from the rain mask to specify color and intensity distribution of the sample raindrop.

4. The method of claim 3, wherein a graphics processing unit (CPU) generates the rain mask, and wherein per-pixel shader logic of the GPU creates the shaded rain mask.

5. The method of claim 3, wherein creating the shaded rain mask further comprises determining the color and intensity distribution based on pre-computed radiance transfer values and a transfer function of a sphere model of the sample raindrop, the sphere model having a refractive index of water.

6. The method of claim 5, wherein the transfer function is based on the reversibility of light.

7. The method of claim 3, wherein rendering the synthetic rain particles further comprises blending an alpha matte with the shaded rain mask and detected background color of the frame to generate a synthetic rain matted frame, the alpha matte having current velocity, shape, position, orientation, and color and intensity distribution attributes, the synthetic rain matted frame being a particular one of the respective frames of video content.

8. The method of claim 1, wherein the method further comprises randomly selecting the images of the real rain from a library of images of real rain and corresponding information.

9. The method of claim 1, wherein the method further comprises:

analyzing static image frames of video content depicting real rain to identify various sizes and shapes of real rain;

extracting attributes associated with identified real rain from respective ones of the static image frames to generate at least the images and the associated information; and

wherein the attributes comprise at least a rain stroke matte including alpha and radiance values, rain stroke direction, and detected background color, the alpha values defining opacity characteristics.

10. The method of claim 9, wherein analyzing the video content and extracting the information is implemented offline.

11. The method of claim 9, wherein the method further comprises:

rotating the rain stroke matte to a vertical orientation; and

removing noise from the rain stroke matte.

12. A computer-readable medium comprising computer-program instructions executable by a processor, the computer-program instructions, when executed by the processor for:

creating a set of rain particles from a set of randomly selected rain stroke samples and associated information, the rain stroke samples being randomly selected from a library of extracted rain stroke samples;

rendering synthetic rain across multiple frames of video content, the synthetic rain being based on respective ones of the rain particles, the rendering comprising;

for each frame of the frames, and for each particle of at least a subset of the rain particles;

computing, using a graphics processing unit (GPU), velocity, 3-D position, orientation, and shape of an alpha matte associated with the particle;

determining, using the GPU, a set of pixels to shade from the alpha matte;

calculating, using per-pixel shader operations of the GPU in view of the pixels to shade, color and intensity distribution of pixels using pre-computed radiance transfer; and

blending, using the CPU, the alpha matte and a background color associated with the frame with a shaded rain mask based on the color and the intensity distribution of the pixels to generated a synthetic rain matted frame for presentation to a user.

13. The computer-readable medium of claim 12, wherein the library of extracted rain stroke samples is generated offline, and wherein at least the rendering is performed in real-time.

14. The computer-readable medium of claim 12, wherein the 3-D position is uniformly randomly distributed in space delineated by the frame.

15. The computer-readable medium of claim 12, wherein the shape comprises a length and a diameter, the length being interpolated based on a current velocity of the particle and an exposure time of a virtual camera, the diameter corresponding to width of the alpha matte after being mapped to scene coordinates of the frame.

16. The computer-readable medium of claim 12, wherein operations associated with the pre-computed radiance transfer are based on an outgoing light intensity distribution transfer function.

17. The computer-readable medium of claim 12, wherein computer-program instructions further comprise instructions for providing volumetric appearance of synthetic rain by:

identifying rough geometry of a scene associated with the frame; and

automatically clipping the alpha matte at surface of the scene prior to the rendering, the surface being defined using a mesh of a synthetic scene or a depth map.

18. The computer-readable medium of claim 12, wherein computer-program instructions further comprise instructions for:

analyzing static image frames of video content depicting real rain to identify various sizes and shapes of the real rain;

extracting rain stroke mattes and information associated with identified real rain from the identifies various sizes and shapes of the real rain; and

wherein the rain stroke mattes and the information represent the library of extracted rain stroke samples

19. A computer comprising:

a processor; and

a memory coupled to the processor, the memory comprising computer-program instructions executable by the processor for;

randomly selecting real rain stroke samples from video content;

modeling, using a graphics accelerator, the real rain stroke samples as synthetic rain particles in view of respective scene radiances; and

rendering the synthetic rain across scenes with uniform random distribution, controllable velocity, and color determined via pre-computed radiance transfer.

20. The computing device of claim 19, wherein modeling the real rain stroke samples as synthetic rain particles further comprises:

converting rain stroke mattes into 3-D coordinate space of respective image frames of video content;

calculating velocity, position, and shape attributes of alpha mattes associates with the rain stroke mattes based on previous values, physical laws, and rain stroke distribution criteria;

identifying respective rain masks from the alpha masks;

using pre-computed radiance transfer to identify pixel color and intensity distributions from the rain masks; and

wherein rendering further comprises alpha blending the alpha masks with information associated with identified pixel color and intensity distributions and detected frame background color to create synthetic rain matted scenes for presentation to a user.