System and method for image-based surface detail transfer

• US 7,200,281 B2
• Filed: 04/18/2005
• Issued: 04/03/2007
• Est. Priority Date: 04/18/2002
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Zhengyou, Liu, Zicheng, Shan, Ying
• Primary Examiner(s): PATEL, KANJIBHAI B
• Application Number:
  US11/108,545
• Publication Number:
  US 20050180657A1
• Time in Patent Office: 715 Days
• Field of Search: 382/108, 382/118, 382/162, 382/264, 382274-276, 382/287, 382/294, 382/284, 382/296, 345/473, 345/582, 345/426, 345/581, 345/584, 345/589, 345/611, 345/648, 345/428, 345/651
• US Class Current: 382/276
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 15/04 Texture mapping

Abstract

A system and method, called Image-Based Surface Detail Transfer, to transfer geometric details from one surface of an object in an image to another with simple 2D image operations. The basic observation is that, without knowing its 3D geometry, geometric details (local deformations) can be extracted from a single image of an object in a way independent of its surface reflectance, and furthermore, these geometric details can be transferred to modify the appearance of other objects directly in images. Examples are shown including surface detail transfer between real objects, as well as between real and synthesized objects.

Claims

1. A computer-implemented process for transferring geometrical details from a first object in one image to a second object in a second image, comprising:

an inputting step for inputting a first image depicting an object whose surface details are to be transferred and a second image depicting an object of similar shape as depicted in the first image;

an aligning step for aligning the objects of similar shape in first and second images;

a computing step for computing a smoothed version of the first and second images;

for each pixel in the first aligned image, a ratio computing step for computing the ratio of the original intensity to the smoothed intensity;

an image creating step for creating a new image by multiplying the ratio of the original intensity to the smoothed intensity of the first image by the smoothed intensity for each corresponding point in the second image.

2. The computer implemented process of claim 1 wherein the aligning step comprises:

a rotation step for performing a rotation of the images to rotationally align the object in the second image with the object of similar shape in the first image;

a translation step for performing a translation of the images to align the object in second image with the object of similar shape in the first image; and

a scaling step for performing a scaling of the images to match the size of the object in the second image with the object of similar shape in first image.

3. The computer-implemented process of claim 1 wherein the aligning step comprises:

a mapping step for establishing a standard map having nodes at the point of important features;

a first overlaying step for overlaying said map on the first image, so its nodes match the important features;

a second overlaying step for overlaying said map on the second image, and causing it to match the important features thereof; and

a warping step for using coordinates of said map on said second image to warp said second image so as to match the first image, thereby allowing the corresponding pixel locations in the first and second images as well as their respective intensities to be identified.

4. The computer-implemented process of claim 1 wherein the smoothing step comprises a filtering step for performing a Gaussian filtering procedure to each pixel in the first and second images to compute a new value of intensity for each pixel.

5. The computer-implemented process of claim 4 wherein the standard deviation of the Gaussian filter determines the level of the surface details to be transferred from the first image to the newly created image.

6. The computer-implemented process of claim 5, wherein the standard deviation of the Gaussian filter is set to be small and only fine details are transferred.

7. The computer-implemented process of claim 5, wherein the standard deviation of the Gaussian filter is set to be larger and larger scale details are transferred.

8. The computer-implemented process of claim 1 wherein the first and second images do not exhibit similar lighting conditions and wherein a relighting step is applied to compensate in the lighting conditions between the first and second image.

9. A computer-implemented process for simulating the appearance of a face at a different age, comprising:

an inputting step for inputting a first image depicting a face whose facial characteristics are to be transferred and a second image depicting a face that is to receive the facial characteristics of the face in the first face;

an aligning step for aligning the faces in first and second images;

a smoothing step for computing a smoothed version of the faces in the first and second images;

for each pixel in the face of the first aligned image, a ratio computing step for computing the ratio of the original intensity to the smoothed intensity; and

an image creating step for creating a new image by multiplying the ratio of the original intensity to the smoothed intensity of the face in the first image by the smoothed intensity for each corresponding point in the second image.

10. The computer-implemented process of claim 9 wherein the aligning step comprises:

a mapping step for establishing a standard map of markers at the point of important facial features;

a first overlaying step for overlaying said standard map on the face in the first image, so its markers match the important facial features;

a second overlaying step for overlaying said standard map on the face in the second image, and causing it to match the important features thereof; and

a warping step for using coordinates of said map on said face in the second image to warp said second image so as to match the first image, thereby allowing the corresponding pixel locations in the first and second images as well as their respective intensities to be identified.

11. The computer-implemented process of claim 9 wherein the facial characteristics of the face in the first image depict an older face than the facial characteristics of the face in the second image, and wherein the newly created image depicts an older face than the face depicted in the second image.

12. The computer-implemented process of claim 9 wherein the facial characteristics of the face in the first image depict a younger face than the facial characteristics of the face in the second image, and wherein the newly created image depicts a younger face than the face depicted in the second image.

13. The computer-implemented process of claim 9 wherein the smoothing step comprises steps for:

for a given area of pixels of predefined size,

averaging the pixel intensity by averaging the intensity of the surrounding pixels.

14. The computer-implemented process of claim 9 wherein comprises a filtering step for applying a Gaussian filter to each pixel in the first and second images.

15. The computer-implemented process of claim 9 wherein certain areas of the first and second images are masked so that they are not changed in the created new image.

16. The computer-implemented process of claim 9 wherein the first image depicting a face whose facial characteristics are to be transferred is a face of different age, race or gender from the face in the second image.

17. A computer-implemented process for transferring the surface details of an object in one image to an object of similar shape in another image, comprising:

a recording step for recording a first image depicting an object whose surface details are to be transferred and a second image depicting an object of similar shape as depicted in the first image;

a smoothing step for smoothing first and second images;

for each pixel in the first image, a computing step for computing the ratio of the original intensity to the smoothed intensity; and

an image creating step for creating a new image by multiplying the ratio of the original intensity to the smoothed intensity of the first image by the smoothed intensity for each corresponding point in the second image.

18. The computer-implemented process of claim 17 wherein the object in the first image is a real-world object and the object in the second image is a synthetic object.

19. The computer-implemented process of claim 17 further comprising a warping step for warping the images prior to smoothing them so that the objects of similar shape a located in the same location in the first and second images.