System and method for invariant-based normal estimation

• US 8,611,674 B1
• Filed: 01/18/2010
• Issued: 12/17/2013
• Est. Priority Date: 01/18/2010
• Status: Active Grant

• Current Assignee: ETH Zurich, Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Inventors: Bickel, Bernd, Gross, Markus, Sumner, Robert, Beeler, Thabo Dominik
• Primary Examiner(s): Le, Vu
• Assistant Examiner(s): WOLDEMARIAM, AKLILU K
• Application Number:
  US12/689,172
• Time in Patent Office: 1,429 Days
• Field of Search: 382/103, 382/210, 382/218, 382/300
• US Class Current: 382/218
• CPC Class Codes: G06T 15/50 Lighting effects
• CPC Class Codes: G06V 10/60 relating to illumination properties, e.g. using a reflectance or lighting model

Abstract

A computer-implemented method for invariant-based normal estimation. The method includes calculating a set of measured invariants for a point associated with a surface of an object, where the set of measured invariants is based on pixel information that includes lighting information, calculating one or more sets of estimated invariants for the point associated with the surface of the object, where each set of estimated invariants is based on a known lighting environment for the object and a different normal for the point associated with the surface of the object, and determining a first normal for the point associated with the surface of the object that results in the set of measured invariants corresponding to a first set of estimated invariants.

Claims

1. A computer-implemented method for invariant-based normal estimation, the method comprising:

calculating a set of measured invariants for a point associated with a surface of an object, wherein the set of measured invariants is based on color gradients from at least two color channels, wherein the two color channels are captured from two distinct viewpoints, wherein the color gradients include at least a horizontal gradient and a linear gradient in one of the color channels for normalization, and wherein the other color channel includes quadratic gradients which enclose angles of ±

45 degrees with a horizontal axis;

calculating one or more sets of estimated invariants for the point associated with the surface of the object, wherein each set of estimated invariants is based on a known lighting environment for the object and a different normal for the point associated with the surface of the object; and

determining a first normal for the point associated with the surface of the object that results in the set of measured invariants corresponding to a first set of estimated invariants.

2. The method of claim 1, wherein determining the first normal comprises determining that the first set of estimated invariants equals the set of measured invariants.

3. The method of claim 1, wherein determining the first normal comprises determining that the first set of estimated invariants has a minimal difference to the set of measured invariants relative to the other sets of estimated invariants.

4. The method of claim 1, wherein the pixel information comprises color intensity values for one or more different color channels.

5. The method of claim 4, wherein the color channels comprise red, green, and blue color channels.

6. The method of claim 1, wherein, within the known lighting environment, a light direction is known for a plurality of points on a light stage surrounding the object.

7. The method of claim 6, wherein the points included in the plurality of points are randomly sampled points associated with the light stage.

8. The method of claim 6, wherein the light stage comprises a cubical light tent or a spherical light tent.

9. The method of claim 6, wherein the known lighting environment comprises a spatially varying lighting gradient applied to the light stage.

10. The method of claim 1, further comprising:

calculating a set of measured invariants for each of a plurality of other points associated with the surface of the object, wherein the set of measured invariants is based on color gradients from two or more color channels, wherein the two or more color channels are captured from two or more viewpoints;

for each of the other points, calculating one or more sets of estimated invariants; and

for each of the other points, determining a normal that results in a set of measured invariants associated with the other point to correspond to a particular set of estimated invariants, wherein a three-dimensional geometry is generated based on the first normal and a normal associated with each of the other points.

11. The method of claim 1, wherein calculating the set of measured invariants comprises calculating a difference between a first color intensity value associated with a first color channel and a first sensing device and a second color intensity value associated with the first color channel and a second sensing device.

12. The method of claim 11, wherein the sensing device comprises a camera or a light sensing device.

13. The method of claim 1, wherein calculating each set of estimated invariants comprises calculating a difference between a first value associated with an incident light angle of a first sensing device relative to a normal and a second value associated with an incident light angle of a second sensing device relative to the normal.

14. The method of claim 1, wherein the object comprises a human head, and the surface comprises a human face.

15. The method of claim 1, further comprising generating a three-dimensional geometry for the object based on the first normal.

24. The computer-readable storage medium of claim 6, wherein the known lighting environment comprises a spatially varying lighting gradient applied to the light stage.

16. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause a computer system to perform invariant-based normal estimation, by performing the steps of:

calculating a set of measured invariants for a point associated with a surface of an object, wherein the set of measured invariants is based on color gradients from at least two color channels, wherein the two color channels are captured from two distinct viewpoints, wherein the color gradients include at least a horizontal gradient and a linear gradient in one of the color channels for normalization, and wherein the other color channel includes quadratic gradients which enclose angles of ±

45 degrees with a horizontal axis;

calculating one or more sets of estimated invariants for the point associated with the surface of the object, wherein each set of estimated invariants is based on a known lighting environment for the object and a different normal for the point associated with the surface of the object; and

determining a first normal for the point associated with the surface of the object that results in the set of measured invariants corresponding to a first set of estimated invariants.

17. The computer-readable storage medium of claim 16, wherein determining the first normal comprises determining that the first set of estimated invariants equals the set of measured invariants.

18. The computer-readable storage medium of claim 16, wherein determining the first normal comprises determining that the first set of estimated invariants has a minimal difference to the set of measured invariants relative to the other sets of estimated invariants.

19. The computer-readable storage medium of claim 16, wherein the pixel information comprises color intensity values for one or more different color channels.

20. The computer-readable storage medium of claim 19, wherein the color channels comprise red, green, and blue color channels.

21. The computer-readable storage medium of claim 16, wherein, within the known lighting environment, a light direction is known for a plurality of points on a light stage surrounding the object.

22. The computer-readable storage medium of claim 21, wherein the points included in the plurality of points are randomly sampled points associated with the light stage.

23. The computer-readable storage medium of claim 21, wherein the light stage comprises a cubical light tent or a spherical light tent.

25. The computer-readable storage medium of claim 16, further comprising:

calculating a set of measured invariants for each of a plurality of other points associated with the surface of the object, wherein the set of measured invariants is based on color gradients from two or more color channels, wherein the two or more color channels are captured from two or more viewpoints;

for each of the other points, calculating one or more sets of estimated invariants; and

for each of the other points, determining a normal that results in a set of measured invariants associated with the other point to correspond to a particular set of estimated invariants, wherein a three-dimensional geometry is generated based on the first normal and a normal associated with each of the other points.

26. The computer-readable storage medium of claim 16, wherein calculating the set of measured invariants comprises calculating a difference between a first color intensity value associated with a first color channel and a first sensing device and a second color intensity value associated with the first color channel and a second sensing device.

27. The computer-readable storage medium of claim 26, wherein the sensing device comprises a camera or a light sensing device.

28. The computer-readable storage medium of claim 16, wherein calculating each set of estimated invariants comprises calculating a difference between a first value associated with an incident light angle of a first sensing device relative to a normal and a second value associated with an incident light angle of a second sensing device relative to the normal.

29. The computer-readable storage medium of claim 16, wherein the object comprises a human head, and the surface comprises a human face.

30. The computer-readable storage medium of claim 16, further comprising generating a three-dimensional geometry for the object based on the first normal.

32. The system of claim 25, further comprising a memory storing instructions that, when executed by the processor, configure the processor to:

calculate the set of measured invariants;

calculate the one or more sets of estimated invariants; and

determine the first normal.

31. A system for invariant-based normal estimation, the system comprising:

a processor configured to;

calculate a set of measured invariants for a point associated with a surface of an object, wherein the set of measured invariants is based on color gradients from at least two color channels, wherein the two color channels are captured from two distinct viewpoints, wherein the color gradients include at least a horizontal gradient and a linear gradient in one of the color channels for normalization, and wherein the other color channel includes quadratic gradients which enclose angles of ±

45 degrees with a horizontal axis,

calculate one or more sets of estimated invariants for the point associated with the surface of the object, wherein each set of estimated invariants is based on a known lighting environment for the object and a different normal for the point associated with the surface of the object, and

determine a first normal for the point associated with the surface of the object that results in the set of measured invariants corresponding to a first set of estimated invariants.