Animation Retargeting

• US 20080231640A1
• Filed: 03/19/2007
• Published: 09/25/2008
• Est. Priority Date: 03/19/2007
• Status: Active Grant

• Current Assignee: Lucasfilm Entertainment Co Ltd (The Walt Disney Company)
• Original Assignee: Lucasfilm Entertainment Co Ltd (The Walt Disney Company)
• Inventors: Pighin, Frederic P., Phillips, Cary, Sullivan, Steve
• Granted Patent: US 8,035,643 B2
• Time in Patent Office: Days
• US Class Current: 345/619
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually

Abstract

Systems and methods are described, which create a mapping from a space of a source object (e.g., source facial expressions) to a space of a target object (e.g., target facial expressions). In certain implementations, the mapping is learned based a training set composed of corresponding shapes (e.g. facial expressions) in each space. The user can create the training set by selecting expressions from, for example, captured source performance data, and by sculpting corresponding target expressions. Additional target shapes (e.g., target facial expressions) can be interpolated and extrapolated from the shapes in the training set to generate corresponding shapes for potential source shapes (e.g., facial expressions).

Claims

1. A method for animation mapping comprising:

mapping a transform of a source shape to a target shape, wherein the mapping is based on a training set of previous associations between the source and target shapes;

applying the mapped transform to the target shape for output of an initial mapping; and

modifying the training set to generate a refined mapping of the transform applied to the target shape.

2. The method of claim 1, wherein the transform of the source shape comprises a first value expressing a first change in position from a first shape to a second shape.

3. The method of claim 2, wherein the transform further comprises a second value expressing a second change in position from a third shape to the second shape.

4. The method of claim 2, wherein the output of the initial mapping comprises the target shape that corresponds to the source shape.

5. The method of claim 1, wherein the transform of the source shape comprises a timing or parameterization for paths taken by source vertices associated with the source shape.

6. The method of claim 5, wherein the output of the initial mapping comprises a timing or parameterization for paths taken by target vertices that correspond to the paths take by the source vertices.

7. The method of claim 1, wherein modifying the training set comprises receiving a modification based on user input received in response to the output of the initial mapping.

8. The method of claim 1, wherein modifying the training set comprises receiving a modification based on execution of a computer-implemented procedure.

9. The method of claim 8, wherein the computer-implemented procedure comprises generating a composite source shape used in the refined mapping.

10. The method of claim 9, wherein the composite source shape is based on the source shape and a second shape that corresponds to the source shape.

11. The method of claim 1, wherein the mapping of the transform comprises an affine transformation of the transformation before applying the mapped transform to the target shape.

12. The method of claim 11, further comprising estimating the affine transformation based on previous affine transformations of transforms mapped between the source and target shapes.

13. The method of claim 12, wherein estimating the affine transformation comprises extrapolating or interpolating the previous affine transformations.

14. The method of claim 1, wherein the source and target shapes each comprise a mesh of vertices linked by edges that specifies geometry of the shape.

15. The method of claim 14, wherein the topology of the meshes of the source and target shapes differs.

16. The method of claim 1, wherein modifying the training set comprises assigning a weighting factor, input by the user, to one or more of the associations in the training set.

17. The method of claim 16, wherein the weighting factor is assigned using an importance function.

18. The method of claim 1, wherein modifying the training set comprises generating new or deleting previous associations between the source shape and the target shape.

19. The method of claim 1, wherein the output comprises a display of the transform mapped to the target shape.

20. The method of claim 1, further comprising iteratively modifying the training set based on user input in response to mappings generated by previous modifications to the training set.

21. The method of claim 1, further comprising generating the previous associations using a sculpting user interface (UI) that links geometry of a source shape to geometry of a target object.

22. The method of claim 1, wherein the source shape comprises information defining an actor face selected from a group consisting of motion capture data, procedurally generated data, simulated data, and artist created data.

23. The method of claim 1, wherein the target shape comprises a computer-generated character'"'"'s face.

24. The method of claim 1, wherein mapping the transform of the source shape to a target shape comprises determining transforms for portions of the source shape and applying the transforms to corresponding portions of the target shape.

25. The method of claim 24, wherein the portions comprise vertices in a mesh.

26. The method of claim 25, wherein a transform for each vertex is determined relative to a local neighborhood of vertices a predetermined distance from the vertex.

27. The method of claim 26, wherein the distance is based on edge distance from the vertex.

28. The method of claim 25, wherein a transform for each vertex is determined relative to a local neighborhood of geometric shapes defined by vertices a predetermined distance from the vertex.

29. A method comprising:

outputting an initial mapping of a transform of a source shape to a target shape; and

iteratively receiving feedback from a user and adjusting the mapping of the transform based on the feedback.

30. A computer-implemented method for mapping a transform of a source shape to a target object, the method comprising:

associating first and second positions of a source object with corresponding first and second positions of a target object, wherein the positions are at least partially defined by a mesh comprising vertices; and

generating, based on the associations, a mapping between a third position of the source object and a third position of the target object, wherein the mapping comprises an affine transformation based on a transform of selected vertices of the source object relative to local vertices within a predetermined distance from vertices selected for transformation.

31. The method of claim 30, wherein the affine transformation is derived from the associations between the source and target objects for the first and second positions.

32. The method of claim 31, wherein the derivation comprises interpolating or extrapolating the affine transformation using the associations.

33. The method of claim 30, wherein the transform of selected vertices is at least partially expressed as a change in location of the selected vertices relative to a location of the selected vertices in the first position.

34. The method of claim 30, wherein the mapping between the third position of the source object and the third position of the target object is based on user input.

35. The method of claim 34, wherein the user input comprises values that modify a weighting factor that indicates a contribution of at least one of the associations on which the mapping is based.

36. The method of claim 34, wherein the user input comprises additional associations between corresponding positions of the source and target objects.

37. The method of claim 30, further comprising generating the mesh of the source object based on motion capture data associated with an actor.

38. The method of claim 30, further comprising generating the mesh of the target object based on a computer-generated character.

39. The method of claim 30, further comprising displaying to a user the mapping between the third position of the source object and the third position of the target object.

40. The method of claim 30, wherein the third position is added to a training set used to generate additional mappings between the source and target object.

41. The method of claim 40, wherein the training set comprises the associations of the first and second positions of a source object with the corresponding first and second positions of the target object.

42. A method of generating animation, the method comprising:

generating a mapping between a source mesh and a target mesh based on previous mappings, wherein generating the mapping comprises applying an affine transformation to transforms of one or more vertices of the source mesh, wherein the transforms are relative to a neighborhood of vertices a predetermined distance from the one or more vertices being transformed.

43. A method of generating animation, the method comprising:

selecting a point on a target object to reflect a transform of a corresponding point on a source object;

identifying a first neighborhood of geometric shapes that are a predetermined distance from the point on the target object and associating the first neighborhood with a corresponding second neighborhood of geometric shapes on the source object; and

determining an affine mapping for a transform of the point on the target object relative to the first neighborhood based on the transform of the corresponding point on the source object relative to the second neighborhood.

44. The method of claim 43, wherein the affine mapping is further based on previous affine mappings between points on the target and source objects.

45. The method of claim 44, wherein determining the affine mapping comprises estimating the affine mapping by interpolating or extrapolating the previous affine mappings.

46. A system comprising:

a mapping function generator for mapping a transform of a source shape to a target shape, wherein the mapping is based on a training set of previous associations between the source and target shapes; and

an interface for receiving modifications, based on user input received in response to an output of the mapping, to the training set for generating a refined mapping of the transform applied to the target shape.

47. A method for animation mapping comprising:

outputting an initial mapping between a first transform of a source shape and a second transform of a target shape, wherein the initial mapping is based on a training set of previous associations between the source and target shapes;

modifying the training set to generate a refined mapping between the first and second transforms; and

outputting the refined mapping.