Animation retargeting

• US 8,035,643 B2
• Filed: 03/19/2007
• Issued: 10/11/2011
• Est. Priority Date: 03/19/2007
• Status: Active Grant

• Current Assignee: Lucasfilm Entertainment Co Ltd (The Walt Disney Company)
• Original Assignee: Lucasfilm Entertainment Co Ltd (The Walt Disney Company)
• Inventors: Pighin, Frederic P., Phillips, Cary, Sullivan, Steve
• Primary Examiner(s): Nguyen; Kimbinh T
• Application Number:
  US11/688,066
• Publication Number:
  US 20080231640A1
• Time in Patent Office: 1,667 Days
• Field of Search: 345/427, 345/473, 345/619, 345/474, 345/475, 703/26, 704/266, 382/107, 382/209
• US Class Current: 345/473
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually

Abstract

Systems and methods are described, which create a mapping from a space of a source object (e.g., source facial expressions) to a space of a target object (e.g., target facial expressions). In certain implementations, the mapping is learned based a training set composed of corresponding shapes (e.g. facial expressions) in each space. The user can create the training set by selecting expressions from, for example, captured source performance data, and by sculpting corresponding target expressions. Additional target shapes (e.g., target facial expressions) can be interpolated and extrapolated from the shapes in the training set to generate corresponding shapes for potential source shapes (e.g., facial expressions).

Claims

1. A computer implemented method for animation mapping comprising:

creating a mapping, using a computer system, from a first configuration of a source shape to a first configuration of a target shape, wherein the mapping is created using a training set of previous associations of other configurations of the source shape with other configurations of the target shape, respectively, wherein the previous associations were created by user selection of source positions of the other configurations of the source shape for association with target positions of the other configurations of the target shape;

applying, using the computer system, the mapping to the target shape and displaying the first configuration of the target shape;

receiving input in the computer system generated by a user sculpting the first configuration of the target shape into a second configuration; and

modifying the training set, using the computer system and based on the input, the training set modified to generate a refined mapping from the first configuration of the source shape to the second configuration of the target shape.

2. The method of claim 1, wherein the mapping from the first configuration of the source shape comprises a first value expressing a first change in position from the first configuration of the source shape to the first configuration of the target shape.

3. The method of claim 2, wherein the mapping further comprises a second value expressing a second change in position from the first configuration of the source shape to the first configuration of the target shape.

4. The method of claim 2, wherein the display of the first configuration of the target shape comprises the target shape that corresponds to the source shape.

5. The method of claim 1, wherein the mapping from the first configuration of the source shape comprises a timing or parameterization for paths taken by source vertices associated with the source shape.

6. The method of claim 5, wherein the display of the first configuration of the target shape comprises a timing or parameterization for paths taken by target vertices that correspond to the paths take by the source vertices.

7. The method of claim 1, wherein modifying the training set further comprises receiving a modification based on execution of a computer-implemented procedure.

8. The method of claim 7, wherein the computer-implemented procedure comprises generating a composite configuration of the source shape used in the refined mapping.

9. The method of claim 8, wherein the composite configuration of the source shape is based on the source shape and a second configuration of the source shape.

10. The method of claim 1, wherein the mapping comprises applying an affine transformation before transforming the target shape.

11. The method of claim 10, further comprising estimating the affine transformation based on previous affine transformations between the source and target shapes.

12. The method of claim 11, wherein estimating the affine transformation comprises extrapolating or interpolating the previous affine transformations.

13. The method of claim 1, wherein the source and target shapes each comprise a mesh of vertices linked by edges that specifies geometry of the shape.

14. The method of claim 13, wherein the topology of the meshes of the source and target shapes differs.

15. The method of claim 1, wherein modifying the training set further comprises assigning a weighting factor, input by the user, to one or more of the associations in the training set.

16. The method of claim 15, wherein the weighting factor is assigned using an importance function.

17. The method of claim 1, wherein modifying the training set further comprises generating new or deleting previous associations between the source shape and the target shape.

18. The method of claim 1, further comprising iteratively modifying the training set based on user input in response to mappings generated by previous modifications to the training set.

19. The method of claim 1, further comprising generating the previous associations using a sculpting user interface (UI) that links geometry of the source shape to geometry of the target shape.

20. The method of claim 1, wherein the source shape comprises information defining an actor face selected from a group consisting of motion capture data, procedurally generated data, simulated data, and artist created data.

21. The method of claim 1, wherein the target shape comprises a computer-generated character'"'"'s face.

22. The method of claim 1, wherein creating the mapping from the source shape to the target shape comprises determining transforms for portions of the source shape and applying the transforms to corresponding portions of the target shape.

23. The method of claim 22, wherein the portions comprise vertices in a mesh.

24. The method of claim 23, wherein a transform for each vertex is determined relative to a local neighborhood of vertices a predetermined distance from the vertex.

25. The method of claim 24, wherein the distance is based on edge distance from the vertex.

26. The method of claim 23, wherein a transform for each vertex is determined relative to a local neighborhood of geometric shapes defined by vertices a predetermined distance from the vertex.

27. The method of claim 1, wherein the mapping is created for a point on a polygon surface not coinciding with any polygon vertex.

28. A method comprising:

displaying, using a computer system, an initial mapping from a first configuration of a source shape to a first configuration of a target shape, wherein the initial mapping is created using a training set of previous associations of other configurations of the source shape with other configurations of the target shape, respectively, created by user selection of source positions of the other configurations of the source shape for association with target positions of the other configurations of the target shape; and

iteratively (i) receiving, using the computer system, feedback generated by a user sculpting the first configuration of the target shape into a second configuration and (ii) adjusting the initial mapping based on the feedback.

29. A computer-implemented method for mapping a source shape to a target shape, the method comprising:

associating, using a computer system, a user-selected first configuration of a source shape with a user-selected first configuration of a target shape;

associating, using the computer system, a user-selected second configuration of the source shape with a user-selected second configuration of the target shape, the configurations being at least partially defined by a mesh comprising vertices; and

generating, using the computer system, based on the associations, a mapping between a third configuration of the source shape and a third configuration of the target shape, wherein the mapping comprises an affine transformation based on a transform of selected vertices of the source shape, the transform being relative to local vertices within a predetermined distance from the selected vertices.

30. The method of claim 29, wherein the affine transformation is derived from the associations between the source and target shapes for the first and second configurations.

31. The method of claim 30, wherein the derivation comprises interpolating or extrapolating the affine transformation using the associations.

32. The method of claim 29, wherein the transform of the selected vertices is at least partially expressed as a change in location of the selected vertices relative to a location of the selected vertices in the first configuration.

33. The method of claim 29, wherein the mapping between the third configuration of the source shape and the third configuration of the target shape is based on user input.

34. The method of claim 33, wherein the user input comprises values that modify a weighting factor that indicates a contribution of at least one of the associations on which the mapping is based.

35. The method of claim 33, wherein the user input comprises additional associations between corresponding positions of the source and target shapes.

36. The method of claim 29, further comprising generating a mesh of the source shape based on motion capture data associated with an actor.

37. The method of claim 29, further comprising generating a mesh of the target shape based on a computer-generated character.

38. The method of claim 29, further comprising displaying to a user the mapping between the third configuration of the source shape and the third configuration of the target shape.

39. The method of claim 29, wherein the third configuration is added to a training set used to generate additional mappings between the source and target shapes.

40. The method of claim 39, wherein the training set comprises the associations of the first and second configurations of the source shape with the corresponding first and second configurations of the target shape.

41. A method of generating animation, the method comprising:

generating, using a computer system, a mapping between a first configuration of a source mesh and a first configuration of a target mesh based on previous mappings, wherein the previous mappings were created by a user selecting source positions of other configurations of the source mesh for association with target positions of other configurations of the target mesh;

wherein generating the mapping comprises applying an affine transformation to transforms of one or more vertices of the source mesh, and wherein the transforms are defined relative to vertices a predetermined distance from the one or more vertices;

receiving input in the computer system generated by a user sculpting the first configuration of the target mesh into a second configuration; and

generating, using the computer system and based on the input, a refined mapping between the first configuration of the source mesh and the second configuration of the target mesh.

42. A system comprising:

a mapping function generator for generating mappings between a plurality of configurations of a source shape and a plurality of configurations of a target shape, respectively, wherein the mappings are based on a training set of previous associations between the source and target shapes created by a user selecting source positions of other configurations of the source shape for association with target positions of other configurations of the target shape; and

an interface for receiving modifications, based on user input received in response to an output of a first configuration of the target shape generated from a first configuration of the source shape by at least one of the mappings, the user input sculpting the first configuration of the target shape into a second configuration of the target shape, and modifying the training set for generating at least one refined mapping from the first configuration of the source shape to the second configuration of the target shape.