ANIMATING OBJECTS USING THE HUMAN BODY

• US 20140035901A1
• Filed: 07/31/2012
• Published: 02/06/2014
• Est. Priority Date: 07/31/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Izadi, Shahram, Fitzgibbon, Andrew William, Chen, Jiawen
• Granted Patent: US 9,536,338 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: A63F 13/52 involving aspects of the displayed game scene
• CPC Class Codes: A63F 13/525 Changing parameters of virtual cameras
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 13/00 Animation
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 13/80 2D [Two Dimensional] animation, e.g. using sprites

Abstract

Methods of animating objects using the human body are described. In an embodiment, a deformation graph is generated from a mesh which describes the object. Tracked skeleton data is received which is generated from sensor data and the tracked skeleton is then embedded in the graph. Subsequent motion which is captured by the sensor result in motion of the tracked skeleton and this motion is used to define transformations on the deformation graph. The transformations are then applied to the mesh to generate an animation of the object which corresponds to the captured motion. In various examples, the mesh is generated by scanning an object and the deformation graph is generated using orientation-aware sampling such that nodes can be placed close together within the deformation graph where there are sharp corners or other features with high curvature in the object.

Claims

1. A method of animating an object comprising:

generating, by a processor, a deformation graph automatically from an input mesh defining the object;

receiving body tracking data defining positions of one or more points on a body;

attaching points on a skeleton to points on the deformation graph using the body tracking data;

transforming the deformation graph in real-time based on motion of the tracked skeleton by computing a plurality of transformations; and

dynamically applying the plurality of transformations to the mesh to render a corresponding animation of the object.

2. A method according to claim 1, further comprising creating the input mesh by:

generating a 3D volumetric reconstruction of a scene scanned by a user with a depth camera;

segmenting an object from the 3D volumetric reconstruction of the scene; and

extracting a geometric isosurface from the segmented portion of the 3D volumetric reconstruction.

3. A method according to claim 1, wherein the deformation graph is generated automatically from the input mesh using orientation-aware sampling.

4. A method according to claim 1, wherein generating a deformation graph automatically from an input mesh defining the object comprises:

defining a plurality of node positions using sampling;

creating a nearest neighbor graph by connecting each node to a number of nearest neighbor nodes; and

adding additional edges to the nearest neighbor graph based on analysis of connected components in the nearest neighbor graph.

5. A method according to claim 4, wherein the sampling and the creating of the nearest neighbor graph both use an orientation-aware distance defined such that distances between a pair of samples increases faster when normals associated with each of the samples deviate from alignment.

6. A method according to claim 4, wherein adding additional edges to the nearest neighbor graph based on analysis of connected components in the nearest neighbor graph comprises:

identifying any connected components in the nearest neighbor graph;

forming a weighted graph comprising nodes corresponding to each identified connected component and weights defined based on a minimum distance between the connected components in the nearest neighbor graph;

computing a minimum spanning tree on the weighted graph; and

adding an additional edge to the nearest neighbor graph where there is an edge between the connected components in the spanning tree.

7. A method according to claim 4, wherein transforming the deformation graph in real-time based on motion of the tracked skeleton by computing a plurality of transformations comprises treating the additional edges as links with greater rigidity than other connections between nodes in the nearest neighbor graph.

8. A method according to claim 1, wherein attaching points on a skeleton to points on the deformation graph comprises:

providing visual feedback to a user showing alignment of the tracked body and the input mesh;

dynamically generating a proposed mapping between points on a skeleton and a number of nearest nodes in the input mesh; and

in response to a verbal command received from the user, storing the proposed mapping,

and wherein the stored mapping is used in dynamically applying the plurality of transformations to the mesh.

9. A method according to claim 1, wherein the object is non-humanoid.

10. A method according to claim 1, wherein the method is implemented interactively at run-time.

11. A method as claimed in claim 1 at least partially carried out using hardware logic.

12. A system comprising:

an input for receiving an input mesh defining an object;

a pre-processing module arranged to generate a deformation graph automatically from the input mesh; and

a warp module arranged to receive the deformation graph, the input mesh and body tracking data defining positions of one or more points on a body, to attach points on a skeleton to points on the deformation graph using the body tracking data, to compute a series of transformations on the deformation graph in real-time based on motion of the tracked skeleton, and to apply the series of translations to the input mesh to generate an animation of the object.

13. A system according to claim 12, further comprising:

a skeleton tracker module arranged receive data from a sensor and to generate the body tracking data from the received data.

14. A system according to claim 13, wherein the sensor is a depth camera.

15. A system according to claim 12, wherein the pre-processing module is arranged to generate the deformation graph using orientation-aware sampling.

16. A system according to claim 15, wherein the pre-processing module is further arranged to generate the deformation graph by defining a nearest neighbor graph from the input mesh using orientation-aware sampling and then adding additional edges to the nearest neighbor graph based on analysis of connected components in the nearest neighbor graph.

17. A system according to claim 16, wherein the warp module is arranged to treat the additional edges as more rigid than other edges in the deformation graph when computing the series of transformations on the deformation graph.

18. A system according to claim 12, further comprising:

a processor; and

a memory arranged to store the pre-processing module and the warp module,

and wherein the pre-processing module and warp module comprise processor-executable instructions.

19. A system as claimed in claim 12, wherein at least one of the pre-processing module and the warp module being at least partially implemented using hardware logic selected from any one or more of:

a field-programmable gate array, a program-specific integrated circuit, a program-specific standard product, a system-on-a-chip, a complex programmable logic device.

20. One or more tangible device-readable media with device-executable instructions that, when executed by a computing system, direct the computing system to:

generate a deformation graph automatically from an input mesh defining an inanimate object using orientation-aware sampling;

receive body tracking data relating to a user, the data defining positions of one or more points on the user'"'"'s body;

attach points on a skeleton to points on the deformation graph using the body tracking data in response to a user command;

transform the deformation graph interactively in real-time based on motion of the tracked skeleton by computing a plurality of transformations, the motion of the tracked skeleton corresponding to motion of the user; and

dynamically apply the plurality of transformations to the mesh to render a corresponding animation of the object.