Animation generation systems and methods

• US 8,462,198 B2
• Filed: 11/08/2010
• Issued: 06/11/2013
• Est. Priority Date: 12/29/2009
• Status: Active Grant

• Current Assignee: Industrial Technology Research Institute
• Original Assignee: Industrial Technology Research Institute
• Inventors: Hsiao, Fu-Jen, Lin, Tzung-Han, Teng, Chih-Jen
• Primary Examiner(s): Czekaj, David
• Assistant Examiner(s): Owens, Tsion B
• Application Number:
  US12/941,098
• Publication Number:
  US 20110157306A1
• Time in Patent Office: 946 Days
• Field of Search: 348/43, 345/419, 345/474
• US Class Current: 348/43
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

The animation generation system includes: an avatar generation module for generating an avatar in a virtual space, wherein the avatar has a set of skeletons and a skin, and the movable nodes of the set of skeletons are manipulated so that a motion of the skin is induced; and an avatar manipulation module for manipulating the movable nodes, including: a position mark which is moved to at least one first real position in a real space; at least one control mark which is moved to at least one second real position in the real space; and a video capturing unit for capturing the images of the real space; an arithmetic unit for identifying the first real position and the second real position from the images of the real space, and converting the first real position into a first virtual position, and the second real position into a second virtual position.

Claims

1. An animation generation system, comprising:

an avatar generation module for generating an avatar in a virtual space, wherein the avatar has a set of skeletons and a skin attached to the set of skeletons, and the set of skeletons has a plurality of movable nodes, and the movable nodes are manipulated so that a motion of the skin is induced; and

an avatar manipulation module for manipulating the movable nodes of the avatar, comprising;

a position mark which is moved by users to at least one first real position in a real space;

at least one control mark which is moved by the users to at least one second real position in the real space;

a video capturing unit for capturing the images of the real space;

an arithmetic unit, coupled to the video capturing unit, for identifying the first real position and the second real position from the images of the real space, and converting the first real position into a first virtual position where the avatar is in the virtual space, and the second real position into a second virtual position where one of the movable nodes of the avatar is in the virtual space,

wherein the relative motions between the second virtual position and the first virtual position make the avatar perform a series of successive motions in the virtual space, and

one of the images captured by the video capturing unit are drawn as a background, while one of the images of a designated object corresponding to the position mark is drawn onto the background, according to the first virtual position, to generate an Augmented Reality (AR) animation,

wherein the avatar generation module further comprises;

a readout unit for reading out skin information in respect of the skin, and skeleton information in respect of a set of template skeletons, wherein the skin information is composed of data of a plurality of mesh vertices, and the skeleton information comprises geometric data of each bone of the set of template skeletons and the linkage relationship between the bones of the set of template skeletons;

an analyzing unit for analyzing the skin information to obtain a plurality of appearance features and a plurality of trunk features of the skin;

a rough fitting unit for adjusting the size of the set of template skeletons according to the appearance features to generate the set of skeletons of the avatar, and fitting the set of skeletons to the appearance features;

a precise fitting unit for fitting the set of skeletons to the trunk features of the skin;

an envelope range calculating unit for calculating an envelope range of a bone of the set of skeletons according to a plurality of mesh vertices in proximity to the bone;

a mesh vertices weight calculating unit for calculating the weight of a mesh vertex of the skin information relative to a bone according the envelope range, wherein the weight of the mesh vertex indicates how the mesh vertex is affected by the movement of the bone when the bone is moving; and

an output unit for outputting the avatar to the arithmetic unit, wherein the outputted avatar comprises data in relation to the skin, the set of skeletons and the relationships therebetween.

2. The animation generation system as claimed in claim 1, wherein the position mark and the control mark are barcodes or other objects which visible appearances have identifiable shapes, patterns, dimensions or colors.

3. The animation generation system as claimed in claim 1, wherein the video capturing unit is composed of at least one camcorder.

4. The animation generation system as claimed in claim 1, wherein the AR animation drawn by the arithmetic unit is stored into a storage unit.

5. The animation generation system as claimed in claim 1, wherein the designated object is a successive motion.

6. The animation generation system as claimed in claim 1, wherein the designated object is a virtual three dimension (3D) model.

7. The animation generation system as claimed in claim 1, wherein the envelope range is two-layer-barrel-shaped, and each of the two ends of the envelope range is a concentric hemisphere.

8. The animation generation system as claimed in claim 1, wherein rules for the mesh vertices weight calculating unit to calculate the mesh vertices weight comprises:

rule (1);

when a mesh vertex is inside the inner layer of the envelope of a bone, the weight of the mesh vertex in relation to the bone is 1.0, which indicates that the mesh vertex is totally affected by the movement of the bone when the bone is moving;

rule (2);

when a mesh vertex is outside of the outer layer of the envelope of a bone, the weight of the mesh vertex in relation to the bone is 0.0, which indicates that the mesh vertex is totally unaffected by the movement of the bone when the bone is moving;

rule (3) when a mesh vertex is between the inner and outer layer of the envelope range of a bone, the weight of the mesh vertex in relation to the bone decreases from 1.0 to 0.0, which indicates that the mesh vertex is partially affected by the movement of the bone according to the distance between the mesh vertex and the bone;

rule (4) when a mesh vertex does not belong to any envelope range of any bone in accordance with the former three rules, the weight of the mesh vertex in relation to a bone closest to the mesh vertex is 1.0, which indicates that the mesh vertex is totally affected by the movement of the bone closest to the mesh vertex; and

rule (5) when all the weights of the mesh vertices affected by all the bones are recorded to form a weight table, each value of the weight of the mesh vertices on the weight table has to be normalized so that the sum of all the values is 1.0.

9. The animation generation system as claimed in claim 1 further comprising:

a display unit, coupled to the arithmetic unit, for displaying the virtual space and the avatar.

10. The animation generation system as claimed in claim 4, wherein the AR animation is a 2D image stream, or a multi-view image stream.

11. The animation generation system as claimed in claim 10, wherein the synthesizing method of the multi-view image stream is an interlaced signal synthesizing method or a collateral signal synthesizing method.

12. An animation generation method for generating an avatar in a virtual space, wherein the avatar has a set of skeletons and a skin attached to the set of skeletons, and the set of skeletons has a plurality of movable nodes, and the movable nodes are manipulated so that a motion of the skin is induced;

comprising;

moving a position mark to at least one first real position in a real space;

moving at least one control mark to at least one second real position in the real space;

capturing the images of the real space;

identifying the first real position and the second real position from the images of the real space;

converting the first real position into a first virtual position where the avatar is in the virtual space;

converting the second real position into a second virtual position where one of the movable nodes of the avatar is in the virtual space, wherein the relative motions between the virtual control position and the first virtual position make the avatar perform a series of successive motions in the virtual space,

drawing the one of images of the real space as a background, and drawing a designated object on the first virtual position onto the background to generate an Augmented Reality (AR) animation,

reading out skin information in respect of the skin, and skeleton information in respect of a set of template skeletons, wherein the skin information is composed of data of a plurality of mesh vertices, and the skeleton information comprises geometric data of each bone of the set of template skeletons and the linkage relationship between the bones of the set of template skeletons;

analyzing the skin information to obtain a plurality of appearance features and a plurality of trunk features of the skin;

adjusting the size of the set of template skeletons according to the appearance features to generate the set of skeletons of the avatar, and fitting the set of skeletons to the appearance features;

fitting the set of skeletons to the trunk features of the skin;

calculating an envelope range of a bone of the set of skeletons according to a plurality of mesh vertices in proximity to the bone;

calculating the weight of a mesh vertex of the skin information relative to a bone according the envelope range, wherein the weight of the mesh vertex indicates how the mesh vertex is affected by the movement of the bone when the bone is moving; and

outputting the avatar to the arithmetic unit, wherein the outputted avatar comprises data in relation to the skin, the set of skeletons and the relationships therebetween.

13. The animation generation method as claimed in claim 12, wherein the designated object is a successive motion.

14. The animation generation method as claimed in claim 12, wherein the designated object is a virtual three dimension (3D) model.

15. The animation generation method as claimed in claim 12, wherein the AR animation is a 2D image stream, or a multi-view image stream.

16. The animation generation method as claimed in claim 12, wherein the envelope range is two-layer-barrel-shaped, and each of the two ends of the envelope range is a concentric hemisphere.

17. The animation generation method as claimed in claim 12, wherein rules for the mesh vertices weight calculating unit to calculate the mesh vertices weight comprises:

rule (1);

when a mesh vertex is inside the inner layer of the envelope of a bone, the weight of the mesh vertex in relation to the bone is 1.0, which indicates that the mesh vertex is totally affected by the movement of the bone when the bone is moving;

rule (2);

when a mesh vertex is outside of the outer layer of the envelope of a bone, the weight of the mesh vertex in relation to the bone is 0.0, which indicates that the mesh vertex is totally unaffected by the movement of the bone when the bone is moving;

rule (3) when a mesh vertex is between the inner and outer layer of the envelope range of a bone, the weight of the mesh vertex in relation to the bone decreases from 1.0 to 0.0, which indicates that the mesh vertex is partially affected by the movement of the bone according to the distance between the mesh vertex and the bone;

rule (4) when a mesh vertex does not belong to any envelope range of any bone in accordance with the former three rules, the weight of the mesh vertex in relation to a bone closest to the mesh vertex is 1.0, which indicates that the mesh vertex is totally affected by the movement of the bone closest to the mesh vertex;

rule (5) when all the weights of the mesh vertices affected by all the set of skeletons are recorded to form a weight table, each value of the weight of the mesh vertices on the weight table has to be normalized so that the sum of all the values is 1.0.

18. A multi-view animation generation system, comprising:

at least one mark which is moved by users to at least one first real position in a real space;

an arithmetic unit, coupled to at least two video capturing units,

wherein the at least two video capturing units capturing image streams of the real space and transmitting the image stream to the arithmetic unit, and the arithmetic unit identifying at least two first real positions from the images of the real space, and converting the at least two real positions into at least two virtual positions in virtual space;

a multi-view animation synthesizing unit, coupled to the arithmetic unit, for drawing one of the images captured by the video capturing unit as a background;

drawing a virtual model corresponding to the mark on one of the at least two virtual positions onto the background;

synthesizing the background and the virtual model to generate an animation; and

transmitting the animation to the multi-view display unit,

an avatar generation module, comprising;

a readout unit for reading out skin information in respect of a skin of an avatar, and skeleton information in respect of a set of template skeletons of the avatar, wherein the skin information is composed of data of a plurality of mesh vertices, and the skeleton information comprises geometric data of each bone of the set of template skeletons and the linkage relationship between bones of the set of template skeletons;

an analyzing unit for analyzing the skin information to obtain a plurality of appearance features and a plurality of trunk features of the skin;

a rough fitting unit for adjusting the size of the set of template skeletons according to the appearance features to generate the set of skeletons of the avatar, and fitting the set of skeletons to the appearance features;

a precise fitting unit for fitting the set of skeletons to the trunk features of the skin;

an envelope range calculating unit for calculating an envelope range of a bone of the set of skeletons according to a plurality of mesh vertices in proximity to the bone;

a mesh vertices weight calculating unit for calculating the weight of a mesh vertex of the skin information relative to a bone according the envelope range, wherein the weight of the mesh vertex indicates how the mesh vertex is affected by the movement of the bone when the bone is moving;

an output unit for outputting the avatar to the arithmetic unit, wherein the outputted avatar comprises data in relation to the skin, the set of skeletons and the relationships therebetween.

19. A multi-view video generation method for synthesizing a multi-view video and drawing a virtual model in the multi-view video, comprising:

placing a mark on at least one real position in a real space;

capturing the images of the real space by using at least two video capturing units;

identifying at least two real positions from the images of the real space;

converting the at least two real positions into at least two virtual positions in a virtual space; and

synthesizing a multi-view video, comprising;

drawing one of the images captured by the video capturing unit as a background;

drawing a virtual model corresponding to the mark on one of the at least two virtual positions onto the background to form a result video; and

synthesizing the background and the virtual model by using known multi-view synthesizing methods to generate a multi-view video, and

reading out skin information in respect of a skin of an avatar, and skeleton information in respect of a set of template skeletons of the avatar, wherein the skin information is composed of data of a plurality of mesh vertices, and the skeleton information comprises geometric data of each bone of the set of template skeletons and the linkage relationship between the bones of the set of template skeletons;

analyzing the skin information to obtain a plurality of appearance features and a plurality of trunk features of the skin;

adjusting the size of the set of template skeletons according to the appearance features to generate the set of skeletons of the avatar, and fitting the set of skeletons to the appearance features;

fitting the set of skeletons to the trunk features of the skin;

calculating an envelope range of a bone of the set of skeletons according to a plurality of mesh vertices in proximity to the bone;

calculating the weight of a mesh vertex of the skin information relative to a bone according the envelope range, wherein the weight of the mesh vertex indicates how the mesh vertex is affected by the movement of the bone when the bone is moving; and

outputting the avatar to the arithmetic unit, wherein the outputted avatar comprises data in relation to the skin, the set of skeletons and the relationships therebetween.