Information Processing Device Capable of Displaying A Character Representing A User, and Information Processing Method Thereof.

• US 20120223952A1
• Filed: 02/06/2012
• Published: 09/06/2012
• Est. Priority Date: 03/01/2011
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Suzuki, Akira, Shimizu, Masao, Koguchi, Takahiro, Kanemaru, Yoshikatsu, Kumagae, Fumikazu
• Granted Patent: US 8,830,244 B2
• Time in Patent Office: Days
• US Class Current: 345/473
• CPC Class Codes: G06T 13/00 Animation
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06V 40/176 Dynamic expression
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G10L 21/10 Transforming into visible information

Abstract

The basic image specifying unit specifies the basic image of a character representing a user of the information processing device. The facial expression parameter generating unit converts the degree of the facial expression of the user to a numerical value. The model control unit determines an output model of the character for respective points of time. The moving image parameter generating unit generates a moving image parameter for generating animated moving image frames of the character for respective points of time. The command specifying unit specifies a command corresponding to the pattern of the facial expression of the user. The playback unit outputs an image based on the moving image parameter and the voice data received from the information processing device of the other user. The command executing unit executes a command based on the identification information of the command.

Claims

1. An information processing device comprising:

an image data storage operative to store data on models of a character representing a user, the model including a plurality of facial expression models providing different facial expressions;

a facial expression parameter generating unit operative to calculate a degree of facial expression for each facial expression type as a facial expression parameter by sequentially analyzing input moving image data acquired by capturing an image of a user, by deriving a numerical value representing the shape of a portion of a face for each input image frame, and by comparing the numerical value with a criteria value defined in advance;

a model control unit operative to first determine a weight for each of the plurality of facial expression models stored in the image data storage by using the facial expression parameter calculated by the facial expression parameter generating unit and a volume level obtained from voice data of the user acquired at the same time with the capturing of the image, to synthesize the plurality of facial expression models according to the weights, and to determine an output model of the character for points of time corresponding to each of the input image frames;

a moving image parameter generating unit operative to generate a moving image parameter for generating animated moving image frames of the character including the output model determined by the model control unit for respective points of time; and

an output unit operative to synchronize the moving image parameter generated by the moving image parameter generating unit and the voice data and to sequentially output.

2. The information processing device according to claim 1, wherein

the plurality of facial expression models stored in the image data storage include a plurality of vocal models indicating the statuses of face when pronouncing different sounds, and

the model control unit synthesizes the plurality of vocal models while changing the weights for the plurality of vocal models with different waveforms with time.

3. The information processing device according to claim. 2, wherein the different waveforms have amplitudes in proportion to the volume level and are absolute values of sine waves having different time cycles.

4. The information processing device according to claim 2, wherein the vocal models include a vocal model representing the status of a face when pronouncing the vowel “

a,”

a vocal model representing the status of a face when pronouncing the vowel “

i,” and

a vocal model representing the status of a face when pronouncing the vowel “

u”

.

5. The information processing device according to claim 2, wherein the facial expression parameter calculated by the facial expression parameter generating unit includes a degree of opening of a mouth, and the model control units adds the degree of opening of the mouth to at least one of the weights for the vocal models.

6. The information processing device according to claim 1, wherein the output unit sequentially transmits the moving image parameters to another information processing device via a network.

7. The information processing device according to claim 1, further comprising a command specifying unit operative to specify, based on the time variation of facial expressions that the facial expression parameter generating unit acquires by analyzing the input moving image data, identification information of the command for changing an image in accordance with the time variation,

wherein the output unit further outputs the identification information of the command specified by the command specifying unit so as to change the animated moving image of the character.

8. The information processing device according to claim 7, wherein

the output unit transmits the moving image parameter and the identification information of the command to another information processing device connected to the present information processing device via a network, and displays side by side on a display device:

a) the animated moving image of the character representing the user who is operating the present information processing device, the image being generated based on the moving image parameter; and

b) the animated moving image of the character representing another user who is operating the other information processing device, the image being generated based on the moving image parameter transmitted from the other information processing device, and

the animated moving image of the characters of both users are changed based on the identification information of the command, on condition that the identification information of the command transmitted to the other information processing device and the identification information of the command transmitted from the other information processing device are same.

9. The information processing device according to claim 7, wherein

the output unit transmits the moving image parameter and the identification information of the command to another information processing device connected to the present information processing device via a network, and displays side by side on a display device:

a) the animated moving image representing the character of the user who is operating the present information processing device, the image being generated based on the moving image parameter; and

b) the animated moving image of the character representing another user who is operating the other information processing device, the image being generated based on the moving image parameter transmitted from the other information processing device, and

the animated moving image of the characters of both users are changed based on either one of the identification information of the command transmitted to the other information processing device or the identification information of the command transmitted from the other information processing device.

10. The information processing device according to claim 1, wherein the model control unit determines the output model so as to set the costume of the user character as the costume of a character of a game that the user played right before by using the information processing device, the costume of the character of the game being associated with the game in advance.

11. The information processing device according to claim 1, wherein the model control unit defines a surrounding image captured simultaneously with the capturing of the image of the user as the back ground image of the character of the user.

12. An information processing method comprising:

calculating a degree of facial expression for a plurality of facial expression types as a facial expression parameter by sequentially analyzing input moving image data acquired by capturing an image of a user, by deriving a numerical value representing the shape of a portion of a face for a plurality of input image frames, and by comparing the numerical value with a criteria value defined in advance;

determining a weight for each of a plurality of models of a character representing a user, the models including a plurality of facial expression models providing different facial expressions stored in a memory by using the calculated facial expression parameter and a volume level obtained from voice data of the user acquired at the same time with the capturing of the image;

reading data of the plurality of facial expression models from the memory;

synthesizing the plurality of facial expression models while weighing with the weights, and determining an output model of the character for points of time corresponding to each of the plurality of input image frames;

generating a moving image parameter for generating animated moving image frames of the character including the output model for respective points of time; and

synchronizing the moving image parameter and the voice data, and outputting sequentially.

13. A computer program embedded on a non-transitory computer-readable recording medium, comprising:

a module configured to calculate a degree of facial expression for a plurality of facial expression types as a facial expression parameter by sequentially analyzing input moving image data acquired by capturing an image of a user, by deriving a numerical value representing the shape of a portion of a face for a plurality of input image frames, and by comparing the numerical value with a criteria value defined in advance;

a module configured to determine a weight for a plurality of models of a character representing a user, the models including a plurality of facial expression models providing different facial expressions stored in a memory by using the calculated facial expression parameter and a volume level obtained from voice data of the user acquired at the same time with the capturing of the image;

a module configured to read data of the plurality of facial expression models from the memory;

a module configured to synthesize the plurality of facial expression models while weighing with the weights, and to determine an output model of the character for points of time corresponding to each of the plurality of input image frames;

a module configured to generate a moving image parameter for generating animated moving image frames of the character including the output model for respective points of time; and

a module configured to synchronize the moving image parameter and the voice data, and to output sequentially.

14. A computer readable medium encoded with a program comprising:

a module configured to calculate a degree of facial expression for a plurality of facial expression types as a facial expression parameter by sequentially analyzing input moving image data acquired by capturing an image of a user, by deriving a numerical value representing the shape of a portion of a face for a plurality of input image frames, and by comparing the numerical value with a criteria value defined in advance;

a module configured to determine a weight for a plurality of models of a character representing a user, the models including a plurality of facial expression models providing different facial expressions stored in a memory by using the calculated facial expression parameter and a volume level obtained from voice data of the user acquired at the same time with the capturing of the image;

a module configured to read data of the plurality of facial expression models from the memory;

a module configured to synthesize the plurality of facial expression models while weighing with the weights, and to determine an output model of the character for points of time corresponding to each of the plurality of input image frames;

a module configured to generate a moving image parameter for generating animated moving image frames of the character including the output model for respective points of time; and

a module configured to synchronize the moving image parameter and the voice data, and to output sequentially.