Virtual television phone apparatus

• US 20030117485A1
• Filed: 12/19/2002
• Published: 06/26/2003
• Est. Priority Date: 12/20/2001
• Status: Active Grant

• Current Assignee: Matsushita Electric Industrial Company Limited (Panasonic Holdings Corporation)
• Original Assignee: Matsushita Electric Industrial Company Limited (Panasonic Holdings Corporation)
• Inventors: Ohtani, Naoki, Asahara, Shigeo, Naka, Toshiya, Mochizuki, Yoshiyuki, Yamamoto, Takeshi, Hijiri, Toshiki, Orimoto, Katsunori
• Granted Patent: US 6,909,453 B2
• Time in Patent Office: Days
• US Class Current: 348/14.01
• CPC Class Codes: A63F 2300/1081 Input via voice recognition
• CPC Class Codes: A63F 2300/572 Communication between players during game play of non game information, e.g. e-mail, chat, file transfer, streaming of audio and streaming of video
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G10L 17/26 Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices
• CPC Class Codes: G10L 2021/105 Synthesis of the lips movements from speech, e.g. for talking heads

Abstract

A communication unit 1 carries out voice communication, and a character background selection input unit 2 selects a CG character corresponding to a communication partner. A voice/music processing unit 5 performs voice/music processing required for the communication, a voice/music converting unit 6 converts voice and music, and a voice/music output unit outputs the voice and music. A voice input unit 8 acquires voice. A voice analyzing unit 9 analyzes the voice, and an emotion presuming unit 10 presumes an emotion based on the result of the voice analysis. A lips motion control unit 11, a body motion control unit 12 and an expression control unit 13 send control information to a 3-D image drawing unit 14 to generate an image, and a display unit 15 displays the image.

Claims

1. A virtual television phone apparatus comprising:

a communication unit operable to carry out voice communication;

a character selecting unit operable to select CG character shape data for at least one of a user and a communication partner;

a voice input unit operable to acquire voice of the user;

a voice output unit operable to output voice of the communication partner;

a voice analyzing unit operable to analyze voice data of the communication partner received by the communication unit or both of the voice data of the communication partner and voice data of the user;

an emotion presuming unit operable to presume an emotion state of the communication partner or emotion states of both of the communication partner and the user using a result of the voice analysis by the voice analyzing unit;

a motion control unit operable to control a motion of the CG character based on the presumption by the emotion presuming unit;

an image generating unit operable to generate an image using the CG character shape data and motion data generated based on control information generated by the motion control unit; and

a displaying unit operable to display the image generated by the image generating unit.

2. The virtual television phone apparatus according to claim 1,

wherein the emotion presuming unit notifies the motion control unit of a result of the presumption by the emotion presuming unit, and

the motion control unit generates the motion data based on the notice.

3. The virtual television phone apparatus according to claim 1,

wherein the motion control unit includes a lips motion control unit operable to generate lips motion control information of the CG character data based on a result of the voice analysis by the voice analyzing unit, and

the image generating unit generates the image using the CG character shape data and the lips motion data generated based on control information generated by the lips motion control unit.

4. The virtual television phone apparatus according to claim 3,

wherein the emotion presuming unit notifies the lips motion control unit of a result of the presumption by said emotion presuming unit, and

the lips motion control unit generates the lips motion data based on the notice.

5. The virtual television phone apparatus according to claim 4 further comprising:

a storage unit operable to store the lips motion data; and

a unit operable to download the lips motion data from an external device and store said lips motion data in the storage unit.

6. The virtual television phone apparatus according to claim 4 further comprising:

a storage unit operable to store lips motion pattern data; and

a unit operable to download the lips motion pattern data from an external device and store said lips motion pattern data in the storage unit.

7. The virtual television phone apparatus according to claim 1,

wherein the motion control unit includes a body motion control unit operable to control a body motion of the CG character, and

the image generating unit generates the image using body motion data generated by the body motion control unit based on body motion control information.

8. The virtual television phone apparatus according to claim 7,

wherein the emotion presuming unit notifies the body motion control unit of a result of the presumption by said emotion presuming unit, and

the body motion control unit generates the body motion data based on the notice.

9. The virtual television phone apparatus according to claim 8 further comprising:

a storage unit operable to store the body motion data; and

a unit operable to download the body motion data from an external device and store said body motion data in the storage unit.

10. The virtual television phone apparatus according to claim 8 further comprising a selecting unit operable to select body motion pattern data which defines a specific body motion,

wherein the body motion control unit controls the body motion based on the body motion pattern data selected by the selecting unit.

11. The virtual television phone apparatus according to claim 9 further comprising:

a storage unit operable to store body motion pattern data; and

a unit operable to download the body motion pattern data from an external device and store said body motion pattern data in the storage unit.

12. The virtual television phone apparatus according to claim 8 further comprising a unit operable to decide the body motion of the CG character and control start of said body motion.

13. The virtual television phone apparatus according to claim 1,

wherein the motion control unit includes an expression control unit operable to control an expression of the CG character, and

the image generating unit generates an image using expression data generated by the expression control unit based on expression control information.

14. The virtual television phone apparatus according to claim 13,

wherein the emotion presuming unit notifies the expression control unit of a result of the presumption by said emotion presuming unit, and

the expression control unit generates the expression data based on the notice.

15. The virtual television phone apparatus according to claim 14 further comprising:

a storage unit operable to store the expression data; and

a unit operable to download the expression data from an external device and store said expression data in the storage unit.

16. The virtual television phone apparatus according to claim 14 further comprising:

a storage unit operable to store expression pattern data; and

a unit operable to download the expression pattern data from an external device and store said expression pattern data in the storage unit.

17. The virtual television phone apparatus according to claim 14 further comprising a unit operable to decide the expression of the CG character and control start of said expression.

18. The virtual television phone apparatus according to claim 1 further comprising a voice converting unit operable to convert the received voice of the communication partner into another voice.

19. The virtual television phone apparatus according to claim 18 further comprising a voice selection input unit operable to select quality of the voice of the communication partner when the voice converting unit converts said voice into another voice.

20. The virtual television phone apparatus according to claim 1,

wherein the image generating unit generates an image of the CG character of the communication partner upon receipt of calling from said partner, and

the display unit displays the image of the CG character during the period from the receipt of the calling until start of voice communication to inform the user of a voice communication waiting state.

21. The virtual television phone apparatus according to claim 1,

wherein the voice output unit outputs music data corresponding to the communication partner upon receipt of the calling from said partner to inform the user of a voice communication waiting state.

22. The virtual television phone apparatus according to claim 21 further comprising:

a storage unit operable to store the music data; and

a unit operable to download the music data from an external device and store said music data in the storage unit.

23. The virtual television phone apparatus according to claim 1,

wherein the image generating unit generates an image using background data.

24. The virtual television phone apparatus according to claim 23 further comprising a background selecting unit operable to select the background data.

25. The virtual television phone apparatus according to claim 24 further comprising:

a storage unit operable to store the background data; and

a unit operable to download the background data from an external device and store said background data in the storage unit.

26. The virtual television phone apparatus according to claim 1,

wherein the image generating unit generates a three-dimensional image.

27. The virtual television phone apparatus according to claim 1 further comprising:

a storage unit operable to store clothing texture data of the CG character; and

a unit operable to download the clothing texture data of the CG character from an external device and store said clothing texture data in the storage unit.

28. The virtual television phone apparatus according to claim 1 further comprising:

a storage unit operable to store the CG character shape data; and

a unit operable to download the CG character shape data from an external device and store said CG character shape data in the storage unit.

29. The virtual television phone apparatus according to claim 1 further comprising a selecting unit operable to select a display mode indicating whether the CG character is displayed or not.

30. The virtual television phone apparatus according to claim 29,

wherein the display mode is one of a communication partner display mode for displaying the CG character of the communication partner only, a concurrent display mode for displaying both the CG characters of the communication partner and the user, and a non-display mode for not displaying the CG character.

31. The virtual television phone apparatus according to claim 1 further comprising a viewpoint changing unit operable to display the CG character from a viewpoint according to the user'"'"'s instruction.

32. A virtual television phone system for communicating between at least a communication device of a user and a communication device of a communication partner, the system comprising at least the communication device of the user and the communication device of the communication partner,

wherein the communication device includes:

a communication unit operable to carry out voice communication;

a character selecting unit operable to select CG character shape data for at least one of a user and a communication partner;

a voice input unit operable to acquire voice of the user;

a voice output unit operable to output voice of the communication partner;

a voice analyzing unit operable to analyze voice data of the communication partner received by the communication unit or both of the voice data of the communication partner and voice data of the user;

an emotion presuming unit operable to presume an emotion state of the communication partner or emotion states of both of the communication partner and the user using a result of the voice analysis by the voice analyzing unit;

a motion control unit operable to control a motion of the CG character based on the presumption by the emotion presuming unit;

an image generating unit operable to generate an image using the CG character shape data and motion data generated based on control information generated by the motion control unit; and

a displaying unit operable to display the image generated by the image generating unit.

33. The virtual television phone system according to claim 32,

wherein the emotion presuming unit notifies the motion control unit of a result of the presumption by said emotion presuming unit, and

the motion control unit generates the motion data based on the notice.

34. A program for virtual television phone communication between at least a communication device of a user and a communication device of a communication partner by communication between the user and the communication partner, the program comprising:

a communication step for carrying out voice communication;

a character selecting step for selecting CG character shape data for at least one of the user and the communication partner;

a voice input step for acquiring voice of the user;

a voice output step for outputting voice of the communication partner;

a voice analyzing step for analyzing voice data of the communication partner received in the communication step or both of the voice data of the communication partner and voice data of the user;

an emotion presuming step for presuming an emotion state of the communication partner or emotion states of both of the communication partner and the user using a result of the voice analysis in the voice analyzing step;

a motion control step for controlling a motion of the CG character based on the presumption in the emotion presuming step;

an image generating step for generating an image using the CG character shape data and motion data generated based on control information generated in the motion control step; and

a displaying step for displaying the image generated in the image generating step.

35. The program according to claim 34,

wherein in the motion control step, the motion data is generated based on a result of the presumption in the emotion presuming step.