Interactive character system

• US 7,478,047 B2
• Filed: 10/29/2001
• Issued: 01/13/2009
• Est. Priority Date: 11/03/2000
• Status: Active Grant

• Current Assignee: Zoesis, Inc.
• Original Assignee: Zoesis, Inc.
• Inventors: Loyall, A. Bryan, Bates, Joseph, Reilly, W. Scott Neal, Leone, Mark Russell
• Primary Examiner(s): Opsasnick; Michael N
• Application Number:
  US10/415,851
• Publication Number:
  US 20040075677A1
• Time in Patent Office: 2,633 Days
• Field of Search: 704/258, 704/265, 704/269, 704/277, 704/278
• US Class Current: 704/258
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06Q 30/0276 Advertisement creation
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 13/033 Voice editing, e.g. manipulating the voice of the synthesiser
• CPC Class Codes: G10L 2021/105 Synthesis of the lips movements from speech, e.g. for talking heads

Abstract

A system and method for controlling a synthetic character using a control system displays the character engaged in an activity, receiving a first input from a user, determines whether the input is relevant to the activity, if the input is relevant to the activity, and shows the character react to the input, the character being highly expressive and highly reactive. A system and method for displaying a synthetic character provides speech data, creates modified speech data by modifying at least one of the pitch or duration of at least a portion of the speech data and generates modified speech sounds associated with the character using the modified speech data.

Claims

1. A method for controlling the voice of a synthetic character, which is autonomous and interacts with others in a shared environment, comprising:

providing speech data corresponding to at least a part of an intended communication generated by the character;

creating modified speech data by modifying, by an automatically determined amount, at least one of the pitch or duration of at least a portion of the speech data;

generating speech sounds associated with the character using the modified speech data; and

associating a representation of an emotional state with the character; and

wherein creating modified speech data comprises modifying the speech data based on the emotional state representation and

wherein the emotional state representation is determined at least in part by interaction of the character with others in a shared environment.

2. A method for controlling the voice of a synthetic character that is autonomous and interacts with others in a shared environment, such character having an associated mental state that changes over time at least in part as a function of said mental state, said mental state comprising a dynamic perceptual state that depends on said shared environment, said shared environment including the entities with which the character is interacting, and an action state defining actions to be taken by the character, the method comprising:

providing speech data corresponding to at least a part of an intended communication generated by the character;

creating modified speech data by modifying, by an automatically determined amount, at least one of the pitch or duration of at least a portion of the speech data, such modification based at least in part on at least a portion of said mental state; and

generating speech sounds associated with the character using the modified speech data;

wherein the mental state is determined at least in part by interaction of the character with others in a shared environment.

3. The method of claim 2 wherein said modification of pitch or duration is performed at least in part using the Synchronous Overlap and Add class of techniques.

4. The method of claim 2 wherein said interactive synthetic character is displayed within an interactive entertainment product.

5. The method of claim 2 wherein said mental state further comprises a defined emotional state.

6. The method of claim 5 wherein said modification of pitch or duration is based at least in part on the degree of intensity of said defined emotional state.

7. The method of claim 5 wherein said modification of pitch or duration is performed at least in part using the Synchronous Overlap and Add class of techniques.

8. The method of claim 2 wherein said mental state further comprises a defined motivational or goal state.

9. The method of claim 8 wherein said modification of pitch or duration is performed at least in part using the Synchronous Overlap and Add class of techniques.

10. The method of claim 8 wherein said mental state further comprises a defined emotional state.

11. The method of claim 10 wherein said modification of pitch or duration is based at least in part on the degree of intensity of said defined emotional state.

12. The method of claim 11 further comprising changing said emotional state in response to a failure or success of a motivation or goal present in said motivational or goal state.

13. The method of claim 8 wherein said motivation or goal state comprises a plurality of goals and subgoals or motivations and submotivations ordered at least in part in a hierarchical relationship.

14. The method of claim 13 wherein said mental state further comprises a defined emotional state.

15. The method of claim 14 wherein said modification of pitch or duration is based at least in part on the degree of intensity of said defined emotional state.

16. The method of claim 15 further comprising changing said emotional state in response to a failure or success of a motivation or goal present in said motivational or goal state.

17. The method of claim 2, wherein said speech sounds have a natural quality such as might be displayed by a living animal or person.

18. The method of claim 17, wherein said speech sounds are generated in coordination with movement of said synthetic character so that said coordination has a natural quality such as might be displayed by a living animal, person, or imaginary living creature.

19. The method of claim 18 wherein said interactive synthetic character is displayed within an interactive entertainment product.

20. The method of claim 18 wherein said mental state further comprises a defined emotional state.

21. The method of claim 20 wherein said modification of pitch or duration is based at least in part on the degree of intensity of said defined emotional state.

22. The method of claim 20 wherein said modification of pitch or duration is performed at least in part using the Synchronous Overlap and Add class of techniques.

23. The method of claim 18 wherein said mental state further comprises a defined motivational or goal state.

24. The method of claim 23 wherein said modification of pitch or duration is performed at least in part using the Synchronous Overlap and Add class of techniques.

25. The method of claim 23 wherein said mental state further comprises a defined emotional state.

26. The method of claim 25 wherein said modification of pitch or duration is based at least in part on the degree of intensity of said defined emotional state.

27. The method of claim 26 further comprising changing said emotional state in response to a failure or success of a motivation or goal present in said motivational or goal state.

28. The method of claim 23 wherein said motivation or goal state comprises a plurality of goals and subgoals or motivations and submotivations ordered at least in part in a hierarchical relationship.

29. The method of claim 28 wherein said mental state further comprises a defined emotional state.

30. The method of claim 29 wherein said modification of pitch or duration is based at least in part on the degree of intensity of said defined emotional state.

31. The method of claim 30 further comprising changing said emotional state in response to a failure or success of a motivation or goal present in said motivational or goal state.

32. The method of claim 18 wherein said modification of pitch or duration is performed at least in part using the Synchronous Overlap and Add class of techniques.

33. A method for controlling the voice of a synthetic character, which is autonomous and interacts with others in a shared environment, comprising:

providing speech data corresponding to at least a part of an intended communication generated by the character;

creating modified speech data by modifying, by an automatically determined amount, at least one of the pitch or duration of at least a portion of the speech data; and

generating speech sounds associated with the character using the modified speech data;

wherein the character has a specified personality; and

wherein the automatically determined amount is determined, at least in part, by a state of the character that depends, at least in part, on interaction of the character with others in the shared environment.

34. A method for controlling the voice of a synthetic character, which is autonomous and interacts with others in a shared environment, comprising:

providing speech data corresponding to at least a part of an intended communication generated by the character;

creating modified speech data by modifying, by an automatically determined amount, at least one of the pitch or duration of at least a portion of the speech data; and

generating speech sounds associated with the character using the modified speech data;

wherein the character is highly expressive, highly reactive and has a specified personality

wherein the automatically determined amount is determined, at least in part, by a state of the character that depends, at least in part, on interaction of the character with others in the shared environment.

35. A method for controlling the voice of a synthetic character, which is autonomous and interacts with others in a shared environment, comprising:

providing speech data corresponding to at least a part of an intended communication generated by the character;

creating modified speech data by modifying, by an automatically determined amount, at least one of the pitch or duration of at least a portion of the speech data; and

generating speech sounds associated with the character using the modified speech data, wherein the character is highly reactive and has at least one of the following group of characteristics;

highly expressive, appearing to be intelligent, exhibiting common sense, exhibiting social knowledge, exhibiting knowledge of social norms, having a specified personality

wherein the automatically determined amount is determined, at least in part, by a state of the character that depends, at least in part, on interaction of the character with others in the shared environment.

36. A computer readable medium encoding computer readable instructions for controlling the voice of a synthetic character, which is autonomous and interacts with others in a shared environment, that cause a processor to perform steps comprising:

providing speech data corresponding to at least a part of an intended communication generated by the character;

creating modified speech data by modifying, by an automatically determined amount, at least one of the pitch or duration of at least a portion of the speech data; and

associating a representation of an emotional state with the character; and

wherein the instructions for creating modified speech data comprise instructions for modifying the speech data based on the emotional state representation and

wherein the emotional state representation is determined at least in part by interaction of the character with others in a shared environment.

37. A computer readable medium encoding computer readable instructions for controlling the voice of a synthetic character, which is autonomous and interacts with others in a shared environment, that cause a processor to perform steps comprising:

providing speech data corresponding to at least a part of an intended communication generated by the character;

creating modified speech data by modifying, by an automatically determined amount, at least one of the pitch or duration of at least a portion of the speech data; and

generating speech sounds associated with the character using the modified speech data, wherein the character is highly expressive

wherein the automatically determined amount is determined, at least in part, by a state of the character that depends, at least in part, on interaction of the character with others in the shared environment.

38. A computer readable medium encoding computer readable instructions for controlling a synthetic character that is autonomous and interacts with others in a shared environment, such character having an associated mental state that changes over time at least in part as a function of said mental state, said mental state comprising a dynamic perceptual state that depends on said shared environment, said shared environment including the entities with which the character is interacting, and an action state defining actions to be taken by the character, that cause a processor to perform steps comprising:

providing speech data corresponding to at least a part of an intended communication generated by the character;

creating modified speech data by modifying, by an automatically determined amount, at least one of the pitch or duration of at least a portion of the speech data, such modification based at least in part on at least a portion of said mental state; and

generating speech sounds associated with the character using the modified speech data

wherein the mental state is determined at least in part by interaction of the character with others in a shared environment.

39. The computer readable medium of claim 38 wherein said modification of pitch or duration is performed at least in part using the Synchronous Overlap and Add class of techniques.

40. The computer readable medium of claim 38 wherein said interactive synthetic character is displayed within an interactive entertainment product.

41. The computer readable medium of claim 38 wherein said mental state further comprises a defined emotional state.

42. The computer readable medium of claim 41 wherein said modification of pitch or duration is based at least in part on the degree of intensity of said defined emotional state.

43. The computer readable medium of claim 41 wherein said modification of pitch or duration is performed at least in part using the Synchronous Overlap and Add class of techniques.

44. The computer readable medium of claim 38 wherein said mental state further comprises a defined motivational or goal state.

45. The computer readable medium of claim 44 wherein said modification of pitch or duration is performed at least in part using the Synchronous Overlap and Add class of techniques.

46. The computer readable medium of claim 44 wherein said mental state further comprises a defined emotional state.

47. The computer readable medium of claim 46 wherein said modification of pitch or duration is based at least in part on the degree of intensity of said defined emotional state.

48. The computer readable medium of claim 47 further comprising instructions that cause a processor to perform the step of changing said emotional state in response to a failure or success of a motivation or goal present in said motivational or goal state.

49. The computer readable medium of claim 44 wherein said motivation or goal state comprises a plurality of goals and subgoals or motivations and submotivations ordered at least in part in a hierarchical relationship.

50. The computer readable medium of claim 49 wherein said mental state further comprises a defined emotional state.

51. The computer readable medium of claim 50 wherein said modification of pitch or duration is based at least in part on the degree of intensity of said defined emotional state.

52. The computer readable medium of claim 51 further comprising instructions that cause a processor to perform the step of changing said emotional state in response to a failure or success of a motivation or goal present in said motivational or goal state.

53. The computer readable medium of claim 38, wherein said speech sounds have a natural quality such as might be displayed by a living animal or person.

54. The computer readable medium of claim 53, wherein generating said speech sounds comprises generating said speech sounds in coordination with movement of said synthetic character so that said coordination has a natural quality such as might be displayed by a living animal, person, or imaginary living creature.

55. The computer readable medium of claim 54 wherein said modification of pitch or duration is performed at least in part using the Synchronous Overlap and Add class of techniques.

56. The computer readable medium of claim 54 wherein said interactive synthetic character is displayed within an interactive entertainment product.

57. The computer readable medium of claim 54 wherein said mental state further comprises a defined emotional state.

58. The computer readable medium of claim 57 wherein said modification of pitch or duration is based at least in part on the degree of intensity of said defined emotional state.

59. The computer readable medium of claim 57 wherein said modification of pitch or duration is performed at least in part using the Synchronous Overlap and Add class of techniques.

60. The computer readable medium of claim 54 wherein said mental state further comprises a defined motivational or goal state.

61. The computer readable medium of claim 60 wherein said modification of pitch or duration is performed at least in part using the Synchronous Overlap and Add class of techniques.

62. The computer readable medium of claim 60 wherein said mental state further comprises a defined emotional state.

63. The computer readable medium of claim 62 wherein said modification of pitch or duration is based at least in part on the degree of intensity of said defined emotional state.

64. The computer readable medium of claim 63 further comprising instructions that cause a processor to perform the step of changing said emotional state in response to a failure or success of a motivation or goal present in said motivational or goal state.

65. The computer readable medium of claim 60 wherein said motivation or goal state comprises a plurality of goals and subgoals or motivations and submotivations ordered at least in part in a hierarchical relationship.

66. The computer readable medium of claim 65 wherein said mental state further comprises a defined emotional state.

67. The computer readable medium of claim 66 wherein said modification of pitch or duration is based at least in part on the degree of intensity of said defined emotional state.

68. The computer readable medium of claim 67 further comprising instructions for changing said emotional state in response to a failure or success of a motivation or goal present in said motivational or goal state.

69. A computer readable medium encoding computer readable instructions for controlling the voice of a synthetic character, which is autonomous and interacts with others in a shared environment, that cause a processor to perform steps comprising:

providing speech data corresponding to at least a part of an intended communication generated by the character;

creating modified speech data by modifying, by an automatically determined amount, at least one of the pitch or duration of at least a portion of the speech data; and

generating speech sounds associated with the character using the modified speech data;

wherein the character has a specified personality; and

wherein the automatically determined amount is determined, at least in part, by a state of the character that depends, at least in part, on interaction of the character with others in the shared environment.

70. A computer readable medium encoding computer readable instructions for controlling the voice of a synthetic character, which is autonomous and interacts with others in a shared environment, that cause a processor to perform steps comprising:

providing speech data corresponding to at least a part of an intended communication generated by the character;

creating modified speech data by modifying, by an automatically determined amount, at least one of the pitch or duration of at least a portion of the speech data; and

generating speech sounds associated with the character using the modified speech data;

wherein the character is highly expressive, highly reactive and has a specified personality; and

wherein the automatically determined amount is determined, at least in part, by a state of the character that depends, at least in part, on interaction of the character with others in the shared environment.

71. A computer readable medium encoding computer readable instructions for controlling the voice of a synthetic character, which is autonomous and interacts with others in a shared environment, that cause a processor to perform steps comprising:

providing speech data corresponding to at least a part of an intended communication generated by the character;

creating modified speech data by modifying, by an automatically determined amount, at least one of the pitch or duration of at least a portion of the speech data; and

generating speech sounds associated with the character using the modified speech data, wherein the character is highly reactive and has at least one of the following group of characteristics;

highly expressive, appearing to be intelligent, exhibiting common sense, exhibiting social knowledge, exhibiting knowledge of social norms, having a specified personality;

wherein the automatically determined amount is determined, at least in part, by a state of the character that depends, at least in part, on interaction of the character with others in the shared environment.