Text-to-speech and image generation of multimedia attachments to e-mail

• US 7,356,470 B2
• Filed: 10/18/2005
• Issued: 04/08/2008
• Est. Priority Date: 11/10/2000
• Status: Expired due to Fees

• Current Assignee: Gabmail IP Holdings, LLC
• Original Assignee: Adam Roth, Barclay A. Dunn, Geoffrey O'Sullivan
• Inventors: Roth, Adam, O'Sullivan, Geoffrey, Dunn, Barclay A.
• Primary Examiner(s): CHAWAN, VIJAY B
• Application Number:
  US11/253,507
• Publication Number:
  US 20060041430A1
• Time in Patent Office: 903 Days
• Field of Search: 704270-276, 704/201, 704/262, 704/231, 704/235, 704/260, 704/270.1, 345/643, 345/473, 435/185, 435/169, 348/515, 434/185
• US Class Current: 704/270
• CPC Class Codes: G06Q 10/107 Computer-aided management of electronic mailing [e-mailing]
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

A multi-mail system and method is disclosed in which a sender may convey and a recipient can realize emotional aspects associated with substantive content of a multi-mail message by receiving a message that is more than textual in nature. Voice recognition technology and programmatic relation of sound and graphics may be used to produce a talking image. In one embodiment, the image may include the user'"'"'s own visual and/or audio likeness. In an alternate embodiment, the image may comprise any available visual and/or audio display selected by the user. The multi-mail message may be inputted by a user in a text format and transposed into a format including the selected image and/or voice. In an alternate embodiment, a spoken message may be converted into a format including the selected image and/or voice. The formatted messages are then stored and/or transmitted via an email system or some other electronic network.

Claims

1. A method for preparing a multi-mail message for a user of a multi-mail system for transmission over a network, comprising:

providing a database configured to store multimedia data that represent human emotions in order to assist the user in conveying an intended emotional context of the user'"'"'s multi-mail message;

receiving audio information relating to content of the multi-mail message from a first user;

translating the audio information to text information;

identifying a portion or portions of the text information, including individual words and or phrases, that can be associated with the intended emotional context of the user'"'"'s multi-mail message;

providing the user with an interface that enables the user to select one of the multimedia data from the database that matches with the intended emotional context of the user'"'"'s multi-mail message;

associating the portion or portions of the text information, including individual words or phrases, to the multimedia data suggested by a program of the multi-mail system and then selected by the user thereby generating the multi-mail message having a dynamic image; and

transmitting the multi-mail message having the dynamic image to a recipient over the network thereby enabling the recipient to read, see and hear the intended emotional context of the user'"'"'s multi-mail message.

2. The method according to claim 1, wherein the audio information is similar to sound generated by a user'"'"'s voice.

3. The method according to claim 1, further comprising:

generating sound similar to a generic voice sample based on received audio information.

4. The method according to claim 1, further comprising:

generating one or more stored sounds based on received audio information.

5. The method according to claim 1, further comprising:

creating one or more multimedia components associated with content of the multi-mail message.

6. The method according to claim 5, wherein the multimedia component comprises image information.

7. The method according to claim 6, wherein the image information may be static or dynamic.

8. The method according to claim 6, wherein synthesis of the image multimedia component further comprises:

identifying speech movement image feature; and

generating frames representing movement of said image features.

9. The method of claim 5, wherein the multimedia component comprises audio information and image information.

10. The method according to claim 9, wherein synthesis of the audio and image information of the multimedia component further comprises:

composing a phoneme sequence;

composing a mouth frame time sequence which matches the phoneme time sequence;

composing speech movement image frame sequence; and

combining the image information and phoneme sequences.

11. The method according to claim 9, further comprising:

varying said multimedia component to convey one or more of the intended emotions the user wishes to convey in the content of the multi-mail message.

12. The method according to claim 11, wherein the intended emotional context of the user'"'"'s multi-mail message is conveyed by manipulating one or more the image information.

13. The method according to claim 11, wherein the intended emotional context of the user'"'"'s multi-mail message is conveyed by manipulating one or more the audio information.

14. A system for preparing a multi-mail message for a user of a multi-mail system for transmission over a network, comprising:

means for providing a database configured to store multimedia data on common human emotions that enables the user to select an intended emotional context of a portion or portions of the user'"'"'s multi-mail message;

means for receiving audio information relating to content of the multi-mail message from the user;

means for translating the audio information to text information;

means for identifying a portion or portions of the text information, including individual words or phrases, that can be associated with the intended emotional context of the user'"'"'s multi-mail message;

means for providing the user with an interface that enables the user to select one of the multimedia data from the database that matches with the intended emotional context of the user'"'"'s multi-mail message;

means for associating the portion or portions of the text information, including individual words or phrases, to the multimedia data suggested by a program of the multi-mail system and then selected by the user thereby generating the multi-mail message having a dynamic image; and

means for transmitting the multi-mail message to a recipient over the network thereby enabling the recipient to read, see and hear the intended emotional context of the user'"'"'s multi-mail message.

15. The system according to claim 14, wherein the audio information is similar to sound generated by a user'"'"'s voice.

16. The system according to claim 14, further comprising:

means for generating sound similar to a generic voice sample based on received audio information.

17. The system according to claim 14, further comprising:

means for generating one or more stored sounds based on received audio information.

18. The system according to claim 14, further comprising:

means for creating one or more multimedia components associated with the intended emotional context of the user'"'"'s multi-mail message.

19. The system according to claim 18, wherein the multimedia component comprises image information.

20. The system according to claim 19, wherein the image information may be static or dynamic.

21. The system according to claim 19, wherein synthesis of the image multimedia component further comprises:

means for identifying speech movement image feature; and

means for generating frames representing movement of said image features.

22. The system of claim 18, wherein the multimedia component comprises audio information and image information.

23. The system according to claim 22, wherein synthesis of the audio and image information of the multimedia component further comprises:

means for composing a phoneme sequence;

means for composing a mouth frame time sequence which matches the phoneme time sequence;

means for composing speech movement image frame sequence; and

means for combining the image information and phoneme sequences.

24. The system according to claim 22, further comprising:

means for varying said multimedia component to convey one or more of the intended emotions the user wishes to convey in the content of the multi-mail message.

25. The system according to claim 24, wherein the intended emotional context of the user'"'"'s multi-mail message is conveyed by manipulating one or more of the image information.

26. The system according to claim 24, wherein the intended emotional context of the user'"'"'s multi-mail message is conveyed by manipulating one or more element of the audio information.

27. A system for preparing a multi-mail message for a user of a multi-mail system for transmission over a network, comprising:

a database configured to store multimedia data that represent human emotions in order to assist the user in conveying an intended emotional context of the user'"'"'s multi-mail message; and

a central processing unit configured to receive audio information relating to content of the multi-mail message, translate the audio information to text information, identify a portion or portions of the text information, including individual words or phrases, that can be associated with the intended emotional context of the user'"'"'s multi-mail message, provide the user with an interface that enables the user to select one of the multimedia data from the database that matches with the intended emotional context of the user'"'"'s multi-mail message, associate a portion or portions of the text information, including individual words or phrases, to the multimedia data suggested by a program of the multi-mail system and then selected by the user thereby generating the multi-media message having a dynamic image, and transmit the multi-mail message to a recipient over the network thereby enabling the recipient to read, see and hear the intended emotional context of the user'"'"'s multi-mail message.

28. The system according to claim 27, wherein the audio information is similar to sound generated by a user'"'"'s voice.

29. The system according to claim 27, wherein the central processing unit is further configured for generating sound similar to a generic voice sample based on received audio information.

30. The system according to claim 27, wherein the central processing unit is further configured for generating one or more stored sounds based on received audio information.

31. The system according to claim 27, wherein the central processing unit is further configured for creating one or more multimedia components associated with the content of the multi-mail message.

32. The system according to claim 31, wherein the multimedia component comprises image information.

33. The system according to claim 32, wherein the image information may be static or dynamic.

34. The system according to claim 32, wherein the central processing unit is further configured for identifying speech movement image feature, and for generating frames representing movement of said image features.

35. The system of claim 31, wherein the multimedia component comprises audio information and image information.

36. The system according to claim 35, wherein the central processing unit is further configured for composing a phoneme sequence, for composing a mouth frame time sequence which matches the phoneme time sequence, for composing speech movement image frame sequence, and for combining the image information and phoneme sequences.

37. The system according to claim 35, wherein the central processing unit is further configured for varying said multimedia component to convey one or more of the intended emotions the user wishes to convey in the content of the user'"'"'s multi-mail message.

38. The system according to claim 37, wherein the central processing unit is further configured for manipulating one or more the image information to convey the intended emotional context of the user'"'"'s multi-mail message.

39. Computer executable software code stored on a computer readable medium, the code for preparing a multi-mail message for a user of a multi-mail system for transmission over a network, comprising:

code to provide a database configured to store multimedia data that represent human emotions in order to assist the user in conveying an intended emotional context of the user'"'"'s multi-mail message;

code to receive audio information relating to content of the multi-mail message from the user;

code to translate the audio information to text information;

code to identify a portion or portions of the text information, including individual words and/or phrase, that can be associated with the intended emotional context of the user'"'"'s multi-mail message;

code to provide the user with an interface that enables the user to select one of the multimedia data from the database that matches with the intended emotional context of the user'"'"'s multi-mail message;

code to associate a portion or portions of the text information, including individual words and/or phrases, to the multimedia data suggested by a program of the multi-mail system and then selected by the user thereby generating the multi-mail message having a dynamic image; and

code to transmit the multi-mail message to a recipient over the network thereby enabling the recipient to read, see and hear the intended emotional context of the user'"'"'s multi-mail message.

40. A programmed computer for preparing a multi-mail message for a user of a multi-mail system for transmission over a network, comprising:

a memory having at least one region for storing computer executable program code; and

a processor for executing the program code stored in memory, wherein the program code includes;

code to provide a database configured to store multimedia data that represent human emotions in order to assist the user in conveying an intended emotional context of the user'"'"'s multi-mail message;

code to receive audio information relating to content of the multi-mail message from a first user;

code to translate the audio information to text information;

code to identify a portion or portions of the text information, including individual words and/or phrases, that can be associated with the intended emotional context of the user'"'"'s multi-mail message;

code to provide the user with an interface that enables the user to select one of the multimedia data from the database that matches with the intended emotional context of the user'"'"'s multi-mail message;

code to associate the portion or portions of the text information, including individual words and/or phrases, to the multimedia data suggested by a program of the multi-mail system and then selected by the user thereby generating the multi-media message having a dynamic image; and

code to transmit the multi-mail message to a recipient over the network thereby enabling the recipient to read, see and hear the intended emotional context of the user'"'"'s multi-mail message.

41. A computer readable medium having computer executable software code stored thereon, the code for preparing a multi-mail message of a user of a multi-mail system for transmission over a network, comprising:

code to provide a database configured to store multimedia data that represent human emotions in order to assist the user in conveying an intended emotional context of the user'"'"'s multi-mail message;

code to receive audio information relating to content of the multi-mail message;

code to translate the audio information to text information; and

code to identify a portion or portions of the text information, including individual words or phrases that can be associated with the intended emotional context of the user'"'"'s multi-mail message;

code to provide the user with an interface that enables the user to select one of the multimedia data from the database that matches with the intended emotional context of the user'"'"'s multi-mail message;

code to associate the portion or portions of the text information, including individual words or phrases, to the multimedia data suggested by a program of the multi-mail system and then selected by the user thereby generating the multi-media message having a dynamic image; and

code to transmit the multi-mail message to a recipient over the network thereby enabling the recipient to read, see and hear the intended emotional context of the user'"'"'s multi-mail message.