Multi modal communications system

• US 7,333,507 B2
• Filed: 08/29/2002
• Issued: 02/19/2008
• Est. Priority Date: 08/31/2001
• Status: Active Grant

• Current Assignee: Communication Service For The DEAF Incorporated
• Original Assignee: Ben Soukup, Philip Bravin, Randy Gerloff
• Inventors: Bravin, Philip, Gerloff, Randy, Soukup, Ben
• Primary Examiner(s): Phunkulh; Bob A.
• Application Number:
  US10/230,757
• Publication Number:
  US 20030069997A1
• Time in Patent Office: 2,000 Days
• Field of Search: None
• US Class Current: 370/466
• CPC Class Codes: G06F 40/279 Recognition of textual entities
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 21/06 Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids G10L15/26 takes precedence
• CPC Class Codes: H04L 51/066 Format adaptation, e.g. format conversion or compression
• CPC Class Codes: H04L 65/103 in the network
• CPC Class Codes: H04L 65/104 in the network
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/1106 Call signalling protocols; H.323 and related
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/306 User profiles
• CPC Class Codes: H04L 67/56 Provisioning of proxy services store-and-forward switching systems in data switching networks H04L12/54
• CPC Class Codes: H04L 67/564 Enhancement of application control based on intercepted application data
• CPC Class Codes: H04L 67/565 Conversion or adaptation of application format or content adding application control or application functional data H04L67/561
• CPC Class Codes: H04L 67/567 Integrating service provisioning from a plurality of service providers
• CPC Class Codes: H04L 69/08 Protocols for interworking; Protocol conversion
• CPC Class Codes: H04L 69/18 Multiprotocol handlers, e.g. single devices capable of handling multiple protocols
• CPC Class Codes: H04L 69/329 in the application layer [OSI layer 7]
• CPC Class Codes: H04L 9/40 Network security protocols
• CPC Class Codes: H04M 3/42391 : where the subscribers are hearing-impaired persons, e.g. telephone devices for the deaf
• CPC Class Codes: H04N 21/234336 : by media transcoding, e.g. video is transformed into a slideshow of still pictures or audio is converted into text
• CPC Class Codes: H04W 4/12 : Messaging; Mailboxes; Announcements
• CPC Class Codes: H04W 76/10 : Connection setup
• CPC Class Codes: H04W 84/00 : Network topologies
• CPC Class Codes: H04W 88/06 : adapted for operation in multiple networks or having at least two operational modes, e.g. multi-mode terminals
• CPC Class Codes: H04W 88/14 : Backbone network devices

Abstract

A multi modal communications system for facilitating interpersonal conversational communications and messaging communications over a network structure. The multi modal communications system includes a portal system for transmitting and receiving communications between various communications modes, a first communications channel providing connectivity between a first user and the portal system, and a second communications channel providing connectivity between a second user and the portal system whereby the first user communicates with the second user through the portal system.

Claims

1. A multi-modal network system for interpersonal communications comprising:

a portal system for transmitting and receiving communications between various communications modes;

a first communications channel providing connectivity between a first user and said portal system, said first communications channel being selected from the group of communications modes consisting of Public Switched Telephone Network, CATV with telephony, wireless telecommunications network, satellite communications network, PCS, GSM, Internet, World Wide Web, SONET, H320 video-conferencing (ISDN), and H323 video-conferencing (IP);

a second communications channel providing connectivity between a second user and said portal system whereby said first user communications with said second user through said portal system, said second communications channel being selected from the group of communications modes consisting of Public Switched Telephone Network, CATV with telephony, wireless telecommunications network, satellite communications network, PCS, GSM, Internet, World Wide Web, SONET, H320 video-conferencing (ISDN), and H323 video-conferencing (IP);

said first communications channel being selected independently from selection of said second communications channel;

a data handling subsystem for identifying a first communication means being utilized over said first communications channel and a second communication means being utilized over said second communications channel;

said first and second communications means being selectable from a group of communications means consisting of voice, facsimile, data, ASCII text, TTY-baudot, video or ASCII soft brail, said first communication means being selected independently from selection of said second communications means;

said first communications means being selectable to accommodate personal communications preferences of the first user, said second communications means being selectable to accommodate personal communications preferences of the second user.

2. The system of claim 1, further comprising a speech recognition engine operationally coupled to said portal system, said speech recognition engine facilitating translation of spoken speech into message data, said message data being routable via said portal system through a selected combination of communications channel, communications mode, and communications means.

3. The system of claim 2, wherein said personal communications preferences further includes a spoken speech table such that said speech recognition engine being trainable for specific users based upon said spoke speech table.

4. The system of claim 1, further comprising a spoken language translator for converting a first spoken language to a second spoken language such that said first and second users do not need to use that same spoken language.

5. The system of claim 1, further comprising:

a messaging subsystem for routing message data from a first user to second user in a delayed mode, said messaging subsystem being operationally coupled to said portal system for receiving said message data;

a messaging profile for pre-selecting a method of delivery for a delayed message.

6. The system of claim 5 wherein said method of delivery being selected from a group of delivery methods consisting of voice mail, electronic mail, electronic paging, web-site access, and facsimile.

7. The system of claim 1, further comprising a visual language translator for converting a first visual language to a second visual language such that said first and second users do not need to use the same visual language.

8. The system of claim 1, further comprising a human interpreter for converting a spoken language to message data and converting message data to a spoken language;

a human interpreter interface operationally coupled to said portal system, said human interpreter interface having a data input means and a data output means facilitating interaction between said human interpreter and said portal system.

9. A multi-modal network system for interpersonal communications comprising:

a portal system for transmitting and receiving communications between various communications modes;

a first communications channel providing connectivity between a first user and said portal system, said first communications channel being selected from the group of communications modes consisting of Public Switched Telephone Network, CATV with telephony, wireless telecommunications network, satellite communications network, PCS, GSM, Internet, World Wide Web, SONET H320 video-conferencing (ISDN), and H323 video-conferencing (IP);

a second communications channel providing connectivity between a second user and said portal system whereby said first user communications with said second user through said portal system, said second communications channel being selected from the group of communications modes consisting of Public Switched Telephone Network, CATV with telephony, wireless telecommunications network, satellite communications network, PCS, GSM, Internet, World Wide Web, SONET, H320 video-conferencing (ISDN), and H323 video-conferencing (IP);

said first communications channel being selected independently from selection of said second communications channel;

a data handling subsystem for identifying a first communication means being utilized over, said first communications channel and a second communications means being utilized over said second communications channel;

said first and second communications means being selectable from a group of communications means consisting of voice, facsimile, data, ASCII text, TTY-baudot, video or ASCII soft brail, said first communications means being selected independently from selection of said second communications means;

said first communications means being selectable to accommodate personal communications preferences of the first user;

said second communications means being selectable to accommodate personal communications preferences of the second user; and

a speech generation engine such that message data may be converted to a spoken language, said spoken language being routable via a selected combination of communications channel, communications mode, and communications means.

10. The system of claim 9, wherein said personal communications preferences further includes a generated speech table such that said spoken language being modifiable based on said generated speech table.

11. The system of claim 10, wherein said generated speech table further includes a gender selection such that the vocal qualities of said spoken language reflect the gender of the user.

12. The system of claim 10, wherein said generated speech table further includes an age group selection such that the vocal qualities of said spoken language reflect the age of the user.

13. The system of claim 10, wherein said generated speech table further comprises a nationality selection such that the vocal qualities of said spoken language reflect the nationality of the user.

14. The system of claim 10, wherein said generated speech table further includes a geographic region selection such that the vocal qualities of said spoken language reflect a geographic region selected by the user.

15. The system of claim 10, wherein said generated speech table further includes a tone selection such that the vocal qualities of said spoken language are modifiable to a tone selected by the user.

16. The system of claim 10, wherein said generated speech table further comprises:

a gender selection such that the vocal qualities of said spoken language reflect the gender of the user;

an age group selection such that the vocal qualities of said spoken language reflect the age of the user;

a nationality selection such that the vocal qualities of said spoken language reflect the nationality of the user, a geographic region selection such that the vocal qualities of said spoken language reflect a geographic region selected by the user, a tone selection such that the vocal qualities of said spoken language are modifiable to a tone selected by the user.

17. The system of claim 9, further comprising a sign language interpreter for reading sign language and translating said sign language to message data using a message format, said sign language interpreter being supplied with a video signal via said portal system.

18. The system of claim 17, wherein said sign language interpreter being a human being, said portal system routing a visual representation of a video signal from a user, said human being providing real time translation of sign language into a message format.

19. The system of claim 17 wherein said sign language interpreter being a sign language recognition engine, said sign language recognition engine converting a visual representation of sign language into message data, said sign language recognition engine being operationally coupled to said portal system.

20. The system of claim 17, further comprising a personal communications preferences including a visual speech table such that said sign language recognition engine being trainable for specific users.

21. A multi-modal network system for interpersonal communications comprising;

a portal system for transmitting and receiving communications between various communications modes;

a first communications channel providing connectivity between a list user and said portal system, said first communications channel being selected from the group of communications modes consisting of Public Switched Telephone Network, CATV with telephony, wireless telecommunications network, satellite communications network, PCS, GSM, Internet, World Wide Web, SONET, H320 video-conferencing (1ISDN), and H323 video-conferencing (IP);

a second communications channel providing connectivity between a second user and said portal system whereby said first user communications with said second user through said portal system, said second communications channel being selected from the group of communications modes consisting of Public Switched Telephone Network, CATV with telephony, wireless telecommunications network, satellite communications network, PCS, GSM, Internet, World Wide Web, SONET, H320 video-conferencing (ISDN), and H323 video-conferencing (IP);

said first communications channel being selected independently from selection of said second communications channel;

a data handling subsystem for identifying a first communication means being utilized over said first communications channel and a second communication means being utilized over said second communications channel;

said first and second communications means being selectable from a group of communications means consisting of voice, facsimile, data, ASCII text, TTY-baudot, video or ASCII soft brail, said first communication means being selected independently from selection of said second communication means, said first communications means being selectable to accommodate a personal communications preferences of the first user, said second communications means being selectable to accommodate communications preferences of the second user; and

a sign language generator for converting message data to sign language, said portal system providing said message data to said sign language generator.

22. The system of claim 21, wherein said sign language generator being a human being, said portal system routing a message data to said human being providing real time translation of said message data into sign language.

23. The system of claim 21, wherein said sign language generator being an avatar, said avatar providing an automated visual representation of sign language, said avatar being operationally coupled to said portal system such that said message data provides and input for said avatar.

24. The system of claim 23 further comprising a personal communications preference including a visual speech generation table such that a form of said avatar is modifiable in accordance with preferences of the user.

25. The system of claim 24, wherein said visual speech generation table further comprises a gender selection such that the visual qualities of the avatar correspond with the gender of the user.

26. The system of claim 24, wherein said visual speech generation table further comprises an age selection such that the visual qualities of the avatar correspond with the age of the user.

27. The system of claim 24, wherein said visual speech generation table further comprises a ethnic background selection such that the visual qualities of the avatar correspond with the ethnic background of the user.

28. The system of claim 24, wherein said visual speech generation table further comprises a general appearance selection such that the visual qualities of the avatar correspond with a general appearance selected by the user.

29. A multi-modal network system for interpersonal communications comprising;

a portal system for transmitting and receiving communications between various communications modes;

a first communications channel providing connectivity between a first time user and said portal system said first communications channel being selected from the group of communications modes consisting of Public Switched Telephone Network, CATV with telephony, wireless telecommunications network, satellite communications network, PCS, GSM, Internet, World Wide Web, SONET H320 video-conferencing (ISDN), and H323 video-conferencing (IP);

a second communications channel providing connectivity between a second user and said portal system whereby said first user communications with said second uses through said portal system, said second communications channel being selected from the group of communications modes consisting of Public Switched Telephone Network,. CATV, with telephony, wireless telecommunications network, satellite communications network, PCS, GSM, Internet, World Wide Web, SONET, H320 video-conferencing (ISDN), and H323 video-conferencing (IP);

said first video-conferencing channel being selected independently from selection of said second communications channel;

a data handling subsystem for identifying a first communications means being utilized over said first communications channel and a second communication means being utilized over said second communications channel;

said first and second communications means being selectable from a group of communications means consisting of voice facsimile, data, ASCII text, TTY-baudot, video or ASCII soft brail said first communication means being selected independently from selection of said second communication means;

said that communications means being selectable to accommodate personal communications preferences of the first user, said second communications means being selectable to accommodate personal communications preferences of the second user and;

a speech recognition engine operationally coupled to said portal system, said speech recognition engine facilitating translation of spoken speech into message data, said message data being routable via said portal system through a selected combination of communications channel, communications mode, and communications means, said personal communications preferences further includes a spoken speech table such that said speech recognition engine being trainable for specific users based upon said spoke speech table;

a speech generation engine such that message data may be converted to a spoken language, said spoken language being routable via a selected combination of communications channel, communications mode, and communications means;

said personal communications preferences further includes a generated speech table such that said spoken language being modifiable based on said generated speech table;

wherein said generated speech table further comprises a gender selection such that the vocal qualities of said spoken language reflect the gender of the user;

an age group selection such that the vocal qualities of said spoken language reflect the age of the user;

a nationality selection such that the vocal qualities of said spoken language reflect the nationality of the user;

a geographic region selection such that the vocal qualities of said spoken language reflect a geographic region selected by the user, a tone selection such that the vocal qualities of said spoken language are modifiable to a tone selected by the user, and a sign language interpreter for reading sign language and translating said sign language to message data using a message format, said sign language interpreter being supplied with a video signal via said portal system.

30. The system of claim 29, wherein said sign language interpreter being a human being, said portal system routing a visual representation of a video signal from a user, said human being providing real time translation of sign language into a message format.

31. The system of claim 29, wherein said sign language interpreter being a sign language recognition engine, said sign language recognition engine converting a visual representation of sign language into message data, said sign language recognition engine being operationally coupled to said portal system;

said personal communications preferences including a visual speech table such that said sign language recognition engine being trainable for specific users; and

a sign language generator for converting message data to sign language, said portal system providing said message data to said sign language generator.

32. The system claim 31, wherein said sign language generator being a human being said portal system routing a message data to said human being providing real time translation of said message data into sign language.

33. The system of claim 31, wherein said sign language generator being an avatar, said avatar providing an automated visual representation of sign language, said avatar being operationally coupled to said portal system such that said message data provides and input for said avatar, said personal communications preference including a visual speech generation table such that a form of said avatar is modifiable in accordance with preferences of the user, wherein said visual speech generation table further comprises:

a gender selection such that the visual qualities of the avatar correspond with the gender of the user;

an age selection such that the visual qualities of the avatar correspond with the age of the user;

an ethnic background selection such that the visual qualities of the avatar correspond with the ethnic background of the user; and

a general appearance selection such that the visual qualities of the avatar correspond with a general appearance selected by the user.

34. A multi-modal communications system for facilitating multiple translations comprising:

a portal system providing high level system control and routing;

a first communications mode providing connectivity between a first user and said portal system;

a second communications mode providing connectivity between a second user and said portal system whereby said first user communications with said second user through said portal system;

said first and second communications modes being independently selectable from the group of communications modes consisting of Public Switched Telephone Network, CATV with telephony, wireless telecommunications network, satellite communications network, PCS, GSM, Internet, World Wide Web, SONNET, and Integrated Services Digital Network (ISDN);

said portal system serving as a switching network and translator between said first and second communication modes;

said first communications mode being associated with a communications means selected from the group of communications means consisting of voice only, voice and video, video, text, or ASCII soft brail;

said second communications mode being associated with a communications means selected from the group of communications means consisting of voice only, voice and video, video, text, or ASCII soft brail;

a mode translator operationally interacting with said portal system, said mode translator facilitating translation between said first communication mode and said second communication mode;

said first user using a first language to communicate with said portal system;

said second user using a second language to communicate with said portal system;

a language translator operationally interacting with said portal system to translate between first language and message data, said language translator operationally interacting with said portal system to translate between said second language and message data.

35. The system of claim 34, wherein said language translator further comprises a human interpreter, said portal system having an interpreter interface to facilitate interaction between said human interpreter and said portal system, said human interpreter translating between a first spoken language and a second spoken language, said human interpreter translating between a spoken language and a visual language, said human interpreter translating between a first visual language and a second visual language, said human interpreter translating between a spoken language and a data format, said human interpreter translating between a visual language and a data format.

36. The system of claim 34, wherein said language translator further comprises a speech recognition engine operationally coupled to said portal system, said speech recognition engine facilitating translation between a spoken language and a visual language, said speech recognition engine facilitating translation between a spoken language and a data format, said speech recognition engine facilitates translation between a first spoken language and a second spoken language.

37. The system of claim 36, further comprising a personal communication preference including a spoken speech table such that said speech recognition engine is trainable for a specific user.

38. The system of claim 34, wherein said language translator further comprises a sign language recognition engine operationally coupled to said portal system, said sign language recognition engine facilitating translation between a visual language and a spoken language, said sign language recognition engine facilitating translation between a visual language and a data format, said sign language recognition engine facilitating translation between a first visual language and a second visual language.

39. The system of claim 38, further comprising a personal communication preference including a visual speech table such that said sign language recognition engine is trainable for a specific user.

40. The system of claim 34, wherein said language translator further comprises a speech generator operationally coupled to said portal system, said speech generator facilitating translation between a first spoken language and a second spoken language;

said speech generator facilitating translation between a spoken language and a visual language, said speech generator facilitating translation between a spoken language and a data format.

41. The system of claim 40, further comprising a personal communications preference including a speech font such that vocal qualities of said speech generator are modifiable by the user.

42. The system of claim 41 wherein said speech font includes parameters for specifying gender, age group, nationality, locality, and tone.

43. The system of claim 34, wherein said language translator further comprises a sign language generator, said sign language generator being operationally coupled to said portal system, said sign language generator facilitating translation between a first visual language and a second visual language said sign language generator facilitating translation between a visual language and a spoken language, said sign language generator facilitating translation between a visual language and a data format.

44. The system of claim 43, where said sign language generator presents an avatar having a torso, arms, hands, and a head, said avatar providing a visual representation of a visual language.

45. The system of claim 44, further comprising a personal communications preference including an avatar font such that visual qualities of said sign language generator are modifiable by the user.

46. The system of claim 45, wherein said avatar font includes parameters for specifying gender, age group, ethnicity, and general appearance.

47. The system of claim 34, wherein said language translator further comprises:

a speech recognition engine operationally coupled to said portal system, said speech recognition engine facilitating translation between a spoken language and a visual language, said speech recognition engine facilitating translation between a spoken language and a data format, said speech recognition engine facilitates translation between a first spoken language and a second spoken language;

a personal communication preference including a spoken speech table such that said speech recognition engine is trainable for a specific user, a sign language recognition engine operationally coupled to said portal system, said sign language recognition engine facilitating translation between a visual language and a spoken language, said sign language recognition engine facilitating translation between a visual language and a data format, said sign language recognition engine facilitating translation between a first visual language and a second visual language;

said personal communication preference further includes a visual speech table such that said sign language recognition engine is trainable for a specific user, a speech generator operationally coupled to said portal system, said speech generator facilitating translation between a first spoken language and a second spoken language, said speech generator facilitating translation between a spoken language and a visual language, said speech generator facilitating translation between a spoken language and a data format;

said personal communications preference further includes a speech font such that vocal qualities of said speech generator are modifiable by the user;

wherein said speech font includes parameters for specifying gender, age group, nationality, locality, and tone;

a sign language generator, said sign language generator being operationally coupled to said portal system, said sign language generator facilitating translation between a first visual language and a second visual language, said sign language generator facilitating translation between a visual language and a spoken language, said sign language generator facilitating translation between a visual language and a data format;

wherein said sign language generator presents an avatar having a torso, arms, hands, and a head, said avatar providing a visual representation of a visual language;

said personal communications preference further includes an avatar font such that visual qualities of said sign language generator are modifiable by the user; and

said avatar font includes parameters for specifying gender, age group, ethnicity, and general appearance.

48. The system of claim 47, wherein said portal system further comprises a storage means for storing said message data for delivery to a second user at a later time.