Global speech user interface

• US 20030078784A1
• Filed: 09/30/2002
• Published: 04/24/2003
• Est. Priority Date: 10/03/2001
• Status: Active Grant

• Current Assignee: Promptu Systems Corporation
• Original Assignee: Promptu Systems Corporation
• Inventors: Stanbach, Victoria, Williams, Jody, Jordan, Adam, Plowman, Tim, Maddux, Scott Lynn
• Granted Patent: US 7,324,947 B2
• Time in Patent Office: Days
• US Class Current: 704/275
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06Q 30/0271 Personalized advertisement
• CPC Class Codes: G06Q 30/0631 Item recommendations
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/221 Announcement of recognition results
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 21/06 Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids G10L15/26 takes precedence
• CPC Class Codes: H04N 21/42203 sound input device, e.g. microphone
• CPC Class Codes: H04N 21/4221 Dedicated function buttons, e.g. for the control of an EPG, subtitles, aspect ratio, picture-in-picture or teletext
• CPC Class Codes: H04N 21/4316 for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window
• CPC Class Codes: H04N 21/4622 Retrieving content or additional data from different sources, e.g. from a broadcast channel and the Internet web site content organization and management for information retrieval from the Internet G06F16/958; transmission by internet of broadcast information H04H60/82; stock exchange data over packet-switching network H04L12/1804; push services including data channel over packet-switching network H04L12/1859
• CPC Class Codes: H04N 21/47 End-user applications
• CPC Class Codes: H04N 21/472 End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content end-user interfaces for retrieving video data from a database G06F16/739; network services for supporting unicast streaming H04L65/612
• CPC Class Codes: H04N 21/47202 for requesting content on demand, e.g. video on demand
• CPC Class Codes: H04N 21/47211 for requesting pay-per-view content payment schemes payment architectures or payment protocols G06Q20/00, G07F
• CPC Class Codes: H04N 21/47214 for content reservation or setting reminders; for requesting event notification, e.g. of sport results or stock market stock exchange data over packet-switching network H04L12/1804; push services over packet-switching network H04L12/1859; notification of incoming messages in packet switching networks H04L51/224
• CPC Class Codes: H04N 21/475 End-user interface for inputting end-user data, e.g. personal identification number [PIN], preference data
• CPC Class Codes: H04N 21/478 Supplemental services, e.g. displaying phone caller identification, shopping application
• CPC Class Codes: H04N 21/4781 Games
• CPC Class Codes: H04N 21/4782 : Web browsing , e.g. WebTV
• CPC Class Codes: H04N 21/4788 : communicating with other users, e.g. chatting arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104
• CPC Class Codes: H04N 21/482 : End-user interface for program selection broadcast systems using EPGs H04H60/72
• CPC Class Codes: H04N 21/4826 : using recommendation lists, e.g. of programs or channels sorted out according to their score
• CPC Class Codes: H04N 21/4828 : for searching program descriptors retrieval of video data G06F16/739
• CPC Class Codes: H04N 21/4852 : for modifying audio parameters, e.g. switching between mono and stereo
• CPC Class Codes: H04N 21/812 : involving advertisement data advertising per se G06Q30/02
• CPC Class Codes: H04N 21/8173 : End-user applications, e.g. Web browser, game

Abstract

A global speech user interface (GSUI) comprises an input system to receive a user'"'"'s spoken command, a feedback system along with a set of feedback overlays to give the user information on the progress of his spoken requests, a set of visual cues on the television screen to help the user understand what he can say, a help system, and a model for navigation among applications. The interface is extensible to make it easy to add new applications.

Claims

1. A computer readable storage medium encoded with instructions, which when loaded into a communications system establishes a global speech user interface (GSUI), said GSUI comprising:

means for transcribing spoken commands into commands acceptable by said communications system;

means for navigating among applications hosted on said communications system; and

means for displaying a set of visual cues to help a user to give proper command.

2. The GSUI of claim 1, wherein said visual cues comprise:

a set of immediate speech feedback overlays, each of which provides simple, non-textual feedback information about a state of said communications system;

a set of help overlays, each of which provides a context-sensitive list of frequently used speech-activated commands for each screen of every speech-activated application;

a set of feedback overlays, each of which provides information about a problem that said communications system is experiencing; and

a main menu overlay that shows a list of services available to the user, each of said services being accessible by spoken command.

3. The GSUI of claim 2, further comprising a user center that provides any of:

training and tutorials on how to use said communications system;

more help with specific speech-activated applications;

user account management; and

user settings and preferences for said communications system.

4. The GSUI of claim 3, wherein each of said immediate speech feedback overlays provides simple, non-textual feedback information about a state of said communications system, said state being any of:

listening to the user'"'"'s spoken command;

non-speech enabled alert;

speech recognition processing;

application alert;

positive speech recognition; and

speech recognition unsuccessful.

5. The GSUI of claim 3, wherein each of said help overlays is accessible at all times.

6. The GSUI of claim 3, wherein said list of speech-activated commands provided by said help overlay comprises any of:

a set of application-specific commands;

a command associated with the user center for more help;

a command associated to said main menu display; and

a command to make said overlay disappear.

7. The GSUI of claim 3, wherein said set of feedback overlays comprises any of:

a set of recognition feedback overlays that informs the user of a situation related to recognition; and

a set of application overlays that informs the user of an error or a problem related to an application used in said GSUI.

8. The GSUI of claim 7, wherein said set of recognition feedback overlays, in responding to unsuccessful recognitions that immediately follow one another, is displayed in three different modes comprising:

a first mode wherein said immediate speech feedback indicator changes to a question mark in responding to the first unsuccessful recognition;

a second mode wherein a textual message and a link to said help overlay are displayed in responding to the second unsuccessful recognition; and

a third mode wherein a textual message, a link to said help overlay, and a link to said more help overlay are displayed in responding to the third and subsequent unsuccessful recognition.

9. The GSUI of claim 2, wherein said visual cues further comprises a treatment of on-screen text which can be activated by a spoken command.

10. The GSUI of claim 9, wherein said treatment is an overlay in round shape and green color.

11. The GSUI of claim 9, wherein said treatment can be turned on or off by the user.

12. The GSUI of claim 9, wherein said on-screen text comprises any of:

a static text used in labels for on-screen graphics or in virtual buttons that may be selected by a cursor; and

a dynamic text used in content wherein one or more words can be activated by a spoken command.

13. The GSUI of claim 2, wherein any of said help overlays, feedback overlays and main menu overlay is implemented in a dialog box, said dialog box comprising any of:

one or more text box for textual information; and

one or more virtual buttons.

14. The GSUI of claim 13, wherein said dialog box further comprises an identity indicator.

15. The GSUI of claim 14, wherein said dialog box has an approximately transparent background.

16. The GSUI of claim 14, wherein said dialog box has an opaque background.

17. The GSUI of claim 15, wherein said approximately transparent background is incorporated with a dynamic image to enhance said identity indicator.

18. The GSUI of claim 15, wherein said approximately transparent background is incorporated with a static image to enhance said identity indicator.

19. The GSUI of claim 15, wherein said text box is overlaid on said approximately background.

20. The GSUI of claim 2, wherein said main menu overlay comprises:

a first sub-menu overlay specifically for access to an interactive program guide system which provides cable television service;

a second sub-menu overlay specifically for access to a video on demand system which provides cable video service; and

a third sub-menu overlay specifically for access to a walled garden system which provides browser-based Internet service;

wherein each of said sub-menus provides a set of speech-activated virtual buttons.

21. The GSUI of claim 1, further comprising a speaker personalization and identification mechanism that allows a user to train said communications system with approximately forty seconds of speech and identifies the user by voice.

22. The GSUI of claim 21, wherein said speaker personalization and identification mechanism can be activated and disabled by said particular user'"'"'s command.

23. The GSUI of claim 22, wherein said speaker personalization and identification mechanism can be used to block any other user'"'"'s access to any application run on said communications system.

24. In a speech-enabled communications system for facilitating a digital information service, said communications system including television, a set top box, a speech input system, and a head-end, wherein a user activates said speech input system by activating a switch associated with operation of a speech input device, a method for providing a set of immediate speech feedback overlays to inform a user of said communications system'"'"'s states, said method comprising the steps of:

(a) checking if a current screen is speech-enabled when said switch is activated;

(b) if the current screen is speech-enabled, displaying a first tab signaling that a speech input system is activated;

(c) if the current screen is not speech-enabled, displaying a second tab signaling a non speech-enabled alert, said second tab staying on screen for a first interval; and

(d) if said switch is re-activated, repeating Step(a).

25. The method of claim 24, wherein said first tab includes a solid image of an identity indicator.

26. The method of claim 24, wherein said second tab comprises a prohibiting sign overlaid on said identity indicator.

27. The method of claim 26, wherein said second tab can further comprises a text box for textual message.

28. The method of claim 24, wherein said first interval in Step (c) is approximately ten seconds.

29. The method of claim 24, wherein said Step (b) further comprises the steps of:

(e) if said switch is not deactivated within a second interval, interrupting recognition;

(f) if said switch is deactivated after a third interval lapsed but before said second interval in Step (e) lapsed, displaying a third tab signaling that speech recognition is in processing; and

(g) if said switch was deactivated before said third interval in Step (f) lapsed, removing any tab on the screen.

30. The method of claim 29, wherein said second interval in Step (e) is approximately ten seconds and said third interval in Step (f) is approximately 0.1 second.

31. The method of claim 29, wherein said third tab is a flashing identity indicator which is approximately 40% transparent.

32. The method of claim 29, wherein said Step (f) further comprises the steps of:

(h) if said set top box takes longer than a fourth interval measured from the time that the user releases said switch to the time that the last speech data is sent to said head-end, interrupting speech recognition processing and displaying a fourth tab signaling an application alert, said fourth tab staying on the screen for a fifth interval; and

(i) if a remote control button other than said switch is pressed while a spoken command is being processed, interrupting speech recognition processing and removing any tab on the screen.

33. The method of claim 32, wherein said fourth interval is approximately five seconds and said fifth interval is approximately ten seconds.

34. The method of claim 32, wherein said fourth tab comprises an exclamation point overlaid on said identity indicator.

35. The method of claim 34, wherein said fourth tab can further comprises a text box for textual message.

36. The method of claim 32, wherein said Step (h) further comprises the steps of:

(j) if said switch is re-activated while said fourth tab on the screen, removing the fourth tab and repeating Step (a); and

(k) when said fifth interval lapses or if a remote control button other than said switch is activated while said fourth tab is on the screen, removing said fourth tab.

37. The method of claim 29, wherein said Step (f), upon a complete recognition, further comprises the steps of:

(l) checking whether the speech recognition is successful;

(m) if the speech recognition is successful, displaying a fifth tab signaling a positive speech recognition, said fifth tab staying on the screen for approximately one second; and

(n) if said switch is re-activated before said fifth tab disappears, repeating Step (a).

38. The method of claim 37, wherein said fifth tab comprises a check mark overlaid on said identity indicator.

39. The method of claim 29, wherein said Step (l) further comprises the steps of:

(o) if the speech recognition is unsuccessful, checking the number of unsuccessful recognitions which is automatically tracked by said communications system, said number being reset to zero after each successful recognition or when any button of said remote control device is pressed;

(p) if the complete recognition is the first unsuccessful recognition, displaying a sixth tab signaling a misrecognition speech, said sixth tab staying on the screen for about one second; and

(q) if said switch is repressed before said sixth tab disappears, repeating Step (a).

40. The method of claim 39, wherein said sixth tab in Step (p) is a question mark overlaid on said identity indicator.

41. The method of claim 39, wherein said Step (o) further comprises the steps of:

(r) if the complete recognition is the second unsuccessful recognition, displaying a first variant of said sixth tab signaling a misrecognition speech and displaying a short textual message, said first variant of said sixth tab staying on the screen for about ten seconds; and

(s) if said switch is repressed before said first variant of said sixth tab disappears, repeating Step (a).

42. The method of claim 41, wherein said first variant of said sixth tab comprises:

a question mark overlaid on said identity indicator; and

a short text box displaying a short textual message.

43. The method of claim 39, wherein said Step (o) further comprises the steps of:

(t) if the complete recognition is the third unsuccessful recognition, displaying a second variant of said sixth tab signaling a misrecognition speech and displaying a long textual message, said second variant of said sixth tab staying on the screen for about ten seconds; and

(u) if said switch is re-activated before said second variant of said sixth tab disappears, repeating Step (a).

44. The method of claim 29, wherein said Step (e) further comprises the steps of:

(v) displaying a first variant of said fourth tab, said first variant staying on the screen for a sixth interval;

(w) removing said first variant of said fourth tab from the screen if said switch is deactivated after said sixth interval lapsed; and

(x) displaying a second variant of said fourth tab, said second variant staying on the screen until said switch is deactivated.

45. The method of claim 44, wherein said first variant comprises an exclamation point and a first textual message.

46. The method of claim 44, wherein said sixth interval is approximately ten seconds.

47. The method of claim 44, wherein said second variant comprises an exclamation point and a second textual message.

48. In a speech-enabled communications system for facilitating a digital information service, said communications system including television, a set top box, a speech input system, and a head-end, wherein a user activates said speech input system by activating a switch associated with operation of a speech input device, a method for providing help information by displaying a set of overlays on the user'"'"'s screen, said method comprising the computer-implemented steps of:

(a) displaying a first help overlay if a help command is successfully recognized, said first help overlay staying on the screen for a specific interval;

(b) removing said first help overlay from the screen if any of the following occurs;

said specific interval lapses;

any button of said speech input device is accidentally activated; and

an exit button incorporated in said first help overlay is selected; and

(c) displaying a second help overlay while said switch is activated for inputting a new spoken command.

49. The method of claim 48, wherein said first help overlay is a dialog box which includes a first tab signaling a positive speech recognition, a text box for textual help information, and one or more virtual buttons.

50. The method of claim 49, wherein said first tab is a check mark overlaid on a non-highlighted identity indicator.

51. The method of claim 49, wherein said text box further includes a “

more help”

link.

52. The method of claim 49, wherein said text box includes one or more speech-activated words indicated by a speakable text indicator.

53. The method of claim 48, wherein said second help overlay is a dialog box which includes a second tab signaling said switch'"'"'s activation, a text box for textual help information, and one or more virtual buttons.

54. In a speech-enabled communications system for facilitating a digital information service, said communications system including television, a set top box, a speech input system, and a head-end, wherein a user activates said speech input system by activating a switch associated with operation of a speech input device, a method for providing a main menu by displaying a set of overlays on the user'"'"'s screen, said method comprising the computer-implemented steps of:

(a) displaying a first main menu overlay if the speech recognition is successful, said first main menu overlay staying on the screen for a specific interval;

(b) removing said first main menu overlay from the screen if any of the following occurs;

said specific interval lapses;

any button of said speech input device other than said switch is accidentally activated; and

an exit virtual button incorporated in said first main menu overlay is selected; and

(h) displaying a second main menu overlay while said switch is activated for inputting a new spoken command.

55. The method of claim 54, wherein said first main menu overlay is a dialog box which includes a first tab signaling a positive speech recognition, a text box for textual menu information, and one or more virtual buttons.

56. The method of claim 54, wherein said first tab is a check mark overlaid on a non-highlighted identity indicator.

57. The method of claim 54, wherein said text box includes one or more speech-activated words indicated by a speakable text indicator.

58. The method of claim 54, wherein said second main menu overlay is a dialog box which includes a second tab signaling said switch'"'"'s activation, a text box for textual menu information, and one or more virtual buttons.

59. A speech-enabled interactive television interfacing system, comprising:

an interconnection device which connects a television set with a television service provider;

a speech-enabled remote control device which transforms a user'"'"'s spoken commands into signals acceptable by said interconnection device; and

means for displaying a set of visual cues on a television screen to help the user give an operable commands.

60. The system of claim 59, wherein said interconnection device comprises a volume indicator, and wherein said speech-enabled remote control device comprises a push-to-talk button, said button being in the same color as said volume indicator and any on-screen graphic indicating speech-enabled user interface elements.

61. The system of claim 59, wherein said means for displaying provides immediate real-time visual feedback indicating various states of speech recognition activities.

62. The system of claim 61, said real-time visual feedback comprises a set of overlays, each of which provides simple, non-textual feedback information about a state of speech recognition activities, said state being any of:

receiving spoken utterance;

processing utterance;

successful recognition;

unsuccessful recognition; and

command not allowed.

63. The system of claim 59, wherein said visual cues provides escalating help feedback when the user'"'"'s spoken command is not recognized with a predefined degree of confidence.

64. The system of claim 63, wherein said escalating help feedback comprises a set of feedback overlays to reveal progressive help information.

65. The system of claim 64, wherein each of said feedback overlays provides a context-sensitive list of frequently used speech-enabled commands for each screen.

66. The system of claim 64, wherein each of said feedback overlays is accessible at all times.

67. The system of claim 65, wherein said list of frequently used speech-enabled commands comprises any of:

a set of application-specific commands;

a command associated with a user center for more help information;

a command associated with a main menu display; and

a command to make said overlay disappear from the screen.

69. The system of claim 59, wherein said means for displaying allows the user to initiate, via spoken command, an overlay display which indicates selectable user interface elements.

70. The system of claim 69, wherein said selectable user interface elements comprise any of:

numeric identifications;

navigation options; and

application control options.

71. The system of claim 59, wherein when the user'"'"'s spoken command is not recognized with a predefined degree of confidence, said means for displaying presents a list of predicted commands prompting the user to select from said list.

72. The system of claim 59, further comprises:

means for navigating on-screen list based information via spoken commands.

73. The system of claim 72, wherein said means for navigating enables the user to direct said on-screen list based information scroll up or scroll down by speaking a corresponding command.

74. The system of claim 72, wherein said means for navigating enables the user to select an item from said on-screen list based information by speaking a letter or a number identifying said item.

75. The system of claim 72, wherein said means for navigating enables the user to select an item from said on-screen list based information by speaking the name of said item.

76. The system of claim 59, further comprises:

means for allowing the user to navigate directly between applications via spoken command or a speech enabled menu.

77. The system of claim 59, further comprises:

means for allowing the user to navigate directly to previously book-marked pages via spoken command.

78. The system of claim 77, wherein said direct navigation to previously book-marked pages operates within and between applications.

79. A speech-enabled interactive television interfacing system, comprising:

an interconnection device which connects a television set with a television service provider;

a speech-enabled remote control device which transforms a user'"'"'s spoken commands into signals acceptable by said interconnection device; and

means for allowing the user to navigate television programs by spoken command.

80. The system of claim 79, further comprising:

means for allowing the user to initiate via spoken command an automatic scan search for television programs pursuant to a search category, wherein each matching program remains on screen for a short period of time before advancing to next matching program.

81. The system of claim 79, further comprising:

means for allowing the user to search, via spoken command, for particular television programs by specific attributes.

82. The system of claim 79, further comprising:

means for allowing the user to perform any of;

adding television programs to categories;

editing television programs in categories; and

deleting television programs from categories.

83. The system of claim 82, further comprising:

means for allowing the user to set parental control, with which children are blocked from accessing controlled television channels or television programs.

84. The system of claim 79, further comprising:

means for allowing the user to filter groups of television programs by specific attributes.

85. A speech-enabled interactive television interfacing system, comprising:

an interconnection device which connects a television set with a television service provider;

a speech-enabled remote control device which transforms a user'"'"'s spoken commands into signals acceptable by said interconnection device; and

an interactive program guide that the user can access via spoken command.

86. The system of claim 85, wherein said interactive program guide comprises:

means for allowing the user to, via spoken command, sort television programs by category.

87. The system of claim 86, wherein said interactive program guide comprises:

means for allowing the user to set parental controls, with which children are blocked from accessing controlled television channels or television programs.

88. The system of claim 85, wherein said interactive program guide comprises:

means for allowing the user to, via spoken command, set reminders for television programs to play in the future.

89. The system of claim 85, wherein said interactive program guide comprises:

means for allowing the user to, via spoken command, search television programs based on a specific criteria.

90. The system of claim 85, wherein said interactive program guide comprises:

means for processing pay per view purchases.

91. The system of claim 85, wherein said interactive program guide comprises:

means for allowing the user to, via spoken command, access and upgrade premium television services.

92. A speech-enabled interactive television interfacing system, comprising:

an interconnection device which connects a television set with a television service provider;

a speech-enabled remote control device which transforms a user'"'"'s spoken commands into signals acceptable by said interconnection device; and

an interactive video on demand service, from which the user can order any video program contained in a list.

93. The system of claim 92, wherein said video on demand service comprises:

means for allowing the user to, via spoken command, sort video programs by categories.

94. The system of claim 92, wherein said video on demand service comprises:

means for allowing the user to, via spoken command, search video programs by properties.

95. The system of claim 92, wherein said video on demand service comprises:

means for allowing the user to, via spoken command, set parental control with which children are blocked from accessing controlled video programs.

96. The system of claim 92, wherein said video on demand service comprises:

means for allowing the user to obtain automatic recommendation based on voiceprint identification.

97. A speech-enabled interactive television interfacing system, comprising:

an interconnection device which connects a television set with a television service provider;

a speech-enabled remote control device which transforms a user'"'"'s spoken commands into signals acceptable by said interconnection device; and

a speech enabled interface that allows the user to, via spoken command, conduct instant messaging communication.

98. A speech-enabled interactive television interfacing system, comprising:

an interconnection device which connects a television set with a television service provider;

a speech-enabled remote control device which transforms a user'"'"'s spoken commands into signals acceptable by said interconnection device; and

a speech enabled interface that allows the user to, via spoken command, activate links to television advertisement or banner advertisement contained in an application screen.

99. A speech-enabled interactive television interfacing system, comprising:

an interconnection device which connects a television set with a television service provider;

a speech-enabled remote control device which transforms a user'"'"'s spoken commands into signals acceptable by said interconnection device; and

means for targeting television advertisement or banner advertisement contained in an application screen to the user based on voiceprint identification.

100. A speech-enabled interactive television interfacing system, comprising:

an interconnection device which connects a television set with a television service provider;

a speech-enabled remote control device which transforms a user'"'"'s spoken commands into signals acceptable by said interconnection device; and

means for targeting television programming recommendations to the user based on voice identification.

101. A speech-enabled interactive television interfacing system, comprising:

an interconnection device which connects a television set with a television service provider;

a speech-enabled remote control device which transforms a user'"'"'s spoken commands into signals acceptable by said interconnection device; and

means for delivering personalized information to the user based on voice identification.

102. A speech-enabled interactive television interfacing system, comprising:

an interconnection device which connects a television set with a television service provider;

a speech-enabled remote control device which transforms the user'"'"'s spoken commands into signals acceptable by said interconnection device; and

means for automatically configuring the user'"'"'s interface preferences based on voiceprint identification.

103. A speech-enabled interactive television interfacing system, comprising:

an interconnection device which connects a television set with a television service provider;

a speech-enabled remote control device which transforms the user'"'"'s spoken commands into signals acceptable by said interconnection device; and

means for allowing the user to complete all aspects of a transaction via spoken commands.

104. A speech-enabled interactive television interfacing system, comprising:

an interconnection device which connects a television set with a television service provider;

a speech-enabled remote control device which transforms the user'"'"'s spoken commands into signals acceptable by said interconnection device; and

means for allowing the user to exercise central control, via spoken commands, over home services and devices.