System and method for supporting interactive operations and storage medium

• US 20020010589A1
• Filed: 07/23/2001
• Published: 01/24/2002
• Est. Priority Date: 07/24/2000
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Nashida, Tatsushi, Ozaki, Naoto
• Granted Patent: US 7,426,467 B2
• Time in Patent Office: Days
• US Class Current: 704/275
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: H04L 12/2805 Home Audio Video Interoperability [HAVI] networks
• CPC Class Codes: H04L 12/282 based on user interaction within the home receiver circuitry for displaying additional information being controlled by a remote control apparatus H04N21/42204
• CPC Class Codes: H04N 21/41265 having a remote control device for bidirectional communication between the remote control device and client device unidirectional remote control device H04N21/42204
• CPC Class Codes: H04N 21/4147 PVR [Personal Video Recorder] H04N5/76 takes precedence ; arrangements for broadcast with accumulation-type receivers H04H20/40
• CPC Class Codes: H04N 21/42203 sound input device, e.g. microphone
• CPC Class Codes: H04N 21/4325 by playing back content from the storage medium reproduction of recorded television signals H04N5/76; reproduction of recorded television signals H04N9/79
• CPC Class Codes: H04N 21/43637 involving a wireless protocol, e.g. Bluetooth, RF or wireless LAN [IEEE 802.11] arrangements for wireless networking or broadcasting of information in indoor or near-field type systems H04B10/114
• CPC Class Codes: H04N 21/4622 Retrieving content or additional data from different sources, e.g. from a broadcast channel and the Internet web site content organization and management for information retrieval from the Internet G06F16/958; transmission by internet of broadcast information H04H60/82; stock exchange data over packet-switching network H04L12/1804; push services including data channel over packet-switching network H04L12/1859
• CPC Class Codes: H04N 21/4786 e-mailing message switching systems, e.g. electronic mail systems H04L51/00
• CPC Class Codes: H04N 21/4788 communicating with other users, e.g. chatting arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104
• CPC Class Codes: H04N 21/4828 for searching program descriptors retrieval of video data G06F16/739
• CPC Class Codes: H04N 21/6131 involving transmission via a mobile phone network wireless downlink channel access H04W74/006
• CPC Class Codes: H04N 21/8586 by using a URL processing chained hypermedia data for information retrieval G06F16/94; information retrieval from the Internet by using URLs G06F16/955; URL in broadcast information H04H20/93; Web-based protocols H04L67/02
• CPC Class Codes: H04N 7/163 by receiver means only

Abstract

There is provided a system for supporting interactive operations for inputting user commands to a household electric apparatus such as a television set/monitor and information apparatuses. According to the system for supporting interactive operations applying an animated character called a personified assistant interacting with a user based on speech synthesis and animation, realizing a user-friendly user interface and simultaneously making it possible to meet a demand for complex commands or providing an entry for services. Further, since the system is provided with a command system producing an effect close to human natural language, the user can easily operate the apparatus with a feeling close to ordinary human conversation.

Claims

1. An interactive operation supporting system for an apparatus, comprising a display unit, a speech input unit, a speech output unit and an operation control unit, wherein said operation control unit includes:

an assistant control means for generating a personified assistant and making said assistant appear on a screen of said display unit;

an output speech control means for determining speech required for said assistant to output said assistant'"'"'s speech to the outside through said speech output unit after speech synthesis;

an input speech recognition means for recognizing user'"'"'s voice as a speech inputted through said speech input unit;

an interaction management means for managing interaction between said assistant and said user according to said assistant'"'"'s speech determined by said output speech control means and said user speech recognized by said input speech recognition means; and

a command interpreting means for specifying a user'"'"'s intention or specifying said inputted user command based on a content of interaction traced by said interaction management means.

2. The interactive operation supporting system according to claim 1, wherein said assistant control means determines an animation of said assistant based on a content of interaction managed by said interaction management means and/or an inputted user command specified by said command interpreting means.

3. The interactive operation supporting system according to claim 1, wherein said output speech control means determines an assistant'"'"'s speech based on a content of interaction managed by said interaction management means and/or an inputted user command specified by said command interpreting means.

4. The interactive operation supporting system according to claim 1, wherein said output speech control means determines an assistant'"'"'s speech for leading a user'"'"'s intention when said command interpreting means fails to specify said user'"'"'s intention or said inputted user command.

5. The interactive operation supporting system according to claim 1, further comprising a connection means for connecting an external apparatus such as a television set/monitor and/or a video deck to said system, wherein

said command interpreting means also interprets a command for controlling a function of said external apparatus including selection of a broadcast program channel and/or recording/reproducing of said broadcast program.

6. The interactive operation supporting system according to claim 1, further comprising a communication means for connecting said system to a communication medium such as an external network or a telephone line, wherein said input speech recognition means recognizes audio data received via said communication medium.

7. The interactive operation supporting system according to claim 1, further comprising a communication means for connecting said system to a communication medium such as an external network and/or a telephone line, and a mail exchange means for making an exchange of electronic mails via said communication medium, wherein said output speech control means determines said assistant'"'"'s speech based on a content of an incoming mail.

8. The interactive operation supporting system according to claim 1, wherein said interaction management means manages an input speech of a user as a message bound for another user, and said output speech control means determines said assistant'"'"'s speech based on said message.

9. The interactive operation supporting system according to claim 1, wherein said assistant control means places said personified assistant in a room scattered with various kinds of objects including a link to an information resource.

10. The interactive operation supporting system according to claim 1, wherein said assistant control means places said personified assistant in a room scattered with various kinds of objects including a link to an information resource, and in response to an interest of said user in a recording medium including a link to a music content placed in said room, said command interpreting means interprets an inputted user command as a command for playing back said music content.

11. The interactive operation supporting system according to claim 1, wherein in response to said command interpreting means interpreting an inputted user command, said assistant control means makes said assistant appear on a screen of said display unit.

12. The interactive operation supporting system according to claim 1, further comprising a connection means for connecting a television set/monitor to said system, and in response to said command interpreting means interpreting an inputted user command as a channel selection command, said assistant control means makes said assistant show a selected broadcasting program display window.

13. The interactive operation supporting system according to claim 1, further comprising a connection means for connecting a television set/monitor to said system, wherein in response to said command interpreting means interpreting an inputted user command as a channel change command, said assistant control means placing a ring-shaped changeable broadcasting program display window around said assistant.

14. The interactive operation supporting system according to claim 13, wherein, in response to selection of a desired channel by moving said display window on said ring upon a channel change command from said user and said assistant control means also zooms up said selected broadcasting program display window.

15. The interactive operation supporting system according to claim 1, further comprising a connection means for connecting a secondary storage device for storing and/or reproducing a broadcasting program content to said system, wherein in response to said command interpreting means interpreting an inputted user command as a recorded program reproduction command, said assistant control means makes said assistant show a binder showing a view of recorded broadcasting program contents.

16. The interactive operation supporting system according to claim 15, wherein in response to a selection for reproducing a recorded broadcasting program content, said assistant control means zooms up said selected recorded broadcasting program content display window.

17. The interactive operation supporting system according to claim 1, further comprising a connection means for connecting a television set/monitor to said system, wherein in response to said command interpreting means interpreting an inputted user command as a channel change command, said assistant control means makes said assistant appear with a list of changeable broadcasting programs arranged in a matrix shape.

18. The interactive operation supporting system according to claim 17, wherein in response to a selection of a desired channel, said assistant control means zooms up said selected broadcasting program display window.

19. The interactive operation supporting system according to claim 17, wherein EPG distributed as part of data broadcast is applied for generating a list of broadcasting programs in a matrix form.

20. The interactive operation supporting system according to claim 1, further comprising a connection means for connecting a television set/monitor to said system, a communication means for connecting said system to a communication medium such as an external network and/or a telephone line, and a mail exchange means for making exchange of electronic mails via said communication medium, wherein said assistant control means makes an incoming mail display image to appear on a screen of said display unit in response to acceptance of mail.

21. The interactive operation supporting system according to claim 1, further comprising a text conversion means for converting a Japanese ideogram Kanji related to text data displayed on a screen of said display unit into a corresponding phonetic character Kana.

22. The interactive operation supporting system according to claim 1, further comprising a communication means for connecting said system to a communication medium such as an external network or a telephone line, and a certifying means for certifying an information terminal connected to said system via said communication medium.

23. The interactive operation supporting system according to claim 1, further comprising a connection means for connecting a television set/monitor to said system, and an extraction means for extracting text information from a received broadcasting program content.

24. The interactive operation supporting system according to claim 23, wherein text information extracted by said extraction means is superimposed on a content of another broadcasting program on said screen.

50. The interactive operation supporting system according to claim 1, further comprising a code conversion means for converting a character related to text data according to a group or system of characters or codes displayed on said display unit into a corresponding character of another group or system of characters or codes.

25. A method for supporting interactive operation of an apparatus or other externally connected apparatus including a display unit, a speech input unit and a speech output unit, comprising:

an assistant control step for generating a personified assistant and making said generated assistant appear on a screen of said display unit;

an output speech control step for determining a speech required for the assistant to output the assistant'"'"'s speech to the outside through said speech output unit after speech synthesis;

an input speech recognition step for recognizing a user'"'"'s voice as a speech inputted through said speech input unit;

an interaction management step for managing interaction between said assistant and said user according to said assistant'"'"'s speech determined by said output speech control step and said user speech recognized by said input speech recognition step; and

a command interpreting step for specifying said user'"'"'s intention or specifying said inputted user command based on a content of interaction traced by said interaction management step.

26. The method for supporting interactive operation according to claim 25, wherein said assistant control step determines an animation of said assistant based on a content of interaction managed by said interaction management step and/or an inputted user command specified by said command interpreting step.

27. The method for supporting interactive operation according to claim 25, wherein said output speech control step determines an assistant'"'"'s speech based on the content of interaction managed by said interaction management step and/or an inputted user command specified by said command interpreting step.

28. The method for supporting interactive operation according to claim 25, wherein said output speech control step determines an assistant'"'"'s speech for leading a user'"'"'s intention when said command interpreting step fails to specify said user'"'"'s intention or said inputted user command.

29. The method for supporting interactive operation according to claim 25, wherein said apparatus further includes a connection means for connecting an external apparatus such as a television set/monitor and/or a video deck to said apparatus, and said command interpreting step also interprets commands for controlling a function of said external apparatus including selection of a broadcast program and/or recording/reproducing of said broadcast program.

30. The method for supporting interactive operation according to claim 25, wherein said apparatus further includes a communication means for connecting said apparatus to a communication medium such as an external network and/or a telephone line, and said input speech recognition step recognizes audio data received via said communication medium.

31. The method for supporting interactive operation according to claim 25, wherein said apparatus further includes a communication means for connecting said apparatus to a communication medium such as an external network and/or a telephone line, and a mail exchange means for making an exchange of electronic mails via said communication medium, said output speech control step determining said assistant'"'"'s speech based on a content of an incoming mail.

32. The method for supporting interactive operation according to claim 25, wherein said interaction management step manages an input speech of a user as a message bound for another user, and said output speech control step determines said assistant'"'"'s speech based on said message.

33. The method for supporting interactive operation according to claim 25, wherein said assistant control step places said personified assistant in a room scattered with various kinds of objects including a link to an information resource.

34. The method for supporting interactive operation according to claim 25, wherein said assistant control step places said personified assistant in a room scattered with various kinds of objects including a link to an information resource, and in response an interest of said user in a recording medium including a link to a music content placed in said room, said command interpreting step interprets an inputted command as a command for playing back said music content.

35. The method for supporting interactive operation according to claim 25, wherein in response to said command interpreting step interpreting an inputted user command, said assistant control step makes said assistant appear on a screen of said display unit.

36. The method for supporting interactive operation according to claim 25, wherein said apparatus further includes a connection means for connecting a television set/monitor to said apparatus, and in response to said command interpreting step interpreting an inputted user command as a channel select command, said assistant control step makes said assistant show a selected broadcasting program display window.

37. The method for supporting interactive operation according to claim 25, wherein said apparatus further includes a connection means for connecting a television set/monitor to said apparatus, and in response to that said command interpreting step interpreting an inputted user command as a channel change command, said assistant control step places a changeable broadcasting program display window with a shape of a ring around said assistant.

38. The method for supporting interactive operation according to claim 37, wherein, in response to selection of a desired channel by moving said display window on said ring upon a channel change command from said user and said assistant control step also zooms up said selected broadcasting program display window.

39. The method for supporting interactive operation according to claim 25, wherein said apparatus further includes a connection means for connecting a secondary storage device for storing and/or reproducing a broadcasting program content to said apparatus, and in response to said command interpreting step interpreting an inputted user command as a recorded program reproduction command, said assistant control step makes said assistant show a binder showing a view of recorded broadcasting program contents.

40. The method for supporting interactive operation according to claim 39, wherein in response to a selection for reproducing a recorded broadcasting program content, said assistant control step zooms up said selected recorded broadcasting program content display window.

41. The method for supporting interactive operation according to claim 25, wherein said apparatus further includes a connection means for connecting a television set/monitor to said apparatus, and in response to said command interpreting step interpreting an inputted user command as a channel change command, said assistant control step makes said assistant appear with a list of changeable broadcasting programs arranged in a matrix shape.

42. The method for supporting interactive operation according to claim 41, wherein in response to a selection of a desired channel, said assistant control step zooms up said selected broadcasting program display window.

43. The method for supporting interactive operation according to claim 41, wherein EPG distributed as part of data broadcast is applied for generating a list of broadcasting programs in a matrix form.

44. The method for supporting interactive operation according to claim 25, wherein said apparatus further includes a connection means for connecting a television set/monitor to said apparatus, a communication means for connecting said apparatus to a communication medium such as an external network and/or a telephone line, and a mail exchange means for making exchange of electronic mails via said communication medium, and said assistant control step makes an incoming mail display image to appear on a screen of said display unit in response to acceptance of mail.

45. The method for supporting interactive operation according to claim 25, further comprising a text conversion step for converting a Japanese ideogram Kanji related to text data displayed on a screen of said display unit into a corresponding phonetic character Kana.

46. The method for supporting interactive operation according to claim 25, further comprising a communication step for connecting said apparatus to a communication medium such as an external network or a telephone line, and a certifying step for certifying an information terminal connected to said apparatus via said communication medium.

47. The method for supporting interactive operation according to claim 25, wherein said apparatus further includes a connection means for connecting a television set/monitor to said apparatus, and an extraction step for extracting text information from a received broadcasting program content.

48. The method for supporting interactive operation according to claim 47, wherein text information extracted by said extraction step is superimposed on a content of another broadcasting program on said screen.

51. The method for supporting interactive operation according to claim 25, further comprising a code conversion step for converting a character related to text data according to a group or system of characters or codes displayed on said display unit into a corresponding character of another group or system of characters or codes.

49. A storage medium storing a software program in computer-readable form, said software program describing an interactive operation support processing for execution on a computer system, said interactive operation support processing applied to an apparatus including a display unit, a speech input unit and a speech output unit for supporting input of a user command to said apparatus and/or another externally connected apparatus, wherein said software program includes:

an assistant control step for generating a personified assistant to make said generated assistant appear on a screen of said display unit;

an output speech control step for determining a speech required to said assistant for output of said assistant'"'"'s speech to the outside through said speech output unit after speech synthesis;

an input speech recognition step for recognizing a user'"'"'s voice inputted through said speech input unit as speech;

an interaction management step for managing interaction between said user and said assistant according to said assistant'"'"'s speech determined by said output speech control step and said user speech recognized by said input speech recognition step; and

a command interpreting step for specifying said user'"'"'s intention or said inputted user command based on a content of interaction traced by said interaction management step.