Editing support system including an interactive interface

• US 5,777,614 A
• Filed: 10/13/1995
• Issued: 07/07/1998
• Est. Priority Date: 10/14/1994
• Status: Expired due to Term

• Current Assignee: Rakuten, Inc. (Rakuten Group, Inc.)
• Original Assignee: Hitachi, Ltd.
• Inventors: Ando, Haru, Hataoka, Nobuo
• Primary Examiner(s): KATBAB, ABDOLLAH
• Application Number:
  US08/542,736
• Time in Patent Office: 998 Days
• Field of Search: 395/336, 395/337, 395/338, 345/333, 345/334, 345/336, 345/337, 345/339, 345/977, 345/978, 345/473
• US Class Current: 715/729
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 9/453 Help systems

Abstract

An editing system having a dialogue operation type interface directs a next operation by referring an operation history. User information is inputted by using speech input/output, pointing by a finger and 3-D CG. A human image representing the system is displayed as an agent on the image output device, and a user error, availability to a queue and a utilization environment are extracted by the system and informed to the user through the agent. The system responds to the user intent by the image display or the speech output by using the agent as a medium so that a user friendly interface for graphics edition and image edition is provided.

Claims

1. In an information processing apparatus utilizing speech comprising:

information display means for displaying information;

position information input means for user inputting continuous position information by a pointing gesture;

speech information input means for inputting speech information;

input information storing means for temporarily storing the position information and the speech information inputted by said input means;

language information storing means for storing an acoustic standard pattern representing vocal sound/syllable or word information and grammar information;

speech information storing means for storing content of speech to be used to output speech;

draw information storing means for storing draw or display information to be used for drawing;

speech information analysis means for comparing the speech inputted by said input means by using at least one of the acoustic standard pattern representing the vocal sound/syllable or the word information and the grammar information stored in said language information storing means;

speech output means for outputting speech information by speech information waveform stored in said speech information storing means;

image output means for synthesizing and outputting an image by using said draw image storing means; and

operation history storing means for storing history of user operations;

said information processing apparatus utilizing speech including an agent type interface comprising;

system utilization status extraction means for extracting a user input error, an availability status for a queue and a utilization status;

agent display control means for displaying an agent on a screen by said information display means and said image output means; and

speech content control means for displaying the operation of the agent in accordance with the information extracted by said system utilization status extraction means and speech content sounded by the agent by said speech output means in synchronism or asynchronism with the operation of the agent.

2. An information processing apparatus utilizing speech including an agent type interface according to claim 1 wherein said system utilization status extraction means includes an intent expression information slot for storing the word information extracted by said speech information analyzing means and the pointing information stored in said input information storing means and means for checking the store status of said intent expression information slot for each user input to determine the system utilization status, the user input form for attribute of the information is determined until the slot is fully filled up by the necessary information, and the user is guided by using the agent so that the user inputs the necessary information by the input form, and wherein when mis-recognition occurs and an error operation is conducted, only the error portion is inputted by speech so that the user can actively accommodate for the error.

3. An information processing apparatus using speech including an agent type interface according to claim 1 wherein said system utilization status extraction means has a function of detecting a non-input status in a series of user operations and detecting incompletion of the series of operations, and when the non-input status is detected, inquiring by the agent to the user to grasp the user status and conduct the user support in accordance with the status.

4. An information processing apparatus using speech including an agent type interface according to claim 1 wherein said agent display control means has a function of presenting information by using one or more of a 3-D graphics, a motion picture and a still image.

5. An information processing apparatus utilizing speech including an agent type interface according to claim 1 wherein each of said agent display control means and said agent speech content control means has a function of grasping the user status in accordance with the information extracted by said system utilization status extraction means to determine the type of the agent response to the user by the speech output and the image output, only the gesture by the image output or only the text display.

6. An information processing apparatus utilizing speech including an agent type interface according to claim 1 wherein each of said agent display control means and said agent speech content control means has a function of adjusting a volume level of the speech output in accordance with the content to be spoken and enlarging or reducing the agent display area and the gesture action range.

7. An information processing apparatus utilizing speech including an agent type interface according to claim 1 wherein said display and speech agent function has a help function for enabling explanation of an operation procedure of said information processing apparatus, a possible input and an operation content as required by a user request.

8. An information processing apparatus utilizing speech including an agent type interface according to claim 2 wherein said display and speech agent function includes an agent type interface having a guide function to enable guidance by the agent and necessary guidance for the process for the mis-recognition.

9. An information processing apparatus utilizing speech including an agent type interface according to claim 1 wherein said display and speech agent function has a guide function of enabling guidance to the user for any more efficient operation method than an operation method taken by the user for a task by using the user operation history stored in said operation history storing means.

10. An information processing apparatus utilizing speech including an agent type interface according to claim 1 wherein said display and speech agent function includes means for detecting vague expression of the user input information and has a function of determining, when the vague expression is detected, a candidate of the user intent by utilizing the user operation history stored in said operation history storing means and presenting it to the user to confirm the intent of the user.

11. An information processing apparatus utilizing speech including an agent type interface according to claim 1 wherein said display and speech agent function has a confirmation function of accessing to a database storing the information required by the user based on the result of the extraction of the user intent and searching the information, and when a plurality of solutions are included in each search step, inquiring to the user to extract a valid solution.

12. An information processing apparatus utilizing speech including an agent type interface according to claim 1 wherein said agent function includes input rate storing means for storing a user input rate and a function to determine the response content to the user based on the input rate stored in said input rate storing means and a learning function to conduct the agent display and speech output to the user in accordance with sid determination.

13. An information processing apparatus utilizing speech including an agent type interface according to claim 1 wherein said agent function includes eye tracking recognition means and has a learning function of determining whether the user eye line watches the agent or not, and when the user does not need the agent function, deleting the display by the agent and the sound by the speech.

14. An information processing apparatus utilizing speech including an agent type interface according to claim 1 wherein said agent function has a function of interrupting the response output when the user conducts other job request while the system sends a response, conducting the response to said job request first, and temporarily storing the content of the interrupted response output by said speech output content storing means.

15. An information processing apparatus utilizing speech including an agent type interface according to claim 1 wherein said agent function has a function of changing, when the mis-recognition by the system continuously occurs, the speech output manner and content of the agent and the look of the agent in accordance with the number of times of mis-recognition and the user status extracted by said system utilization status extraction means.

16. An information processing apparatus utilizing speech including an agent type interface according to claim 1 wherein each of said agent display control means and said agent speech content control means has a function of selecting the response means of the system in accordance with the user request.

17. An editing support system comprising:

display means for displaying an object to be edited and an edition result;

a plurality of types of input means for inputting edit commands for the object to be edited;

output means for outputting an edit status for the object to be edited;

control means for specifying an operation for the edit command and controlling to execute the operation;

storing means for storing operation history;

means for specifying a candidate of operation to be executed next by referring the operation history; and

means for presenting the content of the candidate by speech or an image,

wherein said operation history stores the content of past edit commands and input times of the edit commands, and when the same edit command continues over a predetermined number of times, said system stands by to repeat the same edit command without waiting for an external edit command, and outputs the stand-by state as the editing status.

18. An editing support system comprising:

display means for displaying an object to be edited and an edition result;

a plurality of types of input means for inputting edit commands for the object to be edited;

output means for outputting an edit status for the object to be edited;

control means for specifying an operation for the edit command and controlling to execute the operation;

storing means for storing operation history;

means for specifying a candidate of operation to be executed next by referring the operation history; and

means for presenting the content of the candidate by speech or an image,

wherein said operation history stores the content of past edit commands and the input times of the edit commands, and when a predetermined time elapses from the input time of the immediately previous edit command, a message is outputted to prompt the input of an edit command in accordance with the elapsed time.

19. An editing support system according to claim 18 wherein when a content to negate and correct the immediately previous edit command is inputted by said input means, a result of the execution of the content corrected for the object to be edited by the immediately previous edit command is outputted.

20. An editing support system according to claim 18 wherein said output means for outputting the editing status is a partial area displayed on said display means, and the image displayed in the partial area is changed in accordance with the editing status.