Display system capable of accepting user commands by use of voice and gesture inputs

• US 5,600,765 A
• Filed: 10/19/1993
• Issued: 02/04/1997
• Est. Priority Date: 10/20/1992
• Status: Expired due to Term

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Hitachi, Ltd.
• Inventors: Kitahara, Yoshinori, Ando, Haru
• Primary Examiner(s): Jankus, Almis R.
• Application Number:
  US08/137,788
• Time in Patent Office: 1,204 Days
• Field of Search: 395/161, 395/155-161, 395/133, 395/700, 395/153, 395/154, 395/118
• US Class Current: 345/668
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G09B 5/00 Electrically-operated educational appliances working with questions and answers G09B7/00; simulators G09B9/00; advertising or displaying in general G09F

Abstract

A method of accepting multimedia operation commands wherein, while pointing to either of a display object or a display position on a display screen of a graphics display system through a pointing input device, a user commands the graphics display system to cause an event on a graphics display, through a voice input device; comprising a first step of allowing the user to perform the pointing gesture so as to enter a string of coordinate points which surround one area for either of the display object and any desired display position; a second step of allowing the user to give the voice command together with the pointing gesture; a third step of recognizing a command content of the voice command by a speech recognition process in response to the voice command; a fourth step of recognizing a command content of the pointing gesture in accordance with the recognized result of the third step; and a fifth step of executing the event on the graphics display in accordance with the command contents of the voice command and the pointing gesture. Thus, the method provides a man-machine interface which utilizes the plurality of media of the voice and the pointing gesture, which offers a high operability to the user, and with which an illustration etc. can be easily edited.

Claims

1. A method of accepting multimedia operation commands wherein, while pointing to either of a display object and a display position on a display screen of a graphics display system through a pointing input device, a user commands the graphics display system to cause an event on a graphics display, through a voice input device;

comprising;

a first step of allowing the user to enter a string of coordinate points which surround one area for either of the display object or any desired display position by performing a pointing gesture along the string of coordinate points;

a second step of allowing said user to give a voice command together with said pointing gesture;

a third step of recognizing a command content of said voice command by a speech recognition process in response to said voice command;

a fourth step of recognizing a command content of said pointing gesture in accordance with a recognized result of said third step; and

a fifth step of executing the event on the graphics display in accordance with the command contents of said voice command and said pointing gesture.

2. A method of accepting multimedia operation commands as defined in claim 1, wherein in a case where it has been recognized on the basis of the voice command content at said third step that the entered string of coordinate points designate said display object as a designated subject to-be-handled, and where a plurality of display objects which share common areas with said area formed by said string of coordinate points are existent on the display screen, one of said display objects which has the largest common area with said area of said string of coordinate points is determined as said designated subject at said fourth step.

3. A method of accepting multimedia operation commands as defined in claim 2, wherein in a case where a plurality of display objects none of which share common areas with said area formed by said string of coordinate points are existent, one of said display objects which has a central point nearest a coordinate point which is indicated by a mean value of maximum and minimum X values of said string of coordinate points and that of maximum and minimum Y values thereof is determined as said designated subject at said fourth step.

4. A method of accepting multimedia operation commands as defined in claim 1, wherein in a case where it has been recognized on the basis of the voice command content at said third step that the entered string of coordinate points designate said display position, a coordinate point which is indicated by a mean value of maximum and minimum X values of said string of coordinate points and that of maximum and minimum Y values thereof is determined as the designated display position at said fourth step.

5. A method of accepting multimedia operation commands as defined in claim 1, wherein in a case where it has been recognized on the basis of the voice command content at said third step that the entered string of coordinate points designate a size of the specified display object on the display screen after scale-up/down of said specified display object, a ratio α

between a difference value of minimum and maximum values within X coordinate values of said entered string of coordinate points and a standard maximum length of said display object and a ratio β

between a difference value of minimum and maximum values within Y coordinate values and a standard maximum width of said display object are calculated at said fourth step, and sizes of said display object in X and Y directions thereof are respectively determined on the basis of the calculated ratios α and

β

at said fifth step.

6. A method of accepting multimedia operation commands as defined in claim 5, wherein only one of said ratios α

and β

is calculated at said fourth step, and it is shared for determining said sizes of said display object in both said X and Y directions thereof at said fifth step.

7. A method of accepting multimedia operation commands as defined in claim 1, wherein in a case where it has been recognized on the basis of the voice command content at said third step that the entered string of coordinate points designate a size of a display object which is to be newly entered, a ratio α

between a difference value of minimum and maximum values within X coordinate values of said entered string of coordinate points and a standard maximum length of said display object and a ratio β

between a difference value of minimum and maximum values within Y coordinate values and a standard maximum width of said display object are calculated at said fourth step, and sizes of said display object in X and Y directions thereof are respectively determined on the basis of the calculated ratios α and

β

at said fifth step.

8. A method of accepting multimedia operation commands as defined in claim 7, wherein only one of said ratios α

and β

is calculated at said fourth step, and it is shared for determining said sizes of said display object in both said X and Y directions thereof at said fifth step.

9. A method of accepting multimedia operation commands as defined in claim 1, wherein a plurality of strings of coordinate points are respectively entered through pointing gestures at said first step, a plurality of voice commands are respectively given together with said pointing gestures at said second step, command contents of said voice commands are respectively recognized at said third step, a time interval between input times of the respectively adjoining individual coordinate points which constitute the entered string of coordinate points is checked at said fourth step, thereby sensing a termination of one pointing gesture, and the sensed individual pointing gestures in a sequence of the sensing are respectively brought into correspondence with the individual voice command contents in a sequence of the recognition at said fifth step.

10. A method of accepting multimedia operation commands as defined in claim 1, wherein a part illustrations dictionary in which drawing data of parts required to be illustrated and names of the respective parts are stored is prepared before said first step, said user designates the input position on the display screen at said first step, and said user designates the name of any desired part in said part illustrations dictionary through said voice command at said second step, whereby said command contents of said voice command and said pointing gesture are respectively recognized at said third and fourth steps, and the designated part is displayed at the display position and with a size on said display screen as designated by said pointing gesture, at said fifth step.

11. A method of accepting multimedia operation commands as defined in claim 1, wherein the string of coordinate points specifies the size to which the display object is to be scaled.

12. A display system which is commanded by a user to cause an event concerning a display object on a graphics display, by the use of a voice command and a pointing gesture;

comprising;

pointing input means for entering a string of coordinate points which surround one area for either of the display object on the graphics display and a display position of said display object;

a pointing area table which stores therein said string of coordinate points entered by said pointing input means;

bit map data memory means for storing therein bit map data of various display parts that constitute said display object, and standard maximum widths and standard maximum lengths of said display parts;

a drawing table which stores therein identifiers of said display parts selected from within said bit map data memory means and displayed on said graphics display, widthwise and lengthwise scale-up/down ratios of said display parts relative to the standard maximum widths and lengths on said graphics display, and positional information of said display parts;

a display parts dictionary which holds therein speech-recognizable names of the individual display parts stored in said bit map data memory means;

voice command input means for entering the voice command of the user;

speech recognition means for recognizing said voice command entered by salad voice command input means, with reference to said display parts dictionary;

display parts extraction means for extracting said display parts on said graphics display as designated on the basis of said string of coordinate points in said pointing area table;

target point calculation means for calculating a target point designated on the basis of said string of coordinate points in said pointing area table;

scale-up/down ratio calculation means for calculating the widthwise and lengthwise scale-up/down ratio information of said display parts on the basis of said string of coordinate points in said pointing area table; and

control means for selectively activating at least one of said display parts extraction means, said target point calculation means and said scale-up/down ratio calculation means in accordance with a result of the speech recognition, and for rewriting said drawing table on the basis of a result of the activating.

13. A display system as defined in claim 12, wherein said voice command consists of a command content which expresses a sort of the event concerning said display object, and a command attribute which includes at least one of an object name, a size and said display position of said display object which are collateral with said command content;

and said control means selects and activates any of said display parts extraction means, said target point calculation means and said scale-up/down ratio calculation means in accordance with said command attribute as the result of the recognition of said voice command.

14. A display system as defined in claim 12, wherein in a case where a plurality of display objects which share common areas with said area formed by said string of coordinate points exist on a display screen of said display system, said display parts extraction means determines one of said display objects which has the largest common area with said area of said string of coordinate points, as the designated subject to-be-handled.

15. A display system as defined in claim 14, wherein in a case where a plurality of display objects none of which share common areas with said area formed by said string of coordinate points exist, said display parts extraction means determines one of said display objects which has a central point nearest a coordinate point which is indicated by a mean value of maximum and minimum X values of said string of coordinate points and that of maximum and minimum Y values thereof, as said designated subject.

16. A display system as defined in claim 12, wherein said target point calculation means determines a coordinate point which is indicated by a mean value of maximum and minimum X values of said string of coordinate points and that of maximum and minimum Y values thereof, as the designated display position.

17. A display system as defined in claim 12, wherein said scale-up/down ratio calculation means calculates a ratio α

between a difference value of minimum and maximum values within X coordinate values of the entered string of coordinate points and a standard maximum length of said display object and a ratio β

between a difference value of minimum and maximum values within Y coordinate values and a standard maximum width of said display object, and it determines sizes of said display object in X and Y directions thereof on the basis of the calculated ratios α and

β

, respectively.

18. A display system which is commanded by a user to cause an event concerning a display object on a graphics display, by the use of a voice command and a pointing gesture;

comprising;

pointing input means for entering a string of coordinate points which surround an area on the graphics display;

voice command input means for entering the voice command of the user;

speech recognition means for recognizing said voice command entered by said voice command input means;

execution means for executing the event on the graphics display in accordance with the string of coordinate points entered by the pointing input means and the voice command entered by the voice command input means; and

wherein a plurality of display objects share common areas with said area represented by the string of coordinate points, one of the display objects which has the largest common area with the area represented by the string of coordinate points is selected as the display object.