Image processing device using speech recognition to control a displayed object

• US 6,538,666 B1
• Filed: 12/10/1999
• Issued: 03/25/2003
• Est. Priority Date: 12/11/1998
• Status: Expired due to Term

• Current Assignee: Nintendo Company Limited
• Original Assignee: Nintendo Company Limited
• Inventors: Mitsunari, Koji, Ozawa, Muneaki, Nagareda, Takeshi
• Primary Examiner(s): {haeck over (S)}mits, Ta̅livaldis Ivars
• Application Number:
  US09/458,671
• Time in Patent Office: 1,201 Days
• Field of Search: 704/275, 345/716, 345/728
• US Class Current: 715/728
• CPC Class Codes: A63F 2300/1081 Input via voice recognition
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/221 Announcement of recognition results

Abstract

An image processing device which changes the way speech recognition results are processed as the program progresses. A video game machine body 10 causes a television receiver 30 to display given images and to output given sounds in accordance with a game program stored in a ROM cartridge 20. When a player enters a speech from a microphone 60, a speech recognition unit 50 recognizes a word corresponding to the speech and sends the result to the video game machine body 10. The video game machine body 10 causes the state of a dialogue partner object displayed on the television receiver 30 to change on the basis of the recognized result received from the speech recognition unit 50. The relation between the recognition result and the control of the displayed dialogue partner object is changed as the program progresses, which gives variety to the game and makes it more amusing.

Claims

1. An image processing device for varying action of a dialogue partner object displayed on a display device in response to a spoken word input from a user through a microphone, comprising:

a converter for converting an analog speech signal inputted from said microphone to digital speech data;

a speech recognizer for recognizing a word corresponding to the digital speech data converted by said converter;

a determiner for determining whether the word recognized by said speech recognizer matches a predefined word to be inputted at that time;

a first display control controller for, when said determiner determines match of words, controlling a displayed state of said dialogue partner object to cause said dialogue partner object to perform an action corresponding to the recognized word;

a second display controller for, when said determiner determines a mismatch of words, making a determination display on said display device to deliver information on the determination made by said determiner to the user; and

wherein said second display controller makes a display on said display device, as said determination display, to show that said dialogue partner object cannot understand the input word.

2. The image processing device according to claim 1, further comprising:

an input instructor for instructing to input speech; and

a controller for permitting speech input from said microphone while speech input is instructed by said input instructor.

3. The image processing device according to claim 2, wherein when speech input is not instructed by said input instructor over a given time period, said controller displays a message to prompt to instruct for speech input on said display device.

4. The image processing device according to claim 1, wherein when said determiner continuously determines a mismatch of words over a given time period, said second display controller further displays on said display device, as said determination display, a message containing a word to be inputted at that time.

5. The image processing device according to claim 1, wherein when said determiner repeatedly determines a mismatch of words for a given number of times, said second display controller further displays on said display device, as said determination display, a message containing a word to be inputted at that time.

6. The image processing device according to claim 4, wherein said second display controller controls the display on said display device so that the word to be inputted at that time and the remaining part of said message are displayed in different colors in said message.

7. The image processing device according to claim 5, wherein said second display controller controls the display on said display device so that the word to be inputted at that time and the remaining part of the message are displayed in different colors in said message.

8. A storage medium which contains program data executed in an image processing device for changing action of a dialogue partner object displayed on a display device in response to a spoken word inputted from a user through a microphone,

wherein when executing said program data, said image processing device

converts an analog speech signal inputted from said microphone to digital speech data,

recognizes a word corresponding to said digital speech data converted, and

determines whether said recognized word matches a word to be inputted at that time,

when match of words is determined, controls a displayed state of said dialogue partner object to cause said dialogue partner object to perform an action corresponding to the recognized word,

when mismatch of words is determined, makes a determination delivering display on said display device to deliver the result of the determination to the user;

and

wherein said second display controller makes a display on said display device, as said determination display, to show that said dialogue partner object cannot understand the input word.

9. An image processing device for displaying a given image on a display device according to a set program data and varying action of a dialogue partner object displayed on said display device in response to a spoken word input from a user through a microphone, comprising:

a converter for converting an analog speech signal inputted from said microphone to digital speech data;

a speech recognizer for recognizing a word corresponding to the digital speech data converted by said converter;

a display controller for controlling a displayed state of said dialogue partner object based on a result of recognition made by said speech recognizer; and

a degree of progress detector for detecting a degree of progress of said program data;

wherein said display controller changes, in steps, a way of controlling the displayed state of said dialogue partner object in accordance with the degree of progress of the program data detected by said degree of progress detector;

wherein said display controller comprises,

first display controller for causing said dialogue partner object to perform a predetermined action independently of the word recognized by said speech recognizer when the degree of progress of the program data detected by said degree of progress detector is at a relatively elementary level, and

a second display controller for causing said dialogue partner object to perform a corresponding action in accordance with the word recognized by said speech recognizer when the degree of progress of the program data detected by said degree of progress detector is at a relatively advanced level.

10. The image processing device according to claim 9, wherein said second display controller comprises,

a determiner for determining whether the word recognized by said speech recognizer matches a word to be inputted at that time, and

a corresponding action controller for, when said determiner determines match of words, causing said dialogue partner object to perform an action corresponding to the word determined as the match.

11. The image processing device according to claim 10, wherein said speech recognizer comprises;

a dictionary in which a plurality of pieces of word data are stored for reference, a correlation distance calculator for comparing said digital speech data and each piece of the word data stored in said dictionary to calculate a correlation distance indicating a degree of similarity for each piece of the word data,

a ranker for ranking the pieces of the word data stored in said dictionary in order of similarity, starting from the highest, on the basis of the correlation distances calculated by said correlation distance calculator, and

a candidate word data outputter for outputting, as candidate word data, the word

data of the highest rank to a given rank among the plurality of pieces of the word data stored in said dictionary to said determiner,

and wherein said determiner determines whether the candidate word data provided from said candidate word data outputter matches a word to be inputted at that time, in order starting with the candidate word data having the highest similarity, and stops the determination operation when a match is determined and gives a match determination output to said corresponding action controller.

12. The image processing device according to claim 11, wherein said determiner reduces the number of pieces of the word data to be selected from said candidate word data and subjected to the match determination as the degree of progress of the program data detected by said degree of progress detector advances.

13. The image processing device according to claim 10, wherein said speech recognizer comprises;

a dictionary in which word data to be inputted at that time is stored,

a correlation distance calculator for comparing said digital speech data and each piece of the word data stored in said dictionary to calculate a correlation distance showing a degree of similarity for each piece of the word data, and

a candidate word data outputter for selecting word data having the highest similarity on the basis of the correlation distances calculated by said correlation distance calculator and outputting the selected word data and its correlation distance as candidate word data to said determiner,

and wherein said determiner

detects whether a first similarity defined by the correlation distance contained in said candidate word data is higher than a second similarity defined by a preset threshold, and

when said first similarity is higher than said second similarity, determines that the word recognized by said speech recognizer matches a word to be inputted at that time, and

when said second similarity is higher than said first similarity, determines that the word recognized by said speech recognizer does not match a word to be inputted at that time.

14. A storage medium which contains program data executed in an image processing device for changing action of a dialogue partner object displayed on a display device in response to speech of a word inputted from a user through a microphone,

wherein when executing said program data, said image processing device

converts an analog speech signal inputted from said microphone to digital speech data,

recognizes a word corresponding to said digital speech data converted, and

controls a displayed state of said dialogue partner object on the basis of said recognized word,

and wherein a way of controlling the displayed state of said dialogue partner object is changed in steps in accordance with a degree of progress of said program data;

wherein said display controller comprises,

first display controller for causing said dialogue partner object to perform a predetermined action independently of the word recognized by said speech recognizer when the degree of progress of the program data detected by said degree of progress detector is at a relatively elementary level, and

a second display controller for causing said dialogue partner object to perform a corresponding action in accordance with the word recognized by said speech recognizer when the degree of progress of the program data detected by said degree of progress detector is at a relatively advanced level.