INFORMATION PROCESSING DEVICE, INFORMATION PROCESSING METHOD AND PROGRAM

• US 20110224978A1
• Filed: 03/01/2011
• Published: 09/15/2011
• Est. Priority Date: 03/11/2010
• Status: Abandoned Application

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: SAWADA, Tsutomu
• Application Number:
  US13/038,104
• Publication Number:
  US 20110224978A1
• Time in Patent Office: Days
• US Class Current: 704/231
• CPC Class Codes: G06F 18/25 Fusion techniques
• CPC Class Codes: G06V 10/80 Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level multimodal speaker identification or verification G10L17/10
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G10L 15/142 Hidden Markov Models [HMMs]
• CPC Class Codes: G10L 15/25 using position of the lips, movement of the lips or face analysis
• CPC Class Codes: G10L 15/32 Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems
• CPC Class Codes: G10L 2015/025 Phonemes, fenemes or fenones being the recognition units

Abstract

An information processing device includes an audio-based speech recognition processing unit which is input with audio information as observation information of a real space, executes an audio-based speech recognition process, thereby generating word information that is determined to have a high probability of being spoken, an image-based speech recognition processing unit which is input with image information as observation information of the real space, analyzes mouth movements of each user included in the input image, thereby generating mouth movement information, an audio-image-combined speech recognition score calculating unit which is input with the word information and the mouth movement information, executes a score setting process in which a mouth movement close to the word information is set with a high score, thereby executing a score setting process, and an information integration processing unit which is input with the score and executes a speaker specification process.

Claims

1. An information processing device comprising:

an audio-based speech recognition processing unit which is input with audio information as observation information of a real space, executes an audio-based speech recognition process, thereby generating word information that is determined to have a high probability of being spoken;

an image-based speech recognition processing unit which is input with image information as observation information of the real space, analyzes mouth movements of each user included in the input image, thereby generating mouth movement information in a unit of user;

an audio-image-combined speech recognition score calculating unit which is input with the word information from the audio-based speech recognition processing unit and input with the mouth movement information in a unit of user from the image-based speech recognition processing unit, executes a score setting process in which a mouth movement close to the word information is set with a high score, thereby executing a score setting process in a unit of user; and

an information integration processing unit which is input with the score and executes a speaker specification process based on the input score.

2. The information processing device according to claim 1,

wherein the audio-based speech recognition processing unit executes ASR (Audio Speech Recognition) that is an audio-based speech recognition process to generate a phoneme sequence of word information that is determined to have a high probability of being spoken as ASR information,

wherein the image-based speech recognition processing unit executes VSR (Visual Speech Recognition) that is an image-based speech recognition process to generate VSR information that includes at least viseme information indicating mouth shapes in a word speech period, and

wherein the audio-image-combined speech recognition score calculating unit compares the viseme information in a unit of user included in the VSR information with registered viseme information in a unit of phoneme constituting the word information included in the ASR information to execute a viseme score setting process in which a viseme with high similarity is set with a high score, and calculates an AVSR score which is a score corresponding to a user by the calculation process of an arithmetic mean value or a geometric mean value of a viseme score corresponding to all phonemes further constituting a word.

3. The information processing device according to claim 2, wherein the audio-image-combined speech recognition score calculating unit performs a viseme score setting process corresponding to periods of silence before and after the word information included in the ASR information, and calculates an AVSR score which is a score corresponding to a user by the calculation process of an arithmetic mean value or a geometric mean value of a score including a viseme score corresponding to all phonemes constituting a word and a viseme score corresponding to a period of silence.

4. The information processing device according to claim 2 or 3, wherein the audio-image-combined speech recognition score calculating unit uses a value of prior knowledge that is set in advance as a viseme score for a period when viseme information indicating shapes of the mouth of the word speech period is not input.

5. The information processing device according to any one of claims 1 to 4, wherein the information integration processing unit sets probability distribution data of a hypothesis on user information of the real space and executes a speaker specification process by updating and selecting a hypothesis based on the AVSR score.

6. The information processing device according to any one of claims 1 to 5, further comprising:

an audio event detecting unit which is input with audio information as observation information of the real space and generates audio event information including estimated location information and estimated identification information of a user existing in the real space; and

an image event detecting unit which is input with image information as observation information of the real space and generates image event information including estimated location information and estimated identification information of a user existing in the real space,

wherein the information integration processing unit sets probability distribution data of a hypothesis on location and identification information of a user and generates analysis information including location information of a user existing in the real space by updating and selecting a hypothesis based on the event information.

7. The information processing device according to claim 6, wherein the information integration processing unit is configured to generate analysis information including location information of a user existing in the real space by executing a particle filtering process to which a plurality of particles set with plural pieces of target data corresponding to virtual users are applied, and

wherein the information integration processing unit is configured to set each piece of the target data set in the particles in association with each event input from the audio and the image event detecting units and to update target data corresponding to the event selected from each particle according to an input event identifier.

8. The information processing device according to claim 7, wherein the information integration processing unit performs a process by associating each event in a unit of face image detected by the event detecting units.

9. An information processing method which is implemented in an information processing device comprising the steps of:

processing audio-based speech recognition in which an audio-based speech recognition processing unit is input with audio information as observation information of a real space, executes an audio-based speech recognition process, thereby generating word information that is determined to have a high probability of being spoken;

processing image-based speech recognition in which an image-based speech recognition processing unit is input with image information as observation information of a real space, analyzes mouth movements of each user included in the input image, thereby generating mouth movement information in a unit of user;

calculating an audio-image-combined speech recognition score in which an audio-image-combined speech recognition score calculating unit is input with the word information from the audio-based speech recognition processing unit and input with the mouth movement information in a unit of user from the image-based speech recognition processing unit, executes a score setting process in which a mouth movement close to the word information is set with a high score, and thereby executing a score setting process in a unit of user; and

processing information integration in which an information integration processing unit is input with the score and executes a speaker specification process based on the input score.

10. A program which causes an information processing device to execute an information process comprising the steps of:

processing audio-based speech recognition in which an audio-based speech recognition processing unit is input with audio information as observation information of a real space, executes an audio-based speech recognition process, thereby generating word information that is determined to have a high probability of being spoken;

processing image-based speech recognition in which an image-based speech recognition processing unit is input with image information as observation information of a real space, analyzes mouth movements of each user included in the input image, and thereby generating mouth movement information in a unit of user;

calculating an audio-image-combined speech recognition score in which an audio-image-combined speech recognition score calculating unit is input with the word information from the audio-based speech recognition processing unit and input with the mouth movement information in a unit of user from the image-based speech recognition processing unit, executes a score setting process in which a mouth movement close to the word information is set with a high score, thereby executing a score setting process in a unit of user; and

processing information integration in which an information integration processing unit is input with the score and executes a speaker specification process based on the input score.