Conversational speech analysis method, and conversational speech analyzer

• US 8,036,898 B2
• Filed: 02/14/2007
• Issued: 10/11/2011
• Est. Priority Date: 02/14/2006
• Status: Active Grant

• Current Assignee: Hitachi, Ltd.
• Original Assignee: Hitachi, Ltd.
• Inventors: Sato, Nobuo, Obuchi, Yasunari
• Primary Examiner(s): SKED, MATTHEW J
• Application Number:
  US11/705,756
• Publication Number:
  US 20070192103A1
• Time in Patent Office: 1,700 Days
• Field of Search: None
• US Class Current: 704/270
• CPC Class Codes: G10L 17/26 Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices
• CPC Class Codes: G10L 25/78 Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10

Abstract

The invention provides a conversational speech analyzer which analyzes whether utterances in a meeting are of interest or concern. Frames are calculated using sound signals obtained from a microphone and a sensor, sensor signals are cut out for each frame, and by calculating the correlation between sensor signals for each frame, an interest level which represents the concern of an audience regarding utterances is calculated, and the meeting is analyzed.

Claims

1. A conversational speech analyzing system comprising:

a first microphone and a second microphone, each configured to capture speech data in an area where a meeting is being held;

a first sensor and a second sensor, each configured to capture sensor information in the area where the meeting is being held; and

a computer connected to the first and second microphones and the first and second sensors;

wherein the first microphone and the first sensor are connected to, or in proximity to, a first person, and the second microphone and the second sensor are connected to, or in proximity to, a second person;

wherein the computer is configured to store first speech data captured by the first microphone, second speech data captured by the second microphone, first sensor information captured by the first sensor, and second sensor information captured by the second sensor;

wherein the computer is configured to classify the first speech data captured from the first microphone as first speech frames when speech is detected, and as first nonspeech frames when speech is not detected;

wherein the computer is configured to divide the second sensor information based on the first speech frames and the first nonspeech frames, and

wherein the computer is configured to evaluate an interest level of the second person in the meeting by comparing characteristics of the second sensor information divided based on the first speech frames to characteristics of the second sensor information divided based on the first nonspeech frames.

2. The conversational speech analyzing system according to claim 1,

wherein the computer matches data pertaining to a speaker to at least one of first speech frames and second speech frames, and stores the result;

and

wherein the computer matches data pertaining to the speaker to an interest level in the meeting of a person in a vicinity, and stores the result.

3. The conversational speech analyzing system according to claim 1,

wherein the computer matches sensor information captured from at least one one of the first sensor and the second sensor with data pertaining to a person in a vicinity, and stores the result;

and

wherein the computer matches an interest level in the meeting with the person in a vicinity, and stores the result.

4. The conversational speech analyzing system according to claim 1,

wherein at least one of the first sensor and the second sensor is configured to perform sensing of a body state of a person in a vicinity of the meeting.

5. The conversational speech analyzing system according to claim 1,

wherein the computer extracts, at least at one detection occasion, data obtained by at least one of the first sensor and the second sensor;

and

wherein the computer extracts data from at least one of the first sensor information and the second sensor information pertaining to variation from a stated amount, and treats the variation information as feature information.

6. The conversational speech analyzing system according to claim 1,

wherein speech data with a high interest level in at least one of a first speech frame and a second speech frame is selected and outputted.

7. The conversational speech analyzing system of claim 1, wherein each of the first sensor and the second sensor comprises at least one of:

a load cell, configured to detect the movement of a person by the pressure force on a floor;

a chair weight sensor, configured to detect a bodyweight fluctuation;

an acceleration sensor, configured to be installed on clothes, spectacles or a name card, and configured to detect the movement of a person; and

a an acceleration sensor, configured to be installed on a bracelet, ring or pen, and configured to detect the movement of at least one of a hand and an arm.

8. A conversational speech analysis method in a conversational speech analyzing system having a first microphone, a second microphone, a first sensor, a second sensor, and a computer connected to the first microphone, the second microphone, the first sensor, and the second sensor, the method comprising:

a first step, including using the first microphone and the second microphone to capture speech data in a vicinity of a meeting, and storing the speech data in the memory of the computer;

a second step, including using the first sensor to capture first sensor information in the vicinity of the meeting, and using the second sensor to capture second sensor information in the vicinity of the meeting, and to store the first and second sensor information in the memory of a computer; and

a third step, including using the computer to classify the speech data captured from the first microphone as first speech frames when speech is detected, and to classify the speech data captured from the first microphone as first nonspeech frames when speech is not detected;

a fourth step, including using the computer to divide the first sensor information based on the first speech frames and the first nonspeech frames, and to divide the second sensor information also based on the first speech frames and the first nonspeech frames; and

a fifth step, including using the computer to evaluate an interest level of a person in the meeting by comparing characteristics of the second sensor information divided based on the first speech frames to characteristics of the second sensor information divided based on the first nonspeech frames.

9. The conversational speech analysis method according to claim 8,

wherein in the third step, the computer matches data pertaining to one of the persons in a vicinity of at least one of the first sensor and the second sensor with at least one of first sensor information captured from the first sensor and second sensor information captured from the second sensor, and stores the result;

and

wherein the computer matches an interest level in the meeting with the data pertaining to the person matched to at least one of first sensor information and second sensor information, and stores the result.

10. The conversational speech analysis method according to claim 8,

wherein, in the third step, the computer matches at least one of first sensor information captured from the first sensor and second sensor information captured from the second sensor, with data pertaining to one of the persons in a vicinity, and stores the result;

and

wherein, in the third step, the computer matches an interest level in the meeting with the data pertaining to the person matched to at least one of first sensor information and second sensor information, and stores the result.

11. The conversational speech analysis method according to claim 8,

wherein at least one of the first sensor and the second sensor performs sensing of a body state of a person in a vicinity of the meeting.

12. The conversational speech analysis method according to claim 8,

wherein, in the third step, the computer extracts, from at least at one detection occasion, data of at least one of the first sensor and the second sensor;

and

wherein, in the third step, the computer extracts, from at least one of the first sensor information and the second sensor information, data pertaining to variation from a stated amount, and treats the variation information as feature information.

13. The conversational speech analyzing system according to claim 8,

wherein in the third step, the computer selects and outputs speech data with a high interest level from at least one of a first speech frame and a second speech frame.

14. The conversational speech analysis method of claim 8, wherein each of the first sensor and the second sensor comprises at least one of:

a load cell, configured to detect the movement of a person by the pressure force on a floor;

a chair weight sensor, configured to detect a bodyweight fluctuation;

an acceleration sensor, configured to be installed on clothes, spectacles or a name card, and configured to detect the movement of a person; and

a an acceleration sensor, configured to be installed on a bracelet, ring or pen, and configured to detect the movement of at least one of a hand and an arm.

15. A conversational speech analyzing system comprising:

a first microphone and a second microphone, each configured to capture speech data in an area where a meeting is being held, the first microphone connected to, or in proximity to, a first person, and the second microphone connected to, or in proximity to, a second person;

a first sensor and a second sensor, each configured to capture sensor information in the area where the meeting is being held, the first sensor connected to, or in proximity to, a first person, and the second sensor connected to, or in proximity to, a second person; and

a computer, configured to;

connect to the first and second microphones and the first and second sensors,

store first speech data captured by the first microphone, second speech data captured by the second microphone, first sensor information captured by the first sensor, and second sensor information captured by the second sensor;

classify the first speech data captured from the first microphone as first speech frames when speech is detected, and as first nonspeech frames when speech is not detected,

divide the second sensor information based on the first speech frames and the first nonspeech frames, and

evaluate an interest level of the second person in the meeting by comparing characteristics of the second sensor information divided based on the first speech frames to characteristics of the second sensor information divided based on the first nonspeech frames.

16. The conversational speech analyzing system of claim 15, wherein each of the first sensor and the second sensor comprises at least one of:

a load cell, configured to detect the movement of a person by the pressure force on a floor;

a chair weight sensor, configured to detect a bodyweight fluctuation;

an acceleration sensor, configured to be installed on clothes, spectacles or a name card, and configured to detect the movement of a person; and

a an acceleration sensor, configured to be installed on a bracelet, ring or pen, and configured to detect the movement of at least one of a hand and an arm.