Robot acoustic device and robot acoustic system

• US 7,215,786 B2
• Filed: 06/08/2001
• Issued: 05/08/2007
• Est. Priority Date: 06/09/2000
• Status: Expired due to Fees

• Current Assignee: Japan Science & Technology Trading Co. Limited
• Original Assignee: Japan Science & Technology Trading Co. Limited
• Inventors: Kitano, Hiroaki, Okuno, Hiroshi, Nakadai, Kazuhiro
• Primary Examiner(s): Mei; Xu
• Application Number:
  US10/296,244
• Publication Number:
  US 20030139851A1
• Time in Patent Office: 2,160 Days
• Field of Search: 700/245, 700/246, 700/250, 381/71.1, 381/71.7, 381 941- 943, 381/94.8, 381/91, 381/92, 381/122, 901/50, 318/568.4, 318/568.12, 318/568.13, 31856816-56817
• US Class Current: 381/94.1
• CPC Class Codes: G10L 2021/02165 Two microphones, one receiving mainly the noise signal and the other one mainly the speech signal
• CPC Class Codes: G10L 21/0208 Noise filtering

Abstract

A robot auditory apparatus and system are disclosed which are made capable of attaining active perception upon collecting a sound from an external target with no influence received from noises generated interior of the robot such as those emitted from the robot driving elements. The apparatus and system are for a robot having a noise generating source in its interior, and include: a sound insulating cladding (14) with which at least a portion of the robot is covered; at least two outer microphones (16 and 16) disposed outside of the cladding (14) for collecting an external sound primarily; at least one inner microphone (17) disposed inside of the cladding (14) for primarily collecting noises from the noise generating source in the robot interior; a processing section (23, 24) responsive to signals from the outer and inner microphones (16 and 16; and 17) for canceling from respective sound signals from the outer microphones (16 and 16), noises signal from the interior noise generating source and then issuing a left and a right sound signal; and a directional information extracting section (27) responsive to the left and right sound signals from the processing section (23, 24) for determining the direction from which the external sound is emitted. The processing section (23, 24) is adapted to detect burst noises owing to the noise generating source from a signal from the at least one inner microphone (17) for removing signal portions from the sound signals for bands containing the burst noises.

Claims

1. A robot auditory apparatus for a robot having a noise generating source in its interior, characterized in that it comprises:

a sound insulating cladding with which at least a portion of the robot is covered;

at least two outer microphones disposed outside of said cladding for primarily collecting an external sound;

at least one inner microphone disposed inside of said cladding for primarily collecting noises from said noise generating source in the robot interior;

a processing section responsive to signals from said outer and inner microphones for canceling from respective sound signals from said outer microphones, noises signal from said interior noise generating source while detecting burst noises owing to said noise generating source from a signal from said at least one inner microphone for canceling signal portions from said sound signals for bands containing said burst noises; and

a directional information extracting section responsive to a left and a right sound signals from said processing section for determining a direction from which said external sound is emitted.

3. A robot auditory apparatus as set forth in claim 1 or claim 2, characterized in that said processing section is adapted to remove such signal portions as burst noises if a sound signal from said at least one inner microphone is enough larger in power than a corresponding sound signal from said outer microphones and further if peaks exceeding a predetermined level are detected over said bands in excess of a preselected level.

4. A robot auditory apparatus as set forth in claim 1 or claim 2, characterized in that said directional information extracting section is adapted to determine the direction from which said external sound is emitted by processing directional information of the sound in accordance with auditory epipolar geometry.

5. A robot auditory apparatus as set forth in claim 1 or claim 2, characterized in that said directional information extracting section is adapted to determine the direction from which said external sound is emitted by processing directional information of the sound in accordance with an auditory epipolar geometry based method and, if the sound has a harmonic structure, upon isolating the sound from another sound with the use of such a harmonic structure and by using information as to a difference in intensity between sound signals.

2. A robot auditory apparatus for a robot having a noise generating source in its interior, characterized in that it comprises:

a sound insulating cladding for self-recognition with which at least a portion of the robot is covered;

at least two outer microphones disposed outside of said cladding for primarily collecting an external sound;

at least one inner microphone disposed inside of said cladding for primarily collecting noises from said noise generating source in the robot interior;

a processing section responsive to signals from said outer and inner microphones for canceling from respective sound signals from said outer microphones, noise signals from said interior noise generating source while detecting burst noises owing to said noise generating source from a signal from said at least one inner microphone for canceling signal portions from said sound signals for bands containing said burst noises; and

a directional information extracting section responsive to a left and a right sound signals from said processing section for determining a direction from which said external sound is emitted.

6. A robot auditory system for a robot having a noise generating source in its interior, characterized in that it comprises:

a sound insulating cladding with which at least a portion of the robot is covered;

at least two outer microphones disposed outside of said cladding for collecting external sounds primarily;

at least one inner microphone disposed inside of said cladding for primarily collecting noises from said noise generating source in said robot interior;

a processing section responsive to signals from said outer and inner microphones for canceling from respective sound signals from said outer microphones, noise signals from said interior noise generating source while detecting burst noises owing to said noise generating source from a signal from said at least one inner microphone for canceling signal portions from said sound signals for bands containing said burst noises;

a pitch extracting section for effecting a frequency analysis on each of the a left and a right sound signals from said processing section to provide sound data as to time, frequency and power thereof from a pitch accompanied harmonic structure which the sound data signifies;

a left and right channel corresponding section responsive to left and right sound data from said pitch extracting section for providing respective sets of directional information determining directions from which the sounds are emitted, respectively; and

a sound source separating section for splitting said sound data into those sound data for respective sound sources of said sounds on the basis of such harmonic structures or said sets of directional information provided by said left and right channel corresponding section.

10. A robot auditory system as set forth in any one of claims 6 to 9, characterized in that said robot is further provided with one or more of other perceptual systems including vision and tactile systems furnishing an image of a sound source, and said left and right channel corresponding section is adapted to refer to image information from such system or systems as well to control signals for a drive means for moving the robot and thereby to determine the directions from which the sounds are emitted in coordinating the auditory information with the image and movement information.

11. A robot auditory system as set forth in any one of claims 6 to 9, characterized in:

that said robot is further provided with one or more of other perceptual systems including vision and tactile systems furnishing an image of a sound source, and said left and right channel corresponding section is adapted to refer to image information from such system or systems as well to control signals for a drive means for moving the robot and thereby to determine the directions from which the sounds are emitted in coordinating the auditory directional information with the image and movement information; and

that said left and right channel corresponding section is also adapted to furnish said other perceptual system or systems with the auditory directional information.

12. A robot auditory system as set forth in any one of claims 6 to 9, characterized in that said processing section is adapted to regard noises as the burst noises and remove signal portions for the bands containing those noises upon finding that a difference in intensity between the sound signals of said inner and outer microphones for said noises is close to an intensity in difference between those for template noises by robot drive means, that the spectral intensity and pattern of input sounds to said inner and outer microphone for said noises are close to those in a frequency response for the template noises by the robot drive means and further that the drive means is in operation.

13. A robot auditory system as set forth in any one of claims 6 to 9, characterized in:

that said robot is further provided with one or more of other perceptual systems including vision and tactile systems furnishing an image of a sound source, and said left and right channel corresponding section is adapted to refer to image information from such system or systems as well to control signals for a drive means for moving the robot and thereby to determine the directions from which the sounds are emitted in coordinating the auditory information with the image and movement information; and

that said processing section is adapted to regard noises as the burst noises and remove signal portions for the bands containing those noises upon finding that a difference in intensity between the sound signals of said inner and outer microphones for said noises is close to an intensity in difference between those for template noises by the robot drive means, that the spectral intensity and pattern of input sounds to said inner and outer microphone for said noises are close to those in a frequency response for the template noises by the robot drive means and that the drive means is in operation.

14. A robot auditory system as set forth in any one of claims 6 to 9, characterized in:

that said robot is further provided with one or more of other perceptual systems including vision and tactile systems furnishing an image of a sound source, and said left and right channel corresponding section is adapted to refer to image information from such system or systems as well to control signals for a drive means for moving the robot and thereby to determine the directions from which the sounds are emitted in coordinating the auditory information with the image and movement information;

that said left and right channel corresponding section is also adapted to furnish said other perceptual system or systems with the auditory directional information; and

that said processing section is adapted to regard noises as the burst noises and remove signal portions for the bands containing those noises upon finding that a difference in intensity between the sound signals of said inner and outer microphones for said noises is close to an intensity in difference between those for template noises by the robot drive means, that the spectral intensity and pattern of input sounds to said inner and outer microphone for said noises are close to those in a frequency response for the template noises by the robot drive means and that the drive means is in operation.

16. A robot auditory system as set forth in any one of claims 6 to 9, characterized in that said left and right channel corresponding section is adapted to derive said sets of directional information by computation in accordance with auditory epipolar geometry, thereby determining the directions from which said sounds are emitted, respectively.

17. A robot auditory system as set forth in any one of claims 6 to 9, characterized in:

that said robot is further provided with one or more of other perceptual systems including vision and tactile systems furnishing an image of a sound source, and said left and right channel corresponding section is adapted to refer to image information from such system or systems as well to control signals for a drive means for moving the robot and thereby to determine the directions from which the sounds are emitted in coordinating the auditory directional information with the image and movement information; and

that said left and right channel corresponding section is also adapted to derive said sets of directional information by computation in accordance with auditory epipolar geometry, thereby determining the directions from which said sounds are emitted, respectively.

18. A robot auditory system as set forth in any one of claims 6 to 9, characterized in:

that said robot is further provided with one or more of other perceptual systems including vision and tactile systems furnishing an image of a sound source, and said left and right channel corresponding section is adapted to refer to image information from such system or systems as well to control signals for a drive means for moving the robot and thereby to determine the directions from which the sounds are emitted in coordinating the auditory directional information with the image and movement information;

that said left and right channel corresponding section is also adapted to furnish said other perceptual system or systems with the auditory directional information; and

that said left and right channel corresponding section is further adapted to derive said sets of directional information by computation in accordance with auditory epipolar geometry, thereby determining the directions from which said sounds are emitted, respectively.

19. A robot auditory system as set forth in any one of claims 6 to 9, characterized in:

that said robot is further provided with one or more of other perceptual systems including vision and tactile systems furnishing an image of a sound source, and said left and right channel corresponding section is adapted to refer to image information from such system or systems as well to control signals for a drive means for moving the robot and thereby to determine the directions from which the sounds are emitted in coordinating the auditory directional information with the image and movement information;

that said left and right channel corresponding section is also adapted to furnish said other perceptual system or systems with the auditory directional information;

that said processing section is adapted to regard noises as the burst noises and remove signal portions for the bands containing those noises upon finding that a difference in intensity between the sound signals of said inner and outer microphones for said noises is close to an intensity in difference between those for template noises by robot drive means, that the spectral intensity and pattern of input sounds to said inner and outer microphone for said noises are close to those in a frequency response for the template noises by the robot drive means and that the drive means is in operation; and

that said left and right channel corresponding section is further adapted to derive said sets of directional information by computation in accordance with auditory epipolar geometry, thereby determining the directions from which said sounds are emitted, respectively.

21. A robot auditory system as set forth in any one of claims 6 to 9, characterized in that said left and right channel corresponding section sound direction by processing directional information of the sound in accordance with an auditory epipolar geometry based method and, if the sound has a harmonic structure, upon isolating the sound from another sound with the use of such a harmonic structure and by using information as to a difference in intensity between sound signals.

22. A robot auditory system as set forth in any one of claims 6 to 9, characterized in:

that said robot is further provided with one or more of other perceptual systems including vision and tactile systems furnishing an image of a sound source, and said left and right channel corresponding section is adapted to refer to image information from such system or systems as well to control signals for a drive means for moving the robot and thereby to determine the directions from which the sounds are emitted in coordinating the auditory directional information with the image and movement information; and

that said left and right channel corresponding section is adapted to determine the sound direction by processing directional information of the sound in accordance with an auditory epipolar geometry based method and, if the sound has a harmonic structure, upon isolating the sound from another sound with the use of such a harmonic structure and by using information as to a difference in intensity between sound signals.

23. A robot auditory system as set forth in any one of claims 6 to 9, characterized in:

that said robot is further provided with one or more of other perceptual systems including vision and tactile systems furnishing an image of a sound source, and said left and right channel corresponding section is adapted to refer to image information from such system or systems as well to control signals for a drive means for moving the robot and thereby to determine the directions from which the sounds are emitted in coordinating the auditory directional information with the image and movement information;

that said left and right channel corresponding section is adapted to furnish said other perceptual system or systems with the auditory directional information; and

that said left and right channel corresponding section is also adapted to determine the sound direction by processing directional information of the sound in accordance with an auditory epipolar geometry based method and, if the sound has a harmonic structure, upon isolating the sound from another sound with the use of such a harmonic structure and by using information as to a difference in intensity between sound signals.

24. A robot auditory system as set forth in any one of claims 6 to 9, characterized in:

that said robot is further provided with one or more of other perceptual systems including vision and tactile systems furnishing an image of a sound source, and said left and right channel corresponding section is adapted to refer to image information from such system or systems as well to control signals for a drive means for moving the robot and thereby to determine the directions from which the sounds are emitted in coordinating the auditory directional information with the image and movement information;

that said left and right channel corresponding section is also adapted to furnish said other perceptual system or systems with the auditory directional information;

that said processing section is adapted to regard noises as the burst noises and remove signal portions for the bands containing those noises upon finding that a difference in intensity between the sound signals of said inner and outer microphones for said noises is close to an intensity in difference between those for template noises by the robot drive means, that the spectral intensity and pattern of input sounds to said inner and outer microphone for said noises are close to those in a frequency response for the template noises by the robot drive means and that the drive means is in operation; and

that said left and right channel corresponding section is further adapted to determine the sound direction by processing directional information of the sound in accordance with an auditory epipolar geometry based method and, if the sound has a harmonic structure, upon isolating the sound from another sound with the use of such a harmonic structure and by using information as to a difference in intensity between sound signals.

7. A robot auditory system for a robot having a noise generating source in its interior, characterized in that it comprises:

a sound insulating cladding for self-recognition with which at least a portion of the robot is covered;

at least two outer microphones disposed outside of said cladding for collecting external sounds primarily;

at least one inner microphone disposed inside of said cladding for primarily collecting noises from said noise generating source in said robot interior;

a processing section responsive to signals from said outer and inner microphones for canceling from respective sound signals from said outer microphones, noise signals from said interior noise generating source while detecting burst noises owing to said noise generating source from a signal from said at least one inner microphone for canceling signal portions from said sound signals for bands containing said burst noises;

a pitch extracting section for effecting a frequency analysis on each of a left and a right sound signals from said processing section to provide sound data as to time, frequency and power thereof from a pitch accompanied harmonic structure which the sound data signifies;

a left and right channel corresponding section responsive to left and right sound data from said pitch extracting section for providing respective sets of directional information determining directions from which the sounds are emitted, respectively; and

a sound source separating section for splitting said sound data into those sound data for respective sound sources of said sounds on the basis of such harmonic structures or said sets of directional information provided by said left and right channel corresponding section.

8. A robot auditory system for a humanoid or animaloid robot having a noise generating source in its interior, characterized in that it comprises:

a sound insulating cladding with which at least a head portion of the robot is covered;

at least a pair of outer microphones disposed outside of said cladding and positioned thereon at a pair of ear corresponding areas, respectively, of the robot for collecting external sounds primarily;

at least one inner microphone disposed inside of said cladding for primarily collecting noises from said noise generating source in said robot interior;

a processing section responsive to signals from said outer and inner microphones for canceling from respective sound signals from said outer microphones, noise signals from said interior noise generating source while detecting burst noises owing to said noise generating source from a signal from said at least one inner microphone for canceling signal portions from said sound signals for bands containing said burst noises;

a pitch extracting section for effecting a frequency analysis on each of a left and a right sound signals from said processing section to provide sound data as to time, frequency and power thereof from a pitch accompanied harmonic structure which the sound data signifies;

a left and right channel corresponding section responsive to left and right sound data from said pitch extracting section for providing respective sets of directional information determining directions from which the sounds are emitted, respectively; and

a sound source separating section for splitting said sound data into those sound data for respective sound sources of said sounds on the basis of such harmonic structures or said sets of directional information provided by said left and right channel corresponding section.

15. A robot auditory system as set forth in claim 8 or claim 9, characterized in that said processing section is adapted to regard noises as the burst noises and remove signal portions for the bands containing those noises upon finding that the pattern of spectral power differences between the sound signals from said outer and inner microphones is substantially equal to a pattern of those measured in advance for noises by robot drive means, that the spectral sound pressures and their pattern are substantially equal to those in a frequency response measured in advance for noises by the drive means and that a control signal for the drive means indicates that the drive means is in operation.

20. A robot auditory system as set forth in claim 8 or claim 9, characterized in:

that said processing section is adapted to regard noises as the burst noises and remove signal portions for the bands containing those noises upon finding that the pattern of spectral power differences between the sound signals from said outer and inner microphones is substantially equal to a pattern of those measured in advance for noises by robot drive means, that the spectral sound pressures and their pattern are substantially equal to those in a frequency response measured in advance for noises by the drive means and that a control signal for the drive means indicates that the drive means is in operation; and

that said left and right channel corresponding section is adapted to derive said sets of directional information by computation in accordance with auditory epipolar geometry, thereby determining the directions from which said sounds are emitted, respectively.

25. A robot auditory system as set forth in claim 8 or claim 9, characterized in:

that said processing section is adapted to regard noises as the burst noises and remove signal portions for the bands containing those noises upon finding that the pattern of spectral power differences between the sound signals from said outer and inner microphones is substantially equal to a pattern of those measured in advance for noises by robot drive means, that the spectral sound pressures and their pattern are substantially equal to those in a frequency response measured in advance for noises by the drive means and that a control signal for the drive means indicates that the drive means is in operation; and

that said left and right channel corresponding section is further adapted to determine the directions from which the sounds are emitted by processing directional information of the sound in accordance with an auditory epipolar geometry based method and, if the sound has a harmonic structure, upon isolating the sound from another sound with the use of such a harmonic structure and by using information as to a difference in intensity between sound signals.

9. A robot auditory system for a humanoid or animaloid robot having a noise generating source in its interior, characterized in that it comprises:

a sound insulating cladding for self-recognition with which at least a head portion of the robot is covered;

at least a pair of outer microphones disposed outside of said cladding and positioned thereon at a pair of ear corresponding areas, respectively, of the robot for collecting external sounds primarily;

at least one inner microphone disposed inside of said cladding for primarily collecting noises from said noise generating source in said robot interior;

a processing section responsive to signals from said outer and inner microphones for canceling from respective sound signals from said outer microphones, noise signals from said interior noise generating source while detecting burst noises owing to said noise generating source from a signal from said at least one inner microphone for canceling signal portions from said sound signals for bands containing said burst noises;

a pitch extracting section for effecting a frequency analysis on each of a left and a right sound signals from said processing section to provide sound data as to time, frequency and power thereof from a pitch accompanied harmonic structure which the sound data signifies;

a left and right channel corresponding section responsive to left and right sound data from said pitch extracting section for providing respective sets of directional information determining directions from which the sounds are emitted, respectively; and

a sound source separating section for splitting said sound data into those sound data for respective sound sources of said sounds on the basis of such harmonic structures or said sets of directional information provided by said left and right channel corresponding section.