Robotics visual and auditory system
First Claim
1. Robotics visual and auditory system comprising:
- an audition module including at least a pair of microphones for collecting external sounds;
a face module including a camera for taking images in front of a robot;
a stereo module for extracting a matter by a stereo camera;
a motor control module including a drive motor for rotating the robot in the horizontal direction;
an association module for generating streams by associating events from said audition, said face, said stereo, and said motor control modules; and
an attention control module for conducting attention control based on the stream generated by said association module;
characterized in that;
said audition module determines at least one speaker'"'"'s direction from the sound source separation and localization by grouping based on pitch extraction and harmonic wave structure, based on sound signal from the microphones, and extracts a auditory event;
said face module identifies each speaker from each speaker'"'"'s face recognition and localization based on the image taken by the camera, and extracts a face event;
said stereo module extracts a stereo event by extraction and localization of a longitudinally long matter based on a disparity extracted from the image taken by the stereo camera;
said motor control module extracts a motor event based on the rotational position of the drive motor; and
thereby said association module determines each speaker'"'"'s direction based on directional information of sound source localization by the auditory event, face localization by the face event, and matter localization by the stereo event from the auditory, face, stereo, and motor events, generates a auditory, a face, and a stereo streams by connecting the events in the temporal direction using a Kalman filter, and further generates a association stream by associating these;
said attention control module conducts attention control based on said streams, and drive-control of the motor based on a result of planning for the action accompanying those; and
said audition module collects sub-bands having interaural phase difference (IPD) or interaural intensity difference (IID) within a predetermined range by an active direction pass filter having a pass range which, according to auditory characteristics, becomes minimum in the frontal direction, and larger as the angle becomes wider to the left and right, based on an accurate sound source directional information from the association module, and conducts sound source separation by restructuring a wave shape of a sound source.
1 Assignment
0 Petitions
Accused Products
Abstract
Robotics visual and auditory system is provided which is made capable of accurately conducting the sound source localization of a target by associating a visual and an auditory information with respect to a target. It is provided with an audition module (20), a face module (30), a stereo module (37), a motor control module (40), an association module (50) for generating streams by associating events from said each module (20, 30, 37, and 40), and an attention control module (57) for conducting attention control based on the streams generated by the association module (50), and said association module (50) generates an auditory stream (55) and a visual stream (56) from a auditory event (28) from the auditory module (20), a face event (39) from the face module (30), a stereo event (39a) from the stereo module (37), and a motor event (48) from the motor control module (40), and an association stream (57) which associates said streams, as well as said audition module (20) collects sub-bands having the interaural phase difference (IPD) or the interaural intensity difference (IID) within the preset range by an active direction pass filter (23a) having a pass range which, according to auditory characteristics, becomes minimum in the frontal direction, and larger as the angle becomes wider to the left and right, based on an accurate sound source directional information from the association module (50), and conducts sound source separation by restructuring the wave shape of the sound source.
20 Citations
5 Claims
-
1. Robotics visual and auditory system comprising:
-
an audition module including at least a pair of microphones for collecting external sounds;
a face module including a camera for taking images in front of a robot;
a stereo module for extracting a matter by a stereo camera;
a motor control module including a drive motor for rotating the robot in the horizontal direction;
an association module for generating streams by associating events from said audition, said face, said stereo, and said motor control modules; and
an attention control module for conducting attention control based on the stream generated by said association module;
characterized in that;
said audition module determines at least one speaker'"'"'s direction from the sound source separation and localization by grouping based on pitch extraction and harmonic wave structure, based on sound signal from the microphones, and extracts a auditory event;
said face module identifies each speaker from each speaker'"'"'s face recognition and localization based on the image taken by the camera, and extracts a face event;
said stereo module extracts a stereo event by extraction and localization of a longitudinally long matter based on a disparity extracted from the image taken by the stereo camera;
said motor control module extracts a motor event based on the rotational position of the drive motor; and
therebysaid association module determines each speaker'"'"'s direction based on directional information of sound source localization by the auditory event, face localization by the face event, and matter localization by the stereo event from the auditory, face, stereo, and motor events, generates a auditory, a face, and a stereo streams by connecting the events in the temporal direction using a Kalman filter, and further generates a association stream by associating these;
said attention control module conducts attention control based on said streams, and drive-control of the motor based on a result of planning for the action accompanying those; and
said audition module collects sub-bands having interaural phase difference (IPD) or interaural intensity difference (IID) within a predetermined range by an active direction pass filter having a pass range which, according to auditory characteristics, becomes minimum in the frontal direction, and larger as the angle becomes wider to the left and right, based on an accurate sound source directional information from the association module, and conducts sound source separation by restructuring a wave shape of a sound source. - View Dependent Claims (2, 3, 4, 5)
-
Specification