Robot audiovisual system
First Claim
1. A visuoauditory system for a robot, characterized in that it comprises:
- an audition module including at least a pair of microphones for collecting external sounds;
a vision module including a camera for taking an image in front of the robot;
a motor control module including a drive motor for turning the robot horizontally;
an association module for combining events from said audition module, said visual module and said motor control module to produce streams; and
an attention control module for effecting attention control on the basis of the streams produced by said association module, whereby;
said audition module in response to sound signals from said microphones is adapted to extract pitches therefrom, separate their sound sources from each other and locate sound sources such as to identify a sound source as at least one speaker, thereby extracting an auditory event;
said vision module on the basis of an image taken by the camera is adapted to identify by face, and locate, each such speaker, thereby extracting a visual event therefor;
said motor control module in response to a rotary position of said drive motor is adapted to extract a motor event therefrom;
said association module on the basis of the auditory event from said audition module, the visual event from said vision module and the motor event from said motor control module is adapted to form an auditory stream and a visual stream and then associate these streams with each other to form an association stream; and
said attention control module in effecting the attention control is adapted to make a plan for a course in which to control said drive motor in said motor control module in accordance with said streams.
2 Assignments
0 Petitions
Accused Products
Abstract
A robot visuoauditory system that makes it possible to process data in real time to track vision and audition for an object, that can integrate visual and auditory information on an object to permit the object to be kept tracked without fail and that makes it possible to process the information in real time to keep tracking the object both visually and auditorily and visualize the real-time processing is disclosed. In the system, the audition module (20) in response to sound signals from microphones extracts pitches therefrom, separate their sound sources from each other and locate sound sources such as to identify a sound source as at least one speaker, thereby extracting an auditory event (28) for each object speaker. The vision module (30) on the basis of an image taken by a camera identifies by face, and locate, each such speaker, thereby extracting a visual event (39) therefor. The motor control module (40) for turning the robot horizontally. extracts a motor event (49) from a rotary position of the motor. The association module (60) for controlling these modules forms from the auditory, visual and motor control events an auditory stream (65) and a visual stream (66) and then associates these streams with each other to form an association stream (67). The attention control module (6) effects attention control designed to make a plan of the course in which to control the drive motor, e.g., upon locating the sound source for the auditory event and locating the face for the visual event, thereby determining the direction in which each speaker lies. The system also includes a display (27, 37, 48, 68) for displaying at least a portion of auditory, visual and motor information. The attention control module (64) servo-controls the robot on the basis of the association stream or streams.
-
Citations
26 Claims
-
1. A visuoauditory system for a robot, characterized in that it comprises:
- an audition module including at least a pair of microphones for collecting external sounds;
a vision module including a camera for taking an image in front of the robot;
a motor control module including a drive motor for turning the robot horizontally;
an association module for combining events from said audition module, said visual module and said motor control module to produce streams; and
an attention control module for effecting attention control on the basis of the streams produced by said association module, whereby;
said audition module in response to sound signals from said microphones is adapted to extract pitches therefrom, separate their sound sources from each other and locate sound sources such as to identify a sound source as at least one speaker, thereby extracting an auditory event;
said vision module on the basis of an image taken by the camera is adapted to identify by face, and locate, each such speaker, thereby extracting a visual event therefor;
said motor control module in response to a rotary position of said drive motor is adapted to extract a motor event therefrom;
said association module on the basis of the auditory event from said audition module, the visual event from said vision module and the motor event from said motor control module is adapted to form an auditory stream and a visual stream and then associate these streams with each other to form an association stream; and
said attention control module in effecting the attention control is adapted to make a plan for a course in which to control said drive motor in said motor control module in accordance with said streams. - View Dependent Claims (2, 3, 4, 5, 6, 24, 25, 26)
- an audition module including at least a pair of microphones for collecting external sounds;
-
7. A visuoauditory system for a robot, characterized in that it comprises:
-
an audition module including at least a pair of microphones for collecting external sounds;
a vision module including a camera for taking an image in front of the robot;
a motor control module including a drive motor for turning the robot horizontally;
an association module for combining events from said audition module, said visual module and said motor control module to produce streams; and
an attention control module for effecting attention control on the basis of the streams produced by said association module, whereby;
said audition module in response to sound signals from said microphones is adapted to extract pitches therefrom, separate their sound sources from each other and locate sound sources such as to identify a sound source as at least one speaker, thereby extracting an auditory event;
said vision module on the basis of an image taken by the camera is adapted to identify by face, and locate, each such speaker, and then to identify it as the sound source, thereby extracting a visual event therefor;
said motor control module in response to a rotary position of said drive motor is adapted to extract a motor event therefrom;
said association module on the basis of said auditory event, said visual event and said motor event is adapted to form an auditory stream and a visual stream upon determining a direction in which each such speaker lies on the basis of directional information derived from locating the sound source of said auditory event and that from locating the face of said visual event, and then to associate these streams with each other to form an association stream; and
said attention control module in effecting the attention control is adapted to make a plan for a course in which to control said drive motor in said motor control module in accordance with said streams. - View Dependent Claims (8, 9, 10, 11)
-
-
12. A visuoauditory system for a robot, characterized in that it comprises:
-
an audition module including at least a pair of microphones for collecting external sounds;
a vision module including a camera for taking an image in front of the robot;
a motor control module including a drive motor for turning the robot horizontally;
an association module for combining events from said audition module, said visual module and said motor control module to produce streams; and
an attention control module for effecting attention control on the basis of the streams produced by said association module, whereby;
said audition module in response to sound signals from said microphones is adapted to extract pitches therefrom, separate their sound sources from each other and locate sound sources such as to identify a sound source as at least one speaker, thereby extracting an auditory event;
said vision module on the basis of an image taken by the camera is adapted to identify by face, and locate, each such speaker, thereby extracting a visual event therefor;
said motor control module in response to a rotary position of said drive motor is adapted to extract a motor event therefrom;
said association module on the basis of the auditory event, the visual event and the motor event is adapted to form an auditory stream and a visual stream and then associate these streams with each other to form an association stream; and
said attention control module in effecting the attention control is adapted to make a plan for a course in which to control said drive motor in said motor control module in accordance with said streams;
wherein said system further comprises a display for displaying at least a portion of auditory information for said audition module, visual information for said vision module, motor information for said motor control module and stream information for said association module. - View Dependent Claims (13, 14, 15, 16, 17, 18, 19, 20)
-
-
21. A visuoauditory system for a robot, characterized in that it comprises:
-
an audition module including at least a pair of microphones for collecting external sounds;
a vision module including a camera for taking an image in front of the robot;
a motor control module including a drive motor for turning the robot horizontally;
an association module for combining events from said audition module, said visual module and said motor control module to produce streams; and
an attention control module for effecting attention control on the basis of the streams produced by said association module, whereby;
said audition module in response to sound signals from said microphones is adapted to extract pitches therefrom, separate their sound sources from each other and locate sound sources such as to identify a sound source as at least one speaker, thereby extracting an auditory event;
said vision module on the basis of an image taken by the camera is adapted to identify by face, and locate, each such speaker, thereby extracting a visual event therefor;
said motor control module in response to a rotary position of said drive motor is adapted to extract a motor event therefrom;
said association module-on the basis of the auditory event, the visual event and the motor event is adapted to form an auditory stream and a visual stream and then associate these streams with each other to form an association stream; and
said attention control module in effecting the attention control is adapted to make a plan for a course in which to control said drive motor in said motor control module in accordance with said streams, thereby effecting a visuoauditory servo for the robot. - View Dependent Claims (22, 23)
-
Specification