Robot audiovisual system

• US 20040104702A1
• Filed: 08/29/2003
• Published: 06/03/2004
• Est. Priority Date: 03/09/2001
• Status: Active Grant

• Current Assignee: Japan Science & Technology Trading Co. Limited
• Original Assignee: Japan Science & Technology Trading Co. Limited
• Inventors: Kitano, Hiroaki, Okuno, Hiroshi, Nakadai, Kazuhiro, Hidai, Ken-ichi
• Granted Patent: US 6,967,455 B2
• Time in Patent Office: Days
• US Class Current: 318/568.12
• CPC Class Codes: B25J 13/00 Controls for manipulators programme controls B25J9/16; control in general G05
• CPC Class Codes: B25J 13/003 by means of an audio-responsive input audible safety signals B25J19/061
• CPC Class Codes: B25J 19/023 including video camera means
• CPC Class Codes: G06T 1/0014 Image feed-back for automatic industrial control, e.g. robot with camera robots B25J19/023
• CPC Class Codes: G10L 2021/02087 the noise being separate speech, e.g. cocktail party

Abstract

A robot visuoauditory system that makes it possible to process data in real time to track vision and audition for an object, that can integrate visual and auditory information on an object to permit the object to be kept tracked without fail and that makes it possible to process the information in real time to keep tracking the object both visually and auditorily and visualize the real-time processing is disclosed. In the system, the audition module (20) in response to sound signals from microphones extracts pitches therefrom, separate their sound sources from each other and locate sound sources such as to identify a sound source as at least one speaker, thereby extracting an auditory event (28) for each object speaker. The vision module (30) on the basis of an image taken by a camera identifies by face, and locate, each such speaker, thereby extracting a visual event (39) therefor. The motor control module (40) for turning the robot horizontally extracts a motor event (49) from a rotary position of the motor. The association module (60) for controlling these modules forms from the auditory, visual and motor control events an auditory stream (65) and a visual stream (66) and then associates these streams with each other to form an association stream (67). The attention control module (6) effects attention control designed to make a plan of the course in which to control the drive motor, e.g., upon locating the sound source for the auditory event and locating the face for the visual event, thereby determining the direction in which each speaker lies. The system also includes a display (27, 37, 48, 68) for displaying at least a portion of auditory, visual and motor information. The attention control module (64) servo-controls the robot on the basis of the association stream or streams.

Claims

1. What is claimed is:

2. A robot visuoauditory system as set forth in claim 1, characterized in that said association module when forming said auditory and visual streams is adapted to synchronize said auditory, visual and motor events which are formed asynchronously with each other.

3. A robot visuoauditory system as set forth in claim 1 or claim 2, characterized in that said association module is made a server and each of said audition, vision and motor control modules are made a client connected to said server.

4. A robot visuoauditory system as set forth in claim 3, characterized in that said server and each said client are interconnected via LAN.

5. A robot visuoauditory system as set forth in any one of claims 1 to 4, characterized in that when said attention control module effects the attention control, higher priority is given in the order of the association stream, the auditory stream and the visual stream.

6. A robot visuoauditory system as set forth in any one of claims 1 to 4, characterized in that said association module when forming said auditory and visual streams is adapted to synchronize said auditory, visual and motor events with each other.

24. A robot visuoauditory system as set forth in any one of claims 1 to 23, characterized in that said robot is a humanoid robot.

25. A robot visuoauditory system as set forth in any one of claims 1 to 23, characterized in that said robot is an animal type robot.

26. A robot visuoauditory system as set forth in any one of claims 1 to 23, characterized in that said robot is a plant type robot.

1-1. A visuoauditory system for a robot, characterized in that it comprises:

an audition module including at least a pair of microphones for collecting external sounds;

a vision module including a camera for taking an image in front of the robot;

a motor control module including a drive motor for turning the robot horizontally,

an association module for combining events from said audition module, said visual module and said motor control module to produce streams; and

an attention control module for effecting attention control on the basis of the streams produced by said association module, whereby;

said audition module in response to sound signals from said microphones is adapted to extract pitches therefrom, separate their sound sources from each other and locate sound sources such as to identify a sound source as at least one speaker, thereby extracting an auditory event;

said vision module on the basis of an image taken by the camera is adapted to identify by face, and locate, each such speaker, thereby extracting a visual event therefor;

said motor control module in response to a rotary position of said drive motor is adapted to extract a motor event therefrom;

said association module on the basis of the auditory event from said audition module, the visual event from said vision module and the motor event from said motor control module is adapted to form an auditory stream and a visual stream and then associate these streams with each other to form an association stream; and

said attention control module in effecting the attention control is adapted to make a plan for a course in which to control said drive motor in said motor control module in accordance with said streams.

7. A visuoauditory system for a robot, characterized in that it comprises:

an audition module including at least a pair of microphones for collecting external sounds;

a vision module including a camera for taking an image in front of the robot;

a motor control module including a drive motor for turning the robot horizontally,

an association module for combining events from said audition module, said visual module and said motor control module to produce streams; and

an attention control module for effecting attention control on the basis of the streams produced by said association module, whereby;

said audition module in response to sound signals from said microphones is adapted to extract pitches therefrom, separate their sound sources from each other and locate sound sources such as to identify a sound source as at least one speaker, thereby extracting an auditory event;

said vision module on the basis of an image taken by the camera is adapted to identify by face, and locate, each such speaker, and then to identify it as the sound source, thereby extracting a visual event therefor;

said motor control module in response to a rotary position of said drive motor is adapted to extract a motor event therefrom;

said association module on the basis of said auditory event, said visual event and said motor event is adapted to form an auditory stream and a visual stream upon determining a direction in which each such speaker lies on the basis of directional information derived from locating the sound source of said auditory event and that from locating the face of said visual event, and then to associate these streams with each other to form an association stream; and

said attention control module in effecting the attention control is adapted to make a plan for a course in which to control said drive motor in said motor control module in accordance with said streams.

8. A robot visuoauditory system as set forth in claim 7, characterized in that said association module when forming said auditory and visual streams is adapted to synchronize said auditory, visual and motor events which are formed asynchronously with each other.

9. A robot visuoauditory system as set forth in claim 7 or 8, characterized in that:

said audition module is adapted to identify each such speaker upon detecting a voice MFCC from the sound signals, and

said association module is adapted to select said auditory stream and said visual stream to which said auditory event and said visual event are to be connected, by specifying each such speaker on the basis of speaker identification for said auditory event and speaker identification for said visual event.

10. A robot visuoauditory system as set forth in any one of claims 7 to 9, characterized in that with a plurality of streams coming close to one another, said association module is adapted to select said auditory stream and said visual stream to which said auditory event and said visual event are to be connected, by referring to temporal flows of said auditory and visual events.

11. A robot visuoauditory system as set forth in any one of claims 7 to 10, characterized in that said association module is adapted to associate said auditory stream and said visual stream with each other to form said association stream if they are strongly associated with each other and to disassociate and extinguish said association stream when said auditory and visual streams which make up the same become weakly associated with each other.

12. A visuoauditory system for a robot, characterized in that it comprises:

an audition module including at least a pair of microphones for collecting external sounds;

a vision module including a camera for taking an image in front of the robot;

a motor control module including a drive motor for turning the robot horizontally,

an association module for combining events from said audition module, said visual module and said motor control module to produce streams; and

an attention control module for effecting attention control on the basis of the streams produced by said association module, whereby;

said audition module in response to sound signals from said microphones is adapted to extract pitches therefrom, separate their sound sources from each other and locate sound sources such as to identify a sound source as at least one speaker, thereby extracting an auditory event;

said vision module on the basis of an image taken by the camera is adapted to identify by face, and locate, each such speaker, thereby extracting a visual event therefor;

said motor control module in response to a rotary position of said drive motor is adapted to extract a motor event therefrom;

said association module on the basis of the auditory event, the visual event and the motor event is adapted to form an auditory stream and a visual stream and then associate these streams with each other to form an association stream; and

said attention control module in effecting the attention control is adapted to make a plan for a course in which to control said drive motor in said motor control module in accordance with said streams,

wherein said system further comprises a display for displaying at least a portion of auditory information for said audition module, visual information for said vision module, motor information for said motor control module and stream information for said association module.

13. A robot visuoauditory system as set forth in claim 12, characterized in that said display includes an auditory display for displaying as said auditory information, a spectrum of sound signals from sound sources and peaks extracted therefrom and said auditory event.

14. A robot visuoauditory system as set forth in claim 13, characterized in that said auditory display is adapted to display said auditory event in the form of a circle with its center representing the robot, its axis of ordinates representing relative angles of the direction that the robot may face, its axis of abscissas representing pitches and its diameter representing a belief factor.

15. A robot visuoauditory system as set forth in any one of claims 12 to 14, characterized in that said display includes a visual display for displaying as said visual information, an image of an extracted face taken by the camera and shown within a frame, and said visual event.

16. A robot visuoauditory system as set forth in claim 15, characterized in that said visual display is adapted to display said visual event in the form of list of face identifications and face locations extracted with degrees of firmness.

17. A robot visuoauditory system as set forth in any one of claims 12 to 16, characterized in that said display includes a motor display for displaying as said motor information, the direction in which and the speed at which the robot moves, three-dimensionally in real time.

18. A robot visuoauditory system as set forth in claim 17, characterized in that said display includes a stream display for displaying as said stream information, a stream chart and a radar chart.

19. A robot visuoauditory system as set forth in claim 18, characterized in that said stream display is adapted to display in said stream chart, said auditory, visual and association streams constituting said stream information.

20. A robot visuoauditory system as set forth in claim 18, characterized in that said stream display is adapted to display in said radar chart, an instantaneous status of said streams with a visual field of the camera and upon sound source location.

21. A visuoauditory system for a robot, characterized in that it comprises:

an audition module including at least a pair of microphones for collecting external sounds;

a vision module including a camera for taking an image in front of the robot;

a motor control module including a drive motor for turning the robot horizontally,

an association module for combining events from said audition module, said visual module and said motor control module to produce streams; and

an attention control module for effecting attention control on the basis of the streams produced by said association module, whereby;

said audition module in response to sound signals from said microphones is adapted to extract pitches therefrom, separate their sound sources from each other and locate sound sources such as to identify a sound source as at least one speaker, thereby extracting an auditory event;

said vision module on the basis of an image taken by the camera is adapted to identify by face, and locate, each such speaker, thereby extracting a visual event therefor;

said motor control module in response to a rotary position of said drive motor is adapted to extract a motor event therefrom;

said association module on the basis of the auditory event, the visual event and the motor event is adapted to form an auditory stream and a visual stream and then associate these streams with each other to form an association stream; and

said attention control module in effecting the attention control is adapted to make a plan for a course in which to control said drive motor in said motor control module in accordance with said streams, thereby effecting a visuoauditory servo for the robot.

22. A robot visuoauditory system as set forth in claim 21, characterized in that when said attention control module effects the attention control, higher priority is given in the order of the association stream, the auditory stream and the visual stream.

23. A robot visuoauditory system as set forth in claim 21 or claim 22, characterized in that said attention control module in the absence of more than one auditory or visual streams is adapted to select from them one auditory or visual stream according to circumstances, and to form an association stream, thereby effecting the attention control on the basis of such an auditory, visual or association stream.