SYNCHRONIZED GESTURE AND SPEECH PRODUCTION FOR HUMANOID ROBOTS

• US 20120191460A1
• Filed: 01/17/2012
• Published: 07/26/2012
• Est. Priority Date: 01/26/2011
• Status: Active Grant

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: Ng-Thow-Hing, Victor, Luo, Pengcheng
• Granted Patent: US 9,431,027 B2
• Time in Patent Office: Days
• US Class Current: 704/272
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 13/08 Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 21/10 Transforming into visible information

Abstract

A system or method for generating gestures in a robot during generation of a speech output by the robot by analyzing a speech text and selecting appropriate gestures from a plurality of candidate gestures. The speech text is analyzed and tagged with information relevant to generating of the gestures. Based on the speech text, the tagged information and other relevant information, a gesture identifier is selected. A gesture template corresponding to the gesture identifier is retrieved and then processed by adding relevant parameter to generate a gesture descriptor representing a gesture to be taken by the robot. A gesture motion is planned based on the gesture descriptor and analysis of timing associated with the speech. Actuator signals for controlling the actuators such as arms and hands are generated based on the planned gesture motion.

Claims

1. A computer-implemented method of generating a gesture in a robot, comprising:

receiving a text of a speech including one or more speech elements;

analyzing the speech text by a plurality of pattern modules to identify one or more candidate gestures associated with the speech elements in the speech, each of the plurality of pattern modules configured to apply a different set of rules to identify one or more corresponding candidate gestures;

selecting a gesture from the one or more candidate gestures identified by the plurality of pattern modules; and

generating the selected gesture by controlling actuators in the robot.

2. The method of claim 1, further comprising generating a voice output by synthesizing the speech text, the voice output synchronized with the gesture generated by the robot.

3. The method of claim 2, further comprising:

analyzing timing of the speech elements in the voice output; and

adjusting the selected gesture or the synthesized voice output to synchronize timing of the selected gesture and the timing of the speech elements in the voice output.

4. The method of claim 1, further comprising tagging information on the speech text by analyzing the speech text, wherein the one or more candidate gestures are identified by further analyzing the tagged information.

5. The method of claim 4, wherein the tagged information indicates types of words of the speech elements.

6. The method of claim 1, further comprising:

receive an expressivity parameter representing a degree of expressivity to be expressed by the robot; and

selecting the gesture based further on the expressivity parameter.

7. The method of claim 1, further comprising modifying at least one of amplitude, frequency and speed of the selected gesture based on a random number.

8. The method of claim 6, further comprising planning the selected gesture by adding a preparatory motion before making a motion corresponding to the selected gesture.

9. The method of claim 8, wherein the preparatory motion moves an effector from an end position of a prior gesture or a starting gesture to an initial position of the selected gesture.

10. The method of claim 1, wherein the plurality of pattern modules comprise a first pattern module configured to apply a first set of rules to detect emblems, a second pattern module configured to apply a second set of rules to detect iconics, a third pattern module configured to apply a third set of rules to detect metaphorics, a fourth pattern module configured to apply a fourth set of rules to detect deictics, and a fifth pattern module configured to apply a fifth set of rules to detect beats.

19. The robot of claim 1, wherein the plurality of pattern modules comprise a first pattern module configured to apply a first set of rules to detect emblems, a second pattern module configured to apply a second set of rules to detect iconics, a third pattern module configured to apply a third set of rules to detect metaphorics, a fourth pattern module configured to apply a fourth set of rules to detect deictics, and a fifth pattern module configured to apply a fifth set of rules to detect beats.

11. A robot configured to generate a gesture, comprising:

a gesture generator configured to;

receive a text of a speech including one or more speech elements;

analyze the speech text by a plurality of pattern modules to identify one or more candidate gestures associated with the speech elements in the speech, each of the plurality of pattern modules configured to apply a different set of rules to identify one or more corresponding candidate gestures; and

select a gesture from the one or more candidate gestures identified by the plurality of pattern modules;

a motion generator configured to generate control signals based on the selected gesture; and

at least one actuator configured to cause relative movements on parts of the robot according to the control signals.

12. The robot of claim 11, further comprising a voice synthesizer configured to generate a voice output by synthesizing the speech text, the voice output synchronized with the gesture generated by the robot.

13. The robot of claim 12, wherein the motion generator is further configured to analyze timing of the speech elements in the voice output and adjust the selected gesture to synchronize timing of the selected gesture and the timing of the speech elements in the voice output.

14. The robot of claim 11, wherein the gesture generator is further configured to tag information on the speech text by analyzing the speech text, wherein the one or more candidate gestures are identified by further analyzing the tagged information.

15. The robot of claim 14, wherein the tagged information indicates types of words of the speech elements.

16. The robot of claim 11, wherein the gesture generator is further configured to:

receive an expressivity parameter representing a degree of expressivity to be expressed by the robot; and

select the gesture based further on the expressivity parameter.

17. The robot of claim 11, wherein the motion generator is further configured to modify at least one of amplitude, frequency and speed of the selected gesture based on a random number.

18. The robot of claim 16, wherein the motion generator is further configured to plan the selected gesture by adding a preparatory motion before making a motion corresponding to the selected gesture.

20. A non-transitory computer readable storage medium for recognizing verbal commands, the computer readable storage medium structured to store instructions, when executed, cause a processor to:

receive a text of a speech including one or more speech elements;

analyze the speech text by a plurality of pattern modules to identify one or more candidate gestures associated with the speech elements in the speech, each of the plurality of pattern modules configured to apply a different set of rules to identify one or more corresponding candidate gestures;

select a gesture from the one or more candidate gestures identified by the plurality of pattern modules; and

generate the selected gesture by controlling actuators in a robot.