Robot capable of incorporating natural dialogues with a user into the behaviour of same, and methods of programming and using said robot

• US 10,052,769 B2
• Filed: 04/03/2013
• Issued: 08/21/2018
• Est. Priority Date: 04/04/2012
• Status: Active Grant

• Current Assignee: SoftBank Robotics Europe SAS
• Original Assignee: SoftBank Robotics Europe SAS
• Inventors: Houssin, David, Gate, Gwennael
• Primary Examiner(s): Amin, Bhavesh V
• Application Number:
  US14/390,746
• Publication Number:
  US 20150100157A1
• Time in Patent Office: 1,966 Days
• Field of Search: 700245-264
• CPC Class Codes: B25J 11/0005 Manipulators having means for high-level communication with users, e.g. speech generator, face recognition means
• CPC Class Codes: B25J 11/001 with emotions simulating means
• CPC Class Codes: B25J 9/1694 characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06N 3/008 based on physical entities controlled by simulated intelligence so as to replicate intelligent life forms, e.g. based on robots replicating pets or humans in their appearance or behaviour
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/19 Grammatical context, e.g. disambiguation of the recognition hypotheses based on word sequence rules
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: Y10S 901/46 Sensing device
• CPC Class Codes: Y10S 901/47 Optical

Abstract

A humanoid robot is provided, the robot being capable of holding a dialog with at least one user, the dialog using two modes of voice recognition, one open and the other closed, the closed mode being defined by a concept characterizing a dialog sequence. The dialog may also be influenced by events that are neither speech nor a text. The robot is capable of executing behaviors and generating expressions and emotions. It has the advantage of considerably reducing programming time and latency of execution of dialog sequences, providing a fluency and naturalness close to human dialogs.

Claims

1. A humanoid robot, comprising:

i) at least one sensor selected from a group comprising audio sensors and visual sensors of events generated by at least one user of said robot,

ii) at least one event recognition module at the output of said at least one sensor and,

iii) at least one module for generating events towards said at least one user, a module for dialog with said at least one user, said dialog module receiving as input the outputs of said at least one recognition module and producing outputs to said event generation module selected from a group including speech, movements, expressions and emotions, wherein said robot further includes an artificial intelligence engine configured for controlling the outputs of the event generation module according to a context of dialog and variables defining emotions of the robot as a series of expressions between a neutral state and a terminal state, an evolution between said neutral state and said terminal state being triggered by a pulse generated as a function of time and of one or more of the outputs of the at least a sensor, the event recognition module, the dialog module and the event generation module.

2. The humanoid robot of claim 1,

wherein said at least one event recognition module receives inputs originating from at least two sensors belonging to at least two different types, and said at least one event generation module at the output of said dialog module is capable of producing as outputs events taking into account said inputs originating from said at least two sensors.

3. The humanoid robot of claim 1,

wherein said at least one recognition module is capable of structuring the inputs into concepts according to a dynamic hierarchical tree.

4. The humanoid robot of claim 1,

wherein an input in said at least one recognition module applies to text or voice inputs and activates a grammar in said dialog module.

5. The humanoid robot of claim 4,

wherein an input in said at least one recognition module activates/deactivates recognition of said input.

6. The humanoid robot of claim 4,

wherein said at least one recognition module includes a first and a second submodules, the first submodule operating on a closed list of words linked to at least one concept and the second submodule operating on an open list of words.

7. The humanoid robot of claim 6,

wherein only one output from the first submodule is supplied to the dialog module.

8. The humanoid robot of claim 6,

wherein only an output from the second submodule is supplied to the dialog module.

9. The humanoid robot of claim 6,

wherein an output from the first submodule and an output from the second submodule are supplied together to the dialog module.

10. The humanoid robot of claim 6,

wherein only an output from the first submodule is first supplied to the dialog module, said output from the first submodule being confirmed in the dialog module by an output from the second submodule.

11. The humanoid robot of claim 6,

wherein none of the outputs from the first and second sub-modules generates output from the dialog module and said robot provides at least one output to said at least one user.

12. The humanoid robot of claim 1,

wherein the dialog module further receives as input dynamic elements originating from an application.

13. The humanoid robot of claim 1,

wherein at least one output from the dialog module is supplied to a module capable of performing a function selected from a group of functions including a function for generating at least one expression of said robot, a function for deciding to generate at least one behavior of said robot, and a function for generating at least one emotion of said robot.

14. The humanoid robot of claim 13,

wherein said function selected from the group of functions is said function for generating at least one behavior, and

wherein said function for generating at least one behavior takes into account the system constraints of said robot.

15. The humanoid robot of claim 13,

wherein said function selected from the group of functions is said function for generating at least one emotion, and

wherein said function for generating at least one emotion is capable of generating a series of predefined expressions between a neutral state and a predefined state in response to input events.

16. The humanoid robot of claim 1,

further comprising a visual recognition module,

wherein said visual recognition module is capable of interpreting at least one sign from said at least one user as a beginning or an end of a dialog sequence.

17. The humanoid robot of claim 3,

wherein said dialog module includes a submodule for lexical analysis and a submodule for interpreting the outputs of said lexical analysis submodule capable of generating concepts to which the words of the current dialog are linked.

18. The humanoid robot of claim 1,

wherein said dialog module is capable of processing questions and commands from said at least one user relating to the state of the physical and/or logical system of same.

24. The humanoid robot of claim 1, wherein the series of expressions tends to the neutral state over a defined period of time.

25. The humanoid robot of claim 1, wherein the variables further define one or more of resources available to the robot or system constraints applied to the robot.

26. The humanoid robot of claim 25,

wherein the variables defining the one or more of resources available to the robot or system constraints applied to the robot include battery level and temperature of said robot.

19. A method of dialog between a humanoid robot and at least one user, including:

i) at least one step of recognizing inputs originating from at least one sensor selected from a group including audio sensors and visual sensors of events generated by said at least one user,

ii) a step of generating events to said at least one user, and,

iii) a step of dialog with said dialog with said at least one user, said dialog step receiving as input outputs from said at least one recognition step and producing outputs to said step of generating events selected from a group including speech, movements, expressions and emotions, wherein said method further includes a step of controlling the outputs of the event generation module via an artificial intelligence engine according to a context of dialog and variables defining emotions of the robot and as a series of expressions between a neutral state and a terminal state, an evolution between said neutral state and said terminal state being triggered by a pulse generated as a function of time and of one or more of the outputs of the at least a sensor, the event recognition module, the dialog module and the event generation module.

20. The dialog method of claim 19,

wherein variables defining the one or more of resources available to the robot or system constraints applied to the robot include battery level and temperature of said robot.

21. The dialog method of claim 19,

wherein said robot holds a dialog with at least two users, parameters characterizing said at least two users being stored in a memory of said robot for being used when said robot recognizes one of the at least two users.

22. A computer program embedded in a humanoid robot including program code instructions suitable, when executed on a computer, for managing a dialog between said humanoid robot and at least a user, said computer program including:

i) at least one module for recognizing events at the output of at least one sensor selected from a group including audio sensors and visual sensors of events generated by said at least one user,

ii) at least one module for generating events to said at least one user, and,

iii) a module for dialog with said at least one user, said dialog module receiving as input outputs from said at least one recognition module and producing outputs to said module for generating events selected from a group including speech, movements, expressions and emotions, said program being characterized in that it further includes an artificial intelligence engine configured for controlling the outputs of the event generation module according to a context of dialog and variables defining a configuration of the robot as a series of expressions between a neutral state and a terminal state, an evolution between said neutral state and said terminal state being triggered by a pulse generated as a function of time and of one or more of the outputs of the at least a sensor, the event recognition module, the dialog module and the event generation module.

23. A computer program of claim 22,

wherein the computer code instructions have been generated on a ChatScript type interface.