Sensor use and analysis for dynamic update of interaction in a social robot

• US 9,724,824 B1
• Filed: 07/08/2015
• Issued: 08/08/2017
• Est. Priority Date: 07/08/2015
• Status: Active Grant

• Current Assignee: T-Mobile Innovations LLC (Deutsche Telekom AG)
• Original Assignee: Sprint Communications Company LP (Deutsche Telekom AG)
• Inventors: Gilbert, Deborah M., Annan, Brandon, Cole, Joshua R., Indurkar, Dhananjay
• Primary Examiner(s): Kiswanto, Nicholas
• Assistant Examiner(s): Mott, Adam
• Application Number:
  US14/794,765
• Time in Patent Office: 762 Days
• CPC Class Codes: B25J 11/0005 Manipulators having means for high-level communication with users, e.g. speech generator, face recognition means
• CPC Class Codes: B25J 9/16 Programme controls programme controls in general G05B19/00, e.g. numerical pogramme controls G05B19/18; recording or playback systems G05B19/42
• CPC Class Codes: B25J 9/1615 characterised by special kind of manipulator, e.g. planar, scara, gantry, cantilever, space, closed chain, passive/active joints and tendon driven manipulators
• CPC Class Codes: G05B 2219/33056 Reinforcement learning, agent acts, receives reward, emotion, action selective
• CPC Class Codes: G05B 2219/40411 Robot assists human in non-industrial environment like home or office
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06N 3/008 based on physical entities controlled by simulated intelligence so as to replicate intelligent life forms, e.g. based on robots replicating pets or humans in their appearance or behaviour
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks

Abstract

A method of optimizing social interaction between a robot and a human. The method comprises generating then executing a robot motion script for interaction with a human by a robot based on a characteristic detected by at least one of a plurality of sensors on the robot. The method further comprises detection, by at least one sensor of the robot, a reaction of the human during a first period. The robot then analyzes the reaction of the human and assigns a positive or negative classification to the reaction based on pre-defined mapping stored in the memory of the robot. The method further comprises modifying the robot motion script to incorporate a pre-defined modification based on the determination of a negative classification of the human reaction. The method further comprises executing the modified robot motion script during a second period to obtain an improved interaction with the human.

Claims

1. A social robot comprising:

a speaker;

a plurality of sensors, the plurality of sensors comprising at least one audio sensor and at least one visual sensor;

a processor; and

a non-transitory computer readable storage medium storing programming for execution by the processor, the programming including instructions to;

receive input via at least one of the plurality of sensors;

identify a plurality of characteristics of a human based on the input received via the at least one of the plurality of sensors, the plurality of characteristics of the human comprising a diction level, a voice cadence, a gender, and an age range;

generate, based on the plurality of characteristics, a robot dialog that includes a first control question for interaction with the human, wherein the first control question invokes at least one of a physical reaction or vocal reaction from the human to confirm that the plurality of characteristics are associated with one of a plurality of profiles stored in the storage medium;

playback, via the speaker, the robot dialog as audible speech to initiate a social interaction between the social robot and the human;

in response to playback of the first control question of the robot dialog, detect, via one or more of the plurality of sensors, a reaction of the human, the reaction being at least one of a physical reaction and a vocal reaction;

confirm that a profile corresponding to the human matches the plurality of characteristics based on the detected reaction;

based on the profile, initiate a motion script that activates actuators of the social robot;

irrespective of the detected reaction of the human, insert additional control questions into the robot dialog, wherein the additional control questions are dispersed throughout the robot dialog and invoke one or more of a physical reaction and a vocal reaction from the human; and

continue playback of the robot dialog that includes the additional control questions.

2. The social robot of claim 1, wherein the plurality of characteristics of the human identified further comprises a race and a non-anthropometric identifier chosen by the human.

3. The social robot of claim 1, wherein the plurality of characteristics is identified based on at least one of an audio data acquired by the at least one of the plurality of sensors and a visual data acquired by the at least one of the plurality of sensors.

4. A method for adapting robot motion to improve social interaction between a social robot and a human comprising:

receiving, by a social robot executing a processor, input via a plurality of sensors of the social robot;

identifying, by the social robot, a plurality of characteristics of the human based on the input received via the plurality of sensors of the social robot, the plurality of characteristics of the human comprising a diction level, a voice cadence, a gender, and an age range;

based on identifying the plurality of characteristics, generating, by the social robot, a robot dialog that includes a first control question for interaction with the human, wherein the first control question invokes at least one of a physical reaction or vocal reaction from the human to confirm that the plurality of characteristics are associated with one of a plurality of profiles stored in a non-transitory storage medium communicatively coupled to the social robot;

generating, by the social robot executing a motion script application, a robot motion script for interaction with the human;

executing, by the social robot, the robot motion script that activates actuators of the social robot to initiate a social interaction between the social robot and the human;

playing, via a speaker of the social robot, the robot dialog as audible speech while the robot motion script is executing;

in response to playing the first control question of the robot dialog, detecting, via the plurality of sensors, a reaction of the human, the reaction being at least one of a physical reaction and a vocal reaction;

confirming, by the social robot, that a profile corresponding to the human matches the plurality of characteristics based on the detected reaction;

irrespective of the detected reaction of the human, inserting, by the social robot, additional control questions into the robot dialog, wherein the additional control questions are dispersed throughout the robot dialog and invoke one or more of a physical reaction and a vocal reaction from the human; and

continuing, by the social robot, playback of the robot dialog that includes the additional control questions.

5. The method of claim 4, wherein the plurality of characteristics is identified based on at least one of an audio data acquired by one of the plurality of the sensors and a visual data acquired by one of the plurality of the sensors.

6. The method of claim 4, wherein executing the motion script occurs over a first time period.

7. The method of claim 4, further comprising:

adapting, by the social robot, the robot motion script and the robot dialog in response to reactions from the human following playback of the additional questions.

8. The method of claim 4, wherein a characteristic of the plurality of characteristics corresponds with a race of the human.

9. The method of claim 4, wherein the plurality of characteristics is identified by a cadence range of voice based on a pre-defined mapping stored in a memory of the social robot mapped to at least one of an age range, race, and gender.

10. The method of claim 4, wherein the plurality of characteristics is identified by a visually acquired data based on a pre-defined mapping stored in a memory of the social robot mapped to at least one of an age profile, a race profile, and a gender profile.

11. The method of claim 4, wherein the robot motion script comprises at least one or more of audio and movement commands.

12. The method of claim 4, wherein identifying the plurality of characteristics comprises:

collecting, by the plurality of sensors, audio or visual data corresponding to the human, and

determining, by the social robot, that the audio or visual data is associated with at least one user profile stored in a memory of the social robot.

13. A method for adapting robot motion to improve social interaction between a robot and a human comprising:

identifying, by a social robot, a plurality of characteristics of a human via at least one sensor of the social robot, the plurality of characteristics of the human comprising a diction level, a voice cadence, a gender, and an age range;

based on identifying the plurality of characteristics, generating, by the social robot, a robot dialog that includes a first control question for interaction with the human, wherein the first control question invokes at least one of a physical reaction or vocal reaction from the human to confirm that the plurality of characteristics are associated with one of a plurality of profiles stored in a non-transitory storage medium communicatively coupled to the social robot;

generating, by the social robot, a robot motion script for interaction with the human;

executing, by the social robot, the robot motion script to interact with the human during a first period;

playing, via a speaker of the social robot, the robot dialog as audible speech while the robot motion script is executing;

in response to playing at least the control question of the robot dialog, detecting, by the social robot, a reaction of the human during the first period via a sensor of the social robot;

in response to detecting, analyzing, by the social robot, the reaction of the human;

confirming, by the social robot, that a profile corresponding to the human matches the plurality of characteristics based on analyzing the reaction;

based on the profile, modifying, by the social robot, the robot motion script to obtain a modified robot motion script;

irrespective of the detected reaction of the human, inserting, by the social robot, additional control questions into the robot dialog, wherein the additional control questions are dispersed throughout the robot dialog and invoke one or more of a physical reaction and a vocal reaction from the human;

executing, by the social robot, the modified robot motion script during a second period to obtain an improved interaction with the human; and

continuing, by the social robot, playback of the robot dialog that includes at least some of the additional control questions during the second period.

14. The method of claim 13, wherein the reaction comprises at least one of a vocal reaction or a physical reaction, the vocal reaction comprising at least one voice generated sound by the human and the physical reaction comprising at least one body movement made by the human.

15. The method of claim 14, wherein analyzing the reaction by the social robot comprises determining a positive or negative classification of the reaction based on playback of the first control question.

16. The method of claim 15, wherein the determining the positive or negative classification comprises:

detecting, by the social robot, at least one of a voice generated sound or a body movement of the human during the first period; and

determining, by the social robot, that the voice generated sound or the body movement is associated with one of the positive or negative classification based on a pre-defined mapping stored in a memory of the social robot, wherein the pre-defined mapping maps known body movement or voice-generated sound queues to positive or negative reactions.

17. The method of claim 13, wherein modifying the robot motion script comprises incorporating a pre-defined modification stored in a memory of the social robot, and wherein the pre-defined modification is based on the determination of a negative reaction.

18. The method of claim 13, wherein the robot motion script comprises at least one or more of audio and movement commands of the social robot.

19. The method of claim 13, wherein the modified robot motion script comprises at least one or more of audio and movement commands of the social robot.

20. The method of claim 13, wherein the steps of detecting the reaction, analyzing the reaction of the human, modifying the robot motion script based on the analysis, and executing, by the social robot, the modified robot motion script during a second period are repeated until a positive reaction from the human is determined to occur by the social robot.