System, method and computer program product for handling humanoid robot interaction with human

• US 10,144,135 B2
• Filed: 04/17/2015
• Issued: 12/04/2018
• Est. Priority Date: 04/17/2014
• Status: Active Grant

• Current Assignee: SoftBank Robotics Europe SAS
• Original Assignee: SoftBank Robotics Europe SAS
• Inventors: Le Borgne, Mathias, Maisonnier, Bruno, Monceaux, Jerome, Veltrop, Taylor, Hervier, Thibault, Jacob, Guillaume, Gate, Gwennael
• Primary Examiner(s): Figueroa, Jaime
• Application Number:
  US15/129,401
• Publication Number:
  US 20170100842A1
• Time in Patent Office: 1,327 Days
• Field of Search: 700245
• CPC Class Codes: B25J 11/0005 Manipulators having means for high-level communication with users, e.g. speech generator, face recognition means
• CPC Class Codes: B25J 13/00 Controls for manipulators programme controls B25J9/16; control in general G05
• CPC Class Codes: B25J 9/1694 characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
• CPC Class Codes: B25J 9/1697 Vision controlled systems

Abstract

A method of handling humanoid robot interaction with human is disclosed. The method allows systematically storing all events detected within a humanoid robot environment, and classifying as temporary stimuli. When the robot is set to analyze the events, a process allows selecting one preferred stimulus among the stored stimuli and depending on the current activity of the robot, one or more actions are generated for the robot.

Claims

1. A method, implemented within a humanoid robot, of handling humanoid robot interaction with a human, the method comprising the steps of:

detecting from a plurality of detectors all events within a humanoid robot environment;

processing each event detected and storing each event detected in a stimuli store as a temporary stimulus with at least an indication of a position of each respective event detected, the temporary stimuli being grouped by type in the stimuli store, the stimuli store further storing permanent stimuli not detected by the plurality of detectors;

detecting that the humanoid robot is entering an engagement mode representing a tracking relationship of the humanoid robot with a human, wherein the engagement mode being one of a fully-engaged mode, a semi-engaged mode, or an unengaged mode;

processing a new event detected by one of the plurality of detectors and determining if the new event is a stimulus originating from a human;

using pertinence rules, selecting one temporary or permanent stimulus stored in the stimuli store;

processing said selected stimulus by triggering a specific scenario software routine, wherein the software routine depends on the engagement mode of the humanoid robot and on a current activity of said humanoid robot;

stopping or continuing the engagement mode in response to the processing of said selected stimulus; and

generating one or more physical actions of the humanoid robot.

2. The method of claim 1 wherein the events detected by the plurality of detectors are one or more of movement detection, tactile sensing, sound localization, face detection, visual perception, image recognition.

3. The method of claim 1 wherein the temporary stimuli are grouped by the type of the event detected in at least a movement stimuli group, a sound stimuli group, a touch stimuli group, a people stimuli group.

4. The method of claim 1 wherein the pertinence rules include priority criteria between the temporary stimuli and the permanent stimuli.

5. The method of claim 1 wherein the step of determining if the new event is a stimulus originating from a human consists in using information provided by a 2D camera and/or a 3D camera.

6. The method of claim 1 wherein the step of triggering the software routine further comprises the step of using the indication of the position of the event corresponding to the selected stimulus for determining if a new human is found in an area around the direction of the stimulus.

7. The method of claim 1 wherein the one or more physical actions of the humanoid robot comprise at least actuating robot members.

8. The method of claim 7 wherein the actuating robot members consists in orienting sensors towards the position of the stimulus detected.

9. The method of claim 8 further comprising the step of updating the stimuli store with information on the position of the new human found.

10. The method of claim 1 further comprising a step of maintaining the humanoid robot in a visually alive state even if no stimulus is sensed, by allowing small movements of the robot members like breathing or using LEDs or producing sounds.

11. A system for handling a humanoid robot interaction with human, the system operating under the control of an operating system and comprising:

at least one processor; and

means coupled to the at least one processor for implementing the steps of the method of claim 1.

12. A computer program product comprising:

a non-transitory computer readable storage medium; and

program code stored on the computer readable storage medium and configured, upon execution, to cause at least one processor to operate the steps of the method of claim 1.

13. The method of claim 1, wherein in the fully-engaged mode, the humanoid robot stops listening to any stimuli event and cannot be diverted until the current engaged human is lost.

14. The method of claim 1, wherein in the unengaged mode, the humanoid robot is configured to be diverted by an event and break current activity.

15. The method of claim 1, wherein in the semi-engaged mode, the humanoid robot is configured to be diverted by an event and to revert back to a previous activity.