Robot to human feedback

• US 9,902,061 B1
• Filed: 08/25/2015
• Issued: 02/27/2018
• Est. Priority Date: 08/25/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: X Development LLC (Alphabet Inc.)
• Inventors: Kuffner, James Joseph
• Primary Examiner(s): Moyer, Dale
• Application Number:
  US14/835,411
• Time in Patent Office: 917 Days
• CPC Class Codes: B25J 11/0005 Manipulators having means for high-level communication with users, e.g. speech generator, face recognition means
• CPC Class Codes: B25J 19/06 Safety devices
• CPC Class Codes: B25J 19/061 with audible signals audio controls B25J13/003
• CPC Class Codes: B25J 9/163 learning, adaptive, model based, rule based expert control
• CPC Class Codes: B25J 9/1674 characterised by safety, monitoring, diagnostic
• CPC Class Codes: B25J 9/1676 Avoiding collision or forbidden zones
• CPC Class Codes: G05B 2219/39 Robotics, robotics to robotics hand
• CPC Class Codes: G05B 2219/40 Robotics, robotics mapping to robotics vision
• CPC Class Codes: G05B 2219/40202 Human robot coexistence
• CPC Class Codes: H04W 4/30 Services specially adapted for particular environments, situations or purposes
• CPC Class Codes: H04W 4/80 Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication

Abstract

Example implementations may relate to a robotic system configured to provide feedback. In particular, the robotic system may determine a model of an environment in which the robotic system is operating. Based on this model, the robotic system may then determine one or more of a state or intended operation of the robotic system. Then, based one or more of the state or the intended operation, the robotic system may select one of one or more of the following to represent one or more of the state or the intended operation: visual feedback, auditory feedback, and one or more movements. Based on the selection, the robotic system may then engage in one or more of the visual feedback, the auditory feedback, and the one or more movements.

Claims

1. A method comprising:

determining, by a robotic system, a model of an environment in which the robotic system is operating;

determining, by the robotic system, one or more of a state or intended operation of the robotic system based at least in part on the model of the environment;

determining, by the robotic system, a first performance metric that is associated with a first level of safety of one or more of the state or the intended operation;

determining, by the robotic system, that the first performance metric is above a threshold performance metric;

based at least in part on determining that the first performance metric is above the threshold performance metric, making a selection, by the robotic system, of a first feedback mode corresponding to the first level of safety;

based at least in part on the selection, engaging, by the robotic system, in the first feedback mode;

determining, by the robotic system, that the first performance metric changed to a second performance metric that is associated with a second level of safety, and responsively determining that the second performance metric is below the threshold performance metric;

based at least in part on determining that the second performance metric is below the threshold performance metric, making a further selection, by the robotic system, of a second feedback mode corresponding to the second level of safety; and

based at least in part on the further selection, engaging, by the robotic system, in the second feedback mode.

2. The method of claim 1, wherein engaging in the second feedback mode comprises using one or more light sources to emit light having particular light characteristics.

3. The method of claim 1,

wherein determining the model of the environment comprises determining an area of interest in the environment,

wherein determining one or more of a state or intended operation comprises determining one or more of a state or intended operation related to the area of interest,

wherein making the further selection of the second feedback mode comprises selecting visual feedback that includes projection of light from one or more light sources of the robotic system towards the area of interest, and

wherein engaging in the second feedback mode comprises projecting the light from the one or more light sources of the robotic system towards the area of interest.

4. The method of claim 1,

wherein the first feedback mode comprises one or more of first visual feedback to represent the first level of safety, first auditory feedback to represent the first level of safety, and one or more first movements to represent the first level of safety, and

wherein the second feedback mode comprises one or more of second visual feedback to represent the second level of safety, second auditory feedback to represent the second level of safety, and one or more second movements to represent the second level of safety.

5. The method of claim 1,

wherein determining the model of the environment comprises determining a first distance between the robotic system and an object positioned in the environment,

wherein determining that the first performance metric is above a threshold performance metric comprises determining that the first distance is greater than a threshold distance,

wherein determining that the first performance metric changed to a second performance metric comprises determining that a distance between the robotic system and the object changed from the first distance to a second distance, and

wherein determining that the second performance metric is below the threshold performance metric comprises determining that the second distance is lesser than the threshold distance.

6. The method of claim 1, further comprising:

based at least in part on the model of the environment, predicting, by the robotic system, one or more actions by a human positioned within a threshold distance of the robotic system,

wherein determining one or more of a state or intended operation of the robotic system is further based at least in part on the predicted one or more actions.

7. The method of claim 1, further comprising:

based at least in part on the model of the environment, evaluating, by the robotic system, one or more actions by a human positioned within a threshold distance of the robotic system,

wherein determining one or more of a state or intended operation of the robotic system is further based at least in part on the evaluated one or more actions.

8. The method of claim 1,

wherein engaging in the first feedback mode comprises sending a notification to a computing device.

9. The method of claim 1, wherein determining the model of the environment comprises receiving sensor data, from one or more sensors associated with the robotic system, indicating one or more of:

image data, sound data, temperature data, depth data, proximity data, motion data, speech recognition data, facial recognition data, and location data.

10. The method of claim 1,

wherein engaging in the second feedback mode comprises sending a notification to a computing device.

11. A robotic system comprising:

one or more processors;

a non-transitory computer readable medium; and

program instructions stored on the non-transitory computer readable medium and executable by the one or more processors to;

determine a model of an environment in which the robotic system is operating;

determine one or more of a state or intended operation of the robotic system based at least in part on the model of the environment;

determine a first performance metric that is associated with a first level of safety of one or more of the state or the intended operation;

determine that the first performance metric is above a threshold performance metric;

based at least in part on determining that the first performance metric is above the threshold performance metric, make a selection of a first feedback mode corresponding to the first level of safety;

based at least in part on the selection, engage in the first feedback mode;

determine that the first performance metric changed to a second performance metric that is associated with a second level of safety, and responsively determining that the second performance metric is below the threshold performance metric;

based at least in part on determining that the second performance metric is below the threshold performance metric, make a further selection of a second feedback mode corresponding to the second level of safety; and

based at least in part on the further selection, engage in the second feedback mode.

12. The robotic system of claim 11, further comprising one or more appendages, wherein engaging in the first feedback mode comprises carrying out one or more gestures using the one or more appendages.

13. The robotic system of claim 11, wherein engaging in the second feedback mode comprises repositioning of the robotic system from a first location in the environment to a second location in the environment.

14. The robotic system of claim 11, wherein engaging in the second feedback mode comprises the robotic system repositioning an object in the environment from a first location in the environment to a second location in the environment.

15. The robotic system of claim 14,

wherein determining the model of the environment comprises determining an estimated path of a human in the environment,

wherein the first location is in the estimated path of the human, and

wherein the second location is away from the estimated path of the human.

16. A non-transitory computer readable medium having stored therein instructions executable by one or more processors to cause a robotic system to perform functions comprising:

determining a model of an environment in which the robotic system is operating;

determining one or more of a state or intended operation of the robotic system based at least in part on the model of the environment;

determining a first performance metric that is associated with a first level of safety of one or more of the state or the intended operation;

determining that the first performance metric is above a threshold performance metric;

based at least in part on determining that the first performance metric is above the threshold performance metric, making a selection of a first feedback mode corresponding to the first level of safety;

based at least in part on the selection, engaging in the first feedback mode;

determining that the first performance metric changed to a second performance metric that is associated with a second level of safety, and responsively determining that the second performance metric is below the threshold performance metric;

based at least in part on determining that the second performance metric is below the threshold performance metric, making a further selection of one or more movements to increase performance to a level that is above the threshold performance metric; and

based at least in part on the further selection, engaging in the one or more movements to increase performance to a level that is above the threshold performance metric.

17. The non-transitory computer readable medium of claim 16,

wherein the first level of safety corresponds to one or more of:

safety of the robotic system during one or more of the state or the intended operation, safety of a human in the environment during one or more of the state or the intended operation, and safety of an object in the environment during one or more of the state or the intended operation.

18. The non-transitory computer readable medium of claim 16, the functions further comprising:

based at least in part on the model of the environment, estimating an age of a human in the environment,

wherein making the selection is further based at least in part on the estimated age of the human.

19. The non-transitory computer readable medium of claim 16, the functions further comprising:

based at least in part on the model of the environment, detecting a language spoken by a human in the environment, wherein making the selection comprises selecting auditory feedback that includes speech in the language spoken by the human.

20. The non-transitory computer readable medium of claim 16,

wherein the first feedback mode comprises one or more of first visual feedback to represent the first level of safety, first auditory feedback to represent the first level of safety, and one or more first movements to represent the first level of safety.

21. The non-transitory computer readable medium of claim 16,

wherein engaging in the first feedback mode comprises sending a notification to a computing device.

22. The non-transitory computer readable medium of claim 16,

wherein determining the model of the environment comprises determining a first distance between the robotic system and an object positioned in the environment,

wherein the first distance corresponds to the first performance metric, and

wherein engaging in the one or more movements to increase performance to a level that is above the threshold performance metric comprises repositioning the robotic system such that the first distance changes to a second distance corresponding with performance being at a level that is above the threshold performance metric.