Methods and systems for adjusting operation of a robotic device based on detected sounds

• US 9,815,203 B1
• Filed: 08/24/2015
• Issued: 11/14/2017
• Est. Priority Date: 08/24/2015
• Status: Active Grant

• Current Assignee: X Development LLC (Alphabet Inc.)
• Original Assignee: X Development LLC (Alphabet Inc.)
• Inventors: Stoia, Laura, Latimer, Craig, Guilbert, Matthieu
• Primary Examiner(s): Bendidi, Rachid
• Application Number:
  US14/833,811
• Time in Patent Office: 813 Days
• CPC Class Codes: B25J 9/1671 characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems
• CPC Class Codes: B25J 9/1674 characterised by safety, monitoring, diagnostic
• CPC Class Codes: B25J 9/1689 Teleoperation
• CPC Class Codes: B25J 9/1694 characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 2219/33284 Remote diagnostic
• CPC Class Codes: G05B 2219/37337 Noise, acoustic emission, sound

Abstract

An example method includes determining an expected sound profile corresponding to a given task for a robotic device. The method further includes detecting a sound profile during execution of the given task by the robotic device. The method also includes determining one or more differences in amplitude for at least one frequency range between the detected sound profile and the expected sound profile corresponding to the given task for the robotic device. In response to determining the one or more differences in amplitude for the at least one frequency range between the detected sound profile and the expected sound profile, the method additionally includes identifying at least one component of the robotic device associated with the detected sound profile during execution of the given task. The method further includes adjusting control data for the at least one component of the robotic device.

Claims

1. A method comprising:

receiving an expected sound profile corresponding to a given task for a robotic device from a remote server that aggregates detected sound profiles during performance of the given task by a plurality of other robotic devices;

detecting, based on sensor data from at least one audio sensor associated with the robotic device, a sound profile during execution of the given task by the robotic device;

determining one or more differences in amplitude for at least one frequency range between the detected sound profile and the expected sound profile corresponding to the given task for the robotic device;

in response to determining the one or more differences in amplitude for the at least one frequency range between the detected sound profile and the expected sound profile corresponding to the given task for the robotic device, identifying at least one component of the robotic device associated with the detected sound profile during execution of the given task;

providing the detected sound profile during execution of the given task by the robotic device to the remote server;

based on the provided detected sound profile, receiving one or more commands from the remote server for adjusting control data for the at least one component; and

adjusting the control data for the at least one component of the robotic device.

2. The method of claim 1, wherein the expected sound profile comprises an amplitude of sound in the at least one frequency range over a time domain.

3. The method of claim 1, wherein the expected sound profile comprises one or more wavelets over a frequency domain, wherein the frequency domain comprises the at least one frequency range.

4. The method of claim 1, wherein the expected sound profile comprises a plurality of expected sounds corresponding to contact between the robotic device and a plurality of material types.

5. The method of claim 4, further comprising:

determining that the detected sound profile correlates with a particular sound of the plurality of expected sounds corresponding to the plurality of material types; and

based on the correlation, adjusting control data of an end effector of the robotic device according to a material type corresponding to the particular sound.

6. The method of claim 1, further comprising:

determining that the given task is associated with control of a suction gripper of the robotic device;

based on the one or more differences in amplitude for the at least one frequency range between the detected sound profile and the expected sound profile, identifying an error associated with one or more image sensors of the robotic device during the given task; and

adjusting positioning of the suction gripper based on the identified error associated with the one or more image sensors of the robotic device.

7. A non-transitory computer-readable medium having stored thereon program instructions that when executed by a computing device, cause the computing device to perform functions comprising:

receiving an expected sound profile corresponding to a given task for a robotic device from a remote server that aggregates detected sound profiles during performance of the given task by a plurality of other robotic devices;

detecting, based on sensor data from at least one audio sensor associated with the robotic device, a sound profile during execution of the given task by the robotic device;

determining one or more differences in amplitude for at least one frequency range between the detected sound profile and the expected sound profile corresponding to the given task for the robotic device;

in response to determining the one or more differences in amplitude for the at least one frequency range between the detected sound profile and the expected sound profile corresponding to the given task for the robotic device, identifying at least one component of the robotic device associated with the detected sound profile during execution of the given task;

providing the detected sound profile during execution of the given task by the robotic device to the remote server;

based on the provided detected sound profile, receiving one or more commands from the remote server for adjusting control data for the at least one component; and

adjusting the control data for the at least one component of the robotic device.

8. The non-transitory computer-readable medium of claim 7, wherein the expected sound profile comprises an amplitude of sound in the at least one frequency range over a time domain, the functions further comprising:

determining the one or more differences in amplitude for the at least one frequency range between the detected sound profile and the expected sound profile at corresponding times during execution of the given task.

9. The non-transitory computer-readable medium of claim 7, wherein the at least one component of the robotic device comprises one or more actuators.

10. The non-transitory computer-readable medium of claim 9, wherein adjusting the control data for the one or more actuators comprises causing the one or more actuators to power down.

11. The non-transitory computer-readable medium of claim 9, wherein the control data for the one or more actuators comprises one or more values associated with at least one of position control, velocity control, and current control.

12. A robotic device comprising:

one or more audio sensors, wherein the one or more audio sensors are configured to detect sounds associated with operation of the robotic device;

one or more processors; and

data storage comprising instructions executable by the one or more processors to cause the robotic device to perform operations comprising;

receiving an expected sound profile corresponding to a given task for the robotic device from a remote server that aggregates detected sound profiles during performance of the given task by a plurality of other robotic devices;

detecting, based on sensor data from the one or more audio sensors, a sound profile during execution of the given task by the robotic device;

determining one or more differences in amplitude for at least one frequency range between the detected sound profile and the expected sound profile corresponding to the given task for the robotic device;

in response to determining the one or more differences in amplitude for the at least one frequency range between the detected sound profile and the expected sound profile corresponding to the given task for the robotic device, identifying at least one component of the robotic device associated with the detected sound profile during execution of the given task;

providing the detected sound profile during execution of the given task by the robotic device to the remote server;

based on the provided detected sound profile, receiving one or more commands from the remote server for adjusting control data for the at least one component; and

adjusting the control data for the at least one component of the robotic device.

13. The robotic device of claim 12, wherein the at least one component of the robotic device comprises one or more sensors associated with image capture.

14. The robotic device of claim 13, wherein adjusting the control data for the one or more sensors associated with image capture further comprises:

adjusting one or more values corresponding to image quality.

15. The robotic device of claim 13, wherein adjusting the control data for the one or more sensors associated with image capture further comprises:

adjusting one or more values corresponding to at least one of pan, tilt, or zoom.

16. The robotic device of claim 12, the operations further comprising:

based on the one or more differences in amplitude for at least one frequency range between the detected sound profile and the expected sound profile corresponding to the given task for the robotic device, determining an error associated with execution of the given task; and

providing, to a computing device located remotely from the robotic device, information indicative of the determined error.