Systems and methods to adjust actions based on latency levels

• US 8,947,522 B1
• Filed: 05/04/2012
• Issued: 02/03/2015
• Est. Priority Date: 05/06/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Hickman, Ryan, Kohler, Damon
• Primary Examiner(s): Torrente, Richard
• Application Number:
  US13/464,586
• Time in Patent Office: 1,005 Days
• Field of Search: 348/114
• US Class Current: 348/114
• CPC Class Codes: G05B 2219/40151 Time delay, problems caused by time delay between local and remote
• CPC Class Codes: H04M 2201/50 Telephonic communication in combination with video communication
• CPC Class Codes: H04M 3/487 Arrangements for providing information services, e.g. recorded voice services or time announcements
• CPC Class Codes: H04M 3/567 Multimedia conference systems
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

Systems and methods to adjust actions based on latency levels are described. The method may be executable to determine an action being performed by a robotic device. The method may also be executable to generate data based on execution of the action to provide to a server and receive information indicative of latency attributable to communication between the robotic device and a server. The method may also determine a priority at which to provide the generated data to the server and provide the generated data to the server in an order based on the priority. When the latency is above a threshold amount, the method may identify information of the generated data having a given priority above a threshold level and provide a reduced resolution version of the identified information to the server.

Claims

1. A method comprising:

determining an action being performed by a robotic device, wherein the robotic device includes a mechanical actuator;

based on execution of the action, the robotic device generating data for a computing device, wherein the generated data includes video data and audio data, the video data having a first resolution and protocol data indicative of a communication protocol for the robotic device to communicate the video data;

receiving information from the computing device that is indicative of a latency attributable to communication between the robotic device and the computing device;

based on the latency, determining a latency level associated with the latency attributable to the communication of the generated data and a priority at which to provide the video data and the audio data to the computing device, wherein the priority is based on a frequency at which the generated data changes over a period of time;

responsive to determining that the latency level is above a first threshold and below a second threshold, providing, by the robotic device, the video data and the audio data to the computing device in an order based on the priority at a reduced resolution version of the video data to the computing device and discontinuing transmission of the audio data to the computing device, wherein the second threshold is more than the first threshold; and

responsive to determining that the latency level is above the second threshold, determining an alternate communication pathway by which the robotic device is capable to communicate with the computing device, and communicating with the computing device over the alternate communication pathway.

2. The method of claim 1, wherein the video data corresponds to a field of view of the robotic device.

3. The method of claim 1, wherein the generated data further comprises information based on a safety level of the robotic device during execution of the action.

4. The method of claim 1, wherein the generated data further comprises data collected by a sensor of the robotic device.

5. The method of claim 1, further comprising responsive to determining that the latency level is above the first threshold and below the second threshold, reducing a frame rate at which the robotic device provides the video data to the computing device.

6. The method of claim 1, wherein receiving information from the computing device that is indicative of the latency attributable to the communication between the robotic device and the computing device comprises the robotic device receiving information indicative of a bandwidth constraint of a network comprising a communication path between the robotic device and the computing device.

7. The method of claim 1, further comprising responsive to determining that the latency level is above the first threshold and below the second threshold, providing by the robotic device the generated data to the computing device based on a user datagram protocol (UDP).

8. The method of claim 1, further comprising:

responsive to determining that the latency level is above the first threshold, storing the generated data having the first resolution; and

responsive to determining that the latency level is below the first threshold, providing by the robotic device the stored generated data to the computing device.

9. A system comprising:

a robotic device having a sensor;

a computing system including a non-transitory computer-readable medium; and

program instructions stored on the non-transitory computer-readable medium and executable to cause the computing system to;

determine an action being performed by a robotic device, wherein the robotic device includes a mechanical actuator;

based on execution of the action, the robotic device generating data for a computing device, wherein the generated data includes video data having a first resolution, and protocol data indicative of a communication protocol for the robotic device to communicate the video data;

receive information from the computing device that is indicative of a latency attributable to communication between the robotic device and the computing device;

based on the latency, determine a latency level associated with the latency attributable to the communication of the generated data and a priority at which to provide the generated data to the computing device, wherein the priority is based on a frequency at which the generated data changes over a period of time;

responsive to determining that the latency level is above a first threshold and below a second threshold, provide, by the robotic device, the generated data to the computing device in an order based on the priority at a reduced resolution version of the video data to the computing device and discontinuing transmission of the audio data to the computing device, wherein the second threshold is more than the first threshold; and

responsive to determining that the latency level is above the second threshold, determining an alternate communication pathway by which the robotic device is capable to communicate with the computing device, and communicating with the computing device over the alternate communication pathway.

10. The system of claim 9, further comprising program instructions stored on the non-transitory computer-readable medium and executable to cause the computing system to:

responsive to determining that the latency level is above the first threshold and below a second threshold, reduce a frame rate at which the robotic device provides the generated data to the computing device.

11. The system of claim 9, further comprising program instructions stored on the non-transitory computer-readable medium and executable to cause the computing system to:

responsive to determining that the latency level is above the first threshold and below the second threshold, provide by the robotic device the generated data to the computing device based on a user datagram protocol (UDP).

12. The system of claim 9, further comprising program instructions stored on the non-transitory computer-readable medium and executable to cause the computing system to:

responsive to determining that the latency level is above the first threshold and below the second threshold, store the generated data having the first resolution; and

responsive to determining that the latency level is above the first threshold and below the second threshold, provide by the robotic device the stored generated data to the computing device.

13. A non-transitory computer-readable memory having stored thereon instructions executable by a computing device having at least one processor to cause the computing device to perform functions comprising:

determining an action being performed by a robotic device, wherein the robotic device includes a mechanical actuator;

based on execution of the action, the robotic device generating data for a computing device, wherein the generated data includes video data having a first resolution, and protocol data indicative of a communication protocol for the robotic device to communicate the video data;

receiving information from the computing device that is indicative of a latency attributable to communication between the robotic device and the computing device;

based on the latency, determining a latency level associated with the latency attributable to the communication of the generated data and a priority at which to provide the generated data to the computing device, wherein the priority is based on a frequency at which the generated data changes over a period of time;

responsive to determining that the latency level is above a first threshold and below a second threshold, providing, by the robotic device, the generated data to the computing device in an order based on the priority at a reduced resolution version of the video data to the computing device and discontinuing transmission of the audio data to the computing device, wherein the second threshold is more than the first threshold; and

responsive to determining that the latency level is above the second threshold, determining an alternate communication pathway by which the robotic device is capable to communicate with the computing device, and communicating with the computing device over the alternate communication pathway.

14. The non-transitory computer-readable memory of claim 13, wherein the function of generating data for the computing device comprises generating video data corresponding to a field of view of the robotic device.

15. The non-transitory computer-readable memory of claim 13, wherein the function of providing the reduced resolution version of the generated data to the computing device comprises decreasing an amount of data provided to the computing device.

16. The non-transitory computer-readable memory of claim 15, wherein the function of decreasing the amount of data provided to the computing device comprises dropping an audio feed of the robotic device.