Overlaying sensor data in a user interface

• US 8,718,822 B1
• Filed: 04/05/2012
• Issued: 05/06/2014
• Est. Priority Date: 05/06/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Hickman, Ryan, Lewis, Thor
• Primary Examiner(s): Jen, Ian
• Application Number:
  US13/440,656
• Time in Patent Office: 761 Days
• Field of Search: 700/214, 700/218, 700/245, 700/246, 700/259, 700/260, 700/264, 318/568.11, 340/540, 340/683
• US Class Current: 700/259
• CPC Class Codes: B25J 9/1674 characterised by safety, monitoring, diagnostic
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 2219/40163 Measuring, predictive information feedback to operator
• CPC Class Codes: G05B 2219/40165 Sensor data to display depends on robot status
• CPC Class Codes: Y10S 901/47 Optical

Abstract

Methods and systems for robot functions and user interfaces are described. A server may receive a set of robot parameters, and may predict new parameters based on a robot command. In this manner, a user may receive parameters corresponding to the predicted values and mitigate network and processing latency. In other examples, a robot may provide a forward looking image and a robot speed. When a command to move forward is issued, the server may provide a predicted image and predicted speed. The server may be able to calculate a predicted image and a predicted speed (or other parameter) more quickly than the robot could provide the same information. The predicted information may be displayed on a user interface with a corresponding indication that the values are predicted. The robot may provide the server and the user interface with the actual data when it is available.

Claims

1. A method comprising:

receiving data indicative of a status of a robotic device, wherein the data comprises one or more attributes of the robotic device;

generating a display of at least a subset of the one or more attributes of the robotic device in a user interface;

transmitting a robot command for the robotic device;

receiving data indicative of a revised status of the robotic device, wherein the revised data comprises one or more revised attributes of the robotic device and one or more respective data confidence indicators, wherein the one or more respective data confidence indicators indicate a source of the revised data, wherein the source is the robotic device or a server; and

generating a display of at least a subset of the data indicative of the revised status of the robotic device in the user interface including, for each of the subset of the data indicative of the revised status of the robotic device, a display associated with the respective data confidence indicator.

2. The method of claim 1, wherein:

receiving data indicative of a status of a robotic device comprises receiving data from a server with a control device;

transmitting a robot command comprises transmitting a command from the control device to the server; and

receiving revised data indicative of the robots status comprises receiving data from the server with the control device.

3. The method of claim 1, wherein the data indicative of a status of a robotic device comprises an image from a camera connected to the robotic device.

4. The method of claim 1, wherein the data indicative of a status of a robotic device comprises a video from a camera connected to the robotic device.

5. The method of claim 1, wherein the data indicative of a status of a robotic device comprises data from sensors associated with a robot.

6. The method of claim 5, wherein the data indicative of a status of a robotic device comprises a robot velocity.

7. The method of claim 5, wherein the data indicative of a status of a robotic device status comprises a robot position.

8. The method of claim 1, wherein the display associated with the associated data confidence indicator includes a color, wherein the color changes based on the confidence indicator.

9. A method comprising:

receiving data indicative of a status of a robotic device, wherein the data comprises one or more attributes of the robotic device;

transmitting the data indicative of a status of a robotic device;

receiving a robot command;

transmitting the robot command;

determining revised data indicative of a status of a robotic device, wherein the revised data comprises a calculated prediction of data indicative of the robots status based on the transmitted robot command and wherein the calculating includes an associated data confidence indicator for each calculated prediction, wherein the associated data confidence indicator indicates the data is predicted;

transmitting the revised data indicative of a status of a robotic device and the associated data confidence indicator for each calculated prediction;

receiving data indicative of a status of a robotic device, wherein the data comprises received robot attributes;

determining further revised data indicative of a status of a robotic device, wherein the revised data comprises the received data indicative of a status of a robotic device and wherein the determining includes an associated data confidence indicator for the received data indicative of a status of a robotic device, wherein the associated data confidence indicator indicates the data is received; and

transmitting the further revised data indicative of a status of a robotic device and the associated data confidence indicator.

10. The method of claim 9, wherein the data indicative of a status of a robotic device comprises an image from a camera connected to the robotic device.

11. The method of claim 9, wherein the data indicative of a status of a robotic device comprises a video from a camera connected to the robotic device.

12. The method of claim 9, wherein data indicative of a status of a robotic device comprises data from sensors associated with a bot.

14. The method of claim 12, wherein the data indicative of a status of a robotic device comprises a robot position.

15. The method of claim 9, wherein the display associated with the associated data confidence indicator includes a color, wherein the color changes based on the confidence indicator.

13. The method of claim herein the data indicative of a status of a robotic device comprises a robot velocity.

16. An article of manufacture including a non-transitory computer-readable medium, having stored thereon program instructions that, if executed by a computing device, cause the computing device to perform operations comprising:

receiving data indicative of a status of a robotic device, wherein the data comprises one or more attributes of the robotic device;

generating a display of at least a subset of the one or more attributes of the robotic device in a user interface;

transmitting a robot command;

receiving data indicative of a revised status of the robotic device, wherein the revised data comprises one or more revised attributes of the robotic device and one or more respective data confidence indicators, wherein the one or more respective data confidence indicators indicate a source of the revised data, wherein the source is the robotic device or a server; and

generating a display of at least a subset of the data indicative of the revised status of the robotic device in the user interface including, for each of the subset of the data indicative of the revised status of the robotic device, a display associated with the respective data confidence indicator.

17. The article of manufacture of claim 16, wherein:

receiving data indicative of a status of a robotic device status comprises receiving data from a server with a control device;

transmitting a robot command comprises transmitting a command from the control device to the server; and

receiving revised data indicative of the robots status comprises receiving data from the server with the control device.

18. The article of manufacture of claim 16, wherein the data indicative of a status of a robotic device comprises an image or a video from a camera connected to the robotic device.

19. The article of manufacture of claim 16, wherein the data indicative of a status of a robotic device comprises data from sensors associated with a robot.

20. The article of manufacture of claim 19, wherein the data indicative of a status of a robotic device comprises a robot velocity.

21. The article of manufacture of claim 19, wherein the data indicative of a status of a robotic device comprises a robot position.

22. The article of manufacture of claim 16, wherein the display associated with the associated data confidence indicator includes a color, wherein the color changes based on the confidence indicator.

23. An article of manufacture including a non-transitory computer-readable medium, having stored thereon program instructions that, if executed by a computing device, cause the computing device to perform operations comprising:

receiving data indicative of a status of a robotic device, wherein the data comprises one or more attributes of the robotic device;

transmitting the data indicative of a status of a robotic device;

receiving a robot command;

transmitting the robot command;

determining revised data indicative of a status of a robotic device, wherein the revised data comprises a calculated prediction of data indicative of the robots status based on the transmitted robot command and wherein the calculating includes an associated data confidence indicator for each calculated prediction, wherein the associated data confidence indicator indicates the data is predicted;

transmitting the revised data indicative of a status of a robotic device and the associated data confidence indicator for each calculated prediction;

receiving data indicative of a status of a robotic device, wherein the data comprises received robot attributes;

determining further revised data indicative of a status of a robotic device, wherein the revised data comprises the received data indicative of a status of a robotic device and wherein the determining includes an associated data confidence indicator for the received data indicative of a status of a robotic device, wherein the associated data confidence indicator indicates the data is received; and

transmitting the further revised data indicative of a status of a robotic device and the associated data confidence indicator.

24. The article of manufacture of claim 23, wherein the data indicative of a status of a robotic device comprises an image or a video from a camera connected to the robotic device.

25. The article of manufacture of claim 23, wherein the data indicative of a status of a robotic device comprises data from sensors associated with a robot.

26. The article of manufacture of claim 25, wherein the data indicative of a status of a robotic device comprises a robot velocity.

27. The article of manufacture of claim 25, wherein the data indicative of a status of a robotic device comprises a robot position.

28. The article of manufacture of claim 23, wherein the display associated with the associated data confidence indicator includes a color, wherein the color changes based on the confidence indicator.