Visual annotations in robot control interfaces

• US 9,895,809 B1
• Filed: 08/20/2015
• Issued: 02/20/2018
• Est. Priority Date: 08/20/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: X Development LLC (Alphabet Inc.)
• Inventors: Moore, Kyle, Dureau, Vincent, Rajkumar, Nareshkumar
• Primary Examiner(s): Tran, Khoi
• Assistant Examiner(s): Nguyen, Robert
• Application Number:
  US14/831,549
• Time in Patent Office: 915 Days
• CPC Class Codes: B25J 11/0005 Manipulators having means for high-level communication with users, e.g. speech generator, face recognition means
• CPC Class Codes: B25J 13/006 by means of a wireless system for controlling one or several manipulators
• CPC Class Codes: B25J 13/06 Control stands, e.g. consoles, switchboards
• CPC Class Codes: B25J 13/08 by means of sensing devices, e.g. viewing or touching devices
• CPC Class Codes: B25J 19/023 including video camera means
• CPC Class Codes: B25J 5/00 Manipulators mounted on wheels or on carriages B25J1/00 takes precedence; programme-controlled manipulators B25J9/00 ; vehicle aspects B60, B62, e.g. remote-controlled steering for motor vehicles B62D1/24; control of position of vehicles G05D1/00
• CPC Class Codes: B25J 9/1689 Teleoperation
• CPC Class Codes: G05B 2219/40099 Graphical user interface for robotics, visual robot user interface

Abstract

Methods, apparatus, systems, and computer-readable media are provided for visually annotating rendered multi-dimensional representations of robot environments. In various implementations, an entity may be identified that is present with a telepresence robot in an environment. A measure of potential interest of a user in the entity may be calculated based on a record of one or more interactions between the user and one or more computing devices. In some implementations, the one or more interactions may be for purposes other than directly operating the telepresence robot. In various implementations, a multi-dimensional representation of the environment may be rendered as part of a graphical user interface operable by the user to control the telepresence robot. In various implementations, a visual annotation may be selectively rendered within the multi-dimensional representation of the environment in association with the entity based on the measure of potential interest.

Claims

1. A method, comprising:

identifying, by one or more processors, an entity that is present with a telepresence robot in an environment;

calculating, by the one or more processors, based on a record of one or more interactions between a user and one or more computing devices, a measure of potential interest of the user in the entity, wherein the record of one or more interactions between the user and the one or more computing devices comprises a calendar maintained by the user using the one or more computing devices, wherein the one or more interactions are for purposes other than directly operating the telepresence robot;

rendering, by the one or more processors, as part of a graphical user interface operable by the user to control the telepresence robot, a multi-dimensional representation of the environment; and

selectively rendering, by the one or more processors, based on the measure of potential interest, a visual annotation associated with the entity within the multi-dimensional representation of the environment.

2. The method of claim 1, wherein the selectively rendering comprises rendering the visual annotation with a visual attribute that is selected based on the measure of potential interest.

3. The method of claim 1, wherein the entity comprises a person.

4. The method of claim 3, wherein identifying the entity comprises identifying the person using facial recognition based on a signal provided by the telepresence robot.

5. The method of claim 3, wherein identifying the entity comprises identifying the person based on detection, by the telepresence robot, of a communication mechanism carried by the person.

6. The method of claim 3, wherein identifying the entity comprises identifying the person based at least in part on the record of one or more interactions between the user and the one or more computing devices.

7. The method of claim 1, wherein the multi-dimensional representation of the environment comprises a live video feed provided by the robot, and wherein rendering the visual annotation comprises overlaying the visual annotation on the live video feed.

8. The method of claim 1, wherein the multi-dimensional representation of the environment comprises a live map view of the environment, and wherein rendering the visual annotation comprises overlaying the visual annotation on the live map view.

9. The method of claim 1, wherein the record of one or more interactions between the user and one or more computing devices comprises content consumed by the user using the one or more computing devices.

10. The method of claim 1, wherein the record of one or more interactions between the user and the one or more computing devices further comprises information associated with social network activity of the user.

11. A system including memory and one or more processors operable to execute instructions stored in the memory, comprising instructions to:

identify an entity that is present with a telepresence robot in an environment;

calculate, based on a record of one or more interactions between a user and one or more computing devices, a measure of potential interest of the user in the entity, wherein the record of one or more interactions between the user and the one or more computing devices comprises a calendar maintained by the user using the one or more computing devices, wherein the one or more interactions are for purposes other than directly operating the telepresence robot;

render, as part of a graphical user interface operable by a user to control the telepresence robot, a multi-dimensional representation of the environment; and

selectively render, based on the measure of potential interest, a visual annotation associated with the entity within the multi-dimensional representation of the environment.

12. The system of claim 11, further comprising instructions to render the visual annotation with a visual attribute that is selected based on the measure of potential interest.

13. The system of claim 11, wherein the entity comprises a person.

14. The system of claim 13, further comprising instructions to identify the person using facial recognition based on a signal provided by the telepresence robot.

15. The system of claim 13, further comprising instructions to identify the person based on detection, by the telepresence robot, of a communication mechanism carried by the person.

16. The system of claim 13, further comprising instructions to identify the person based at least in part on the record of one or more interactions between the user and the one or more computing devices.

17. The system of claim 11, wherein the multi-dimensional representation of the environment comprises a live video feed provided by the robot, and wherein the system further comprises instructions to overlay the visual annotation on the live video feed.

18. The system of claim 11, wherein the record of one or more interactions between the user and the one or more computing devices further comprises content items created, received, or otherwise consumed by the user using the one or more computing devices.

19. At least one non-transitory computer-readable medium comprising instructions that, in response to execution of the instructions by a computing device, cause the computing device to perform the following operations:

identifying an entity in an environment based on a record of one or more interactions between a user and one or more computing devices, wherein the record of one or more interactions between the user and the one or more computing devices comprises information associated with social network activity of the user, wherein the one or more interactions are for purposes other than directly operating a telepresence robot;

calculating, based on the record of one or more interactions between the user and the one or more computing devices, a measure of potential interest of the user in the entity;

rendering, as part of a graphical user interface operable by the user to control the telepresence robot, a multi-dimensional representation of the environment; and

selectively rendering, based on the measure of potential interest, a visual annotation associated with the entity within the multi-dimensional representation of the environment.