Methods and Systems for Augmented Reality to Display Virtual Representations of Robotic Device Actions

• US 20160055677A1
• Filed: 08/19/2015
• Published: 02/25/2016
• Est. Priority Date: 08/25/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Kuffner, James Joseph
• Granted Patent: US 10,896,543 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: B62D 57/032 with alternately or sequentially lifted supporting base and legs; with alternately or sequentially lifted feet or skid B62D57/024 takes precedence
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Example methods and systems for augmented reality interfaces to display virtual representations of robotic device actions are provided. An example method includes receiving information that indicates an action or an intent of a robotic device to perform a task, and the action or the intent includes one or more of a planned trajectory of the robotic device to perform at least a portion of the task and an object to be handled by the robotic device to perform at least a portion of the task. The method also includes providing, for display by a computing device on an augmented reality interface, a virtual representation of the action or the intent, and the virtual representation includes as annotations on the augmented reality interface at least a portion of the planned trajectory of the robotic device or highlighting the object to be handled by the robotic device.

Claims

1. A computer-implemented method comprising:

receiving information that indicates an action or an intent of a robotic device to perform a task, wherein the action or the intent includes one or more of a planned trajectory of the robotic device to perform at least a portion of the task and an object to be handled by the robotic device to perform at least a portion of the task; and

providing, for display by a computing device on an augmented reality interface, a virtual representation of the action or the intent, wherein the virtual representation includes as annotations on the augmented reality interface at least a portion of the planned trajectory of the robotic device or highlighting the object to be handled by the robotic device.

2. The method of claim 1, wherein the computing device is located remotely from the robotic device, and wherein providing, for display, comprises providing the virtual representation of the action or the intent overlaid onto a field of view of the computing device.

3. The method of claim 1, wherein the computing device is located remotely from the robotic device, and wherein providing, for display, comprises providing the virtual representation overlaid onto a live video feed of the computing device.

4. The method of claim 1, further comprising:

receiving a camera feed from a field of view of the robotic device; and

wherein providing, for display, comprises providing the virtual representation overlaid onto the camera feed from the field of view of the robotic device.

5. The method of claim 1, further comprising:

receiving, for display by the computing device on the augmented reality interface, a representation of an environment in which the robotic device resides including the robotic device in the environment; and

wherein providing, for display, comprises providing the virtual representation overlaid onto the representation of the environment in which the robotic device resides.

6. The method of claim 5, further comprising:

animating the virtual representation on the augmented reality interface to illustrate the robotic device performing the task according to the planned trajectory.

7. The method of claim 1, wherein the computing device includes a head mountable device (HMD), and wherein providing, for display, comprises providing the virtual representation overlaid onto a wearer-view of the HMD.

8. The method of claim 1, wherein the virtual representation includes a set of virtual footprints to represent future movement of the robotic device along the portion of the planned trajectory.

9. The method of claim 1, wherein the virtual representation includes a line on a floor covering the portion of the planned trajectory of the robotic device.

10. The method of claim 1, wherein the virtual representation includes an indication on the object where the robotic device plans to handle the object.

11. The method of claim 1, wherein receiving the information that indicates the action or the intent of the robotic device to perform the task comprises receiving the information before the robotic device initiates performance of the task.

12. The method of claim 1, further comprising:

determining one or more robotic devices in proximity to the computing device;

requesting, from the one or more robotic devices in proximity to the computing device, information that indicates actions or intents of the one or more robotic devices; and

wherein providing, for display by the computing device on the augmented reality interface, the virtual representation comprises providing for display one or more virtual representations per the one or more robotic devices that respectively indicate the actions or the intents.

13. The method of claim 1, further comprising:

determining a plurality of computing devices in proximity to the robotic device, wherein the computing device is one of the plurality of computing devices; and

providing, for display by the plurality of computing devices, the virtual representation of the action or the intent.

14. The method of claim 13, wherein the method is performed by the robotic device, and the method further comprises the robotic device broadcasting the information that indicates the action or the intent of the robotic device to the plurality of computing devices in proximity to the robotic device.

15. The method of claim 13, wherein the method is performed by the robotic device, and the method further comprises the robotic device projecting at least a portion of the virtual representation onto an environment of the robotic device.

16. The method of claim 1, further comprising:

providing by the computing device information to the robotic device to modify the action or the intent of the robotic device, wherein the information includes outputs of an accelerometer of the computing device based on motion of the computing device; and

in response, receiving from the robotic device one or more of an updated planned trajectory of the robotic device and an updated object to be handled by the robotic device.

17. The method of claim 1, wherein the computing device is a first computing device, and the method further comprises:

providing by a second computing device information to the robotic device to modify the action or the intent of the robotic device, wherein the information includes outputs of an accelerometer of the second computing device based on motion of the second computing device; and

in response, receiving by the first computing device from the robotic device one or more of an updated planned trajectory of the robotic device and an updated object to be handled by the robotic device.

18. The method of claim 1, further comprising:

providing, by a device registered with the robotic device, a projection of light along a pathway to modify the action or the intent of the robotic device; and

in response, receiving from the robotic device one or more of an updated planned trajectory of the robotic device and an updated object to be handled by the robotic device.

19. A non-transitory computer readable medium having stored thereon instructions that, upon execution by a computing device, cause the computing device to perform functions comprising:

receiving information that indicates an action or an intent of a robotic device to perform a task, wherein the action or the intent includes one or more of a planned trajectory of the robotic device to perform at least a portion of the task and an object to be handled by the robotic device to perform at least a portion of the task; and

providing, for display on an augmented reality interface, a virtual representation of the action or the intent, wherein the virtual representation includes as annotations on the augmented reality interface at least a portion of the planned trajectory of the robotic device or highlighting the object to be handled by the robotic device.

20. The non-transitory computer readable medium of claim 19, wherein:

receiving the information that indicates the action or the intent of the robotic device to perform the task comprises receiving the information before the robotic device initiates performance of the task; and

providing, for display, comprises providing the virtual representation of the action or the intent overlaid onto a representation of an environment in which the robotic device resides including the robotic device in the environment.

21. A computer-implemented method comprising:

receiving information that indicates an action or an intent of a robotic device to perform a task in a physical environment, wherein the action or the intent includes one or more of a planned trajectory of the robotic device to perform at least a portion of the task and an object to be handled by the robotic device to perform at least a portion of the task; and

providing, by a device into the physical environment, a virtual representation of the action or the intent, wherein the virtual representation includes an indication of at least a portion of the planned trajectory of the robotic device or highlighting the object to be handled by the robotic device.

22. The method of claim 21, wherein the device comprises a projector, and the method further comprises the projector projecting the virtual representation of the action or the intent overlaid onto the physical environment.

23. The method of claim 22, wherein the projector is included on the robotic device, and the method comprises the robotic device projecting the virtual representation of the action or the intent overlaid onto the physical environment before or during performance of the task.

24. The method of claim 21, wherein the device is located remotely from the robotic device, and the method comprises the device providing the virtual representation of the action or the intent overlaid onto a surface of the physical environment.

25. The method of claim 21, wherein the device comprises a projector, and the method further comprises the projector projecting a light onto the object to be handled by the robotic device.

26. The method of claim 21, wherein the virtual representation includes a set of virtual footprints to represent future movement of the robotic device along the portion of the planned trajectory.

27. The method of claim 21, wherein the virtual representation includes a line on a floor covering the portion of the planned trajectory of the robotic device.

28. The method of claim 21, wherein the virtual representation includes an indication on the object where the robotic device plans to handle the object.