Visualizing robot trajectory points in augmented reality

• US 9,919,427 B1
• Filed: 07/25/2015
• Issued: 03/20/2018
• Est. Priority Date: 07/25/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: X Development LLC (Alphabet Inc.)
• Inventors: Guilbert, Matthieu
• Primary Examiner(s): Mott, Adam R
• Application Number:
  US14/809,180
• Time in Patent Office: 969 Days
• CPC Class Codes: B25J 9/1664 characterised by motion, path, trajectory planning
• CPC Class Codes: G05B 2219/39449 Pendant, pda displaying camera images overlayed with graphics, augmented reality
• CPC Class Codes: G05B 2219/39451 Augmented reality for robot programming
• CPC Class Codes: G05B 2219/40512 Real time path planning, trajectory generation

Abstract

An example method includes receiving, from a sensor, an image of an environment, where the environment includes a robotic device. The method also includes determining, based on the received image of the environment, a pose of the robotic device relative to the sensor. The method further includes determining, based on the pose of the robotic device relative to the sensor, respective positions of a plurality of trajectory points for the robotic device relative to the sensor. The method additionally includes providing for display of the image of the environment and an overlaid plurality of virtual trajectory points corresponding to the plurality of trajectory points, where the plurality of virtual trajectory points are positioned in the image based on the determined respective positions of the plurality of trajectory points for the robotic device relative to the sensor.

Claims

1. A method, comprising:

providing, within an environment, (i) a robotic device having a first fiducial marker on a base of the robotic device and (ii) a second fiducial marker on a fixed object disconnected from the base of the robotic device within the environment;

referencing the second fiducial marker with the first fiducial marker to determine a pose of the second fiducial marker relative to the first fiducial marker;

receiving, from a sensor, an image of the environment, wherein the image includes the second fiducial marker and excludes the robotic device and the first fiducial marker;

determining, based on the image, a pose of the second fiducial marker relative to the sensor;

determining a pose of the robotic device relative to the sensor based on the pose of the second fiducial marker relative to the first fiducial marker and based on the pose of the second fiducial marker relative to the sensor;

determining, based on the pose of the robotic device relative to the sensor, respective positions of a plurality of trajectory points for the robotic device relative to the sensor; and

providing for display of the image of the environment and an overlaid plurality of virtual trajectory points corresponding to the plurality of trajectory points, wherein the plurality of virtual trajectory points are positioned in the image based on the determined respective positions of the plurality of trajectory points for the robotic device relative to the sensor.

2. The method of claim 1, further comprising receiving a second image from the sensor, wherein the second image includes at least a portion of the robot device, the method further comprising determining an additional pose of the robotic device relative to the sensor based on the portion of the robotic device within the second image.

3. The method of claim 2, wherein determining the additional pose of the robotic device relative to the sensor further comprises:

identifying the first fiducial marker on the base of the robotic device within the received second image;

determining, based on the received second image, a pose of the first fiducial marker relative to the sensor; and

determining the additional pose of the robotic device relative to the sensor based on the pose of the first fiducial marker relative to the sensor.

4. The method of claim 1, further comprising providing for display of an overlaid virtual representation of a trajectory for the robotic device, wherein the trajectory comprises a sequence of at least some of the plurality of trajectory points.

5. The method of claim 1, further comprising providing for display of a virtual simulation of the robotic device moving through the plurality of trajectory points.

6. The method of claim 1, further comprising providing for display of one or more virtual indications of one or more collision points between a trajectory that includes the plurality of trajectory points and one or more obstacles within the environment.

7. The method of claim 1, further comprising:

receiving the plurality of trajectory points from a trajectory point list, wherein the trajectory point list includes a plurality of semantic labels corresponding to the plurality of trajectory points; and

providing for display of the plurality of semantic labels at the determined respective positions of the plurality of corresponding trajectory points.

8. The method of claim 1, further comprising:

receiving, from an interface of a device that includes the sensor, input data indicating one or more adjustments to one or more positions of one or more of the plurality of virtual trajectory points in the image;

based on the received input data, determining one or more adjusted positions of one or more of the trajectory points that correspond to the one or more of the plurality of virtual trajectory points; and

providing an output signal indicating the one or more adjusted positions of the one or more of the trajectory points.

9. The method of claim 1, further comprising:

receiving, from an interface of a device that includes the sensor, input data indicating a selection of a particular one of the plurality of virtual trajectory points; and

providing instructions for the robotic device to move to one of the trajectory points that corresponds to the particular one of the plurality of virtual trajectory points.

10. The method of claim 1, wherein the fixed object containing the second fiducial marker comprises a supporting surface for one or more objects to be moved by the robotic device.

11. A system comprising:

a robotic device having a first fiducial marker on a base of the robotic device;

a second fiducial marker on a fixed object disconnected from the base of the robotic device;

a sensor; and

a computing system configured to;

receive, from the sensor, an image of an environment, wherein the image includes the second fiducial marker and excludes the robotic device and the first fiducial marker;

determine, based on the image, a pose of the second fiducial marker relative to the sensor;

determine a pose of the robotic device relative to the sensor based on the pose of the second fiducial marker relative to the first fiducial marker and based on the pose of the second fiducial marker relative to the sensor;

determine, based on the pose of the robotic device relative to the sensor, respective positions of a plurality of trajectory points for the robotic device relative to the sensor; and

provide for display of the image of the environment and an overlaid plurality of virtual trajectory points corresponding to the plurality of trajectory points, wherein the plurality of virtual trajectory points are positioned in the image based on the determined respective positions of the plurality of trajectory points for the robotic device relative to the sensor.

12. The system of claim 11, wherein the computing system is further configured to receive a second image from the sensor, wherein the second image includes at least a portion of the robot device, and wherein the computing system is further configured to determine an additional pose of the robotic device relative to the sensor based on the portion of the robotic device within the second image.

13. The system of claim 11, wherein the computing system is further configured to:

identify the first fiducial marker on the robotic device within a second received image;

determine, based on the second received image, a pose of the first fiducial marker relative to the sensor; and

determine an additional pose of the robotic device relative to the sensor based on the pose of the first fiducial marker relative to the sensor.

14. The system of claim 11, wherein the computing system is further configured to provide for display of one or more virtual indications of one or more collision points between a trajectory that includes the plurality of trajectory points and one or more obstacles within the environment.

15. The system of claim 11, wherein the fixed object containing the second fiducial marker comprises a supporting surface for one or more objects to be moved by the robotic device.

16. A non-transitory computer-readable medium having stored therein instructions, that when executed by a computing system, cause the computing system to perform functions comprising:

referencing a second fiducial marker with a first fiducial marker to determine a pose of the second fiducial marker relative to the first fiducial marker, wherein the first fiducial marker is provided on a base of a robotic device in an environment, and wherein the second fiducial marker is provided on on a fixed object disconnected from the base of the robotic device in the environment;

receiving, from a sensor, an image of the environment, wherein the image includes the second fiducial marker and excludes the robotic device and the first fiducial marker;

determining, based on the image, a pose of the second fiducial marker relative to the sensor;

determining a pose of the robotic device relative to the sensor based on the pose of the second fiducial marker relative to the first fiducial marker and based on the pose of the second fiducial marker relative to the sensor;

determining, based on the pose of the robotic device relative to the sensor, respective positions of a plurality of trajectory points for the robotic device relative to the sensor; and

providing for display of the image of the environment and an overlaid plurality of virtual trajectory points corresponding to the plurality of trajectory points, wherein the plurality of virtual trajectory points are positioned in the image based on the determined respective positions of the plurality of trajectory points for the robotic device relative to the sensor.

17. The non-transitory computer-readable medium of claim 16, wherein the functions further comprise receiving a second image from the sensor, wherein the second image includes at least a portion of the robot device, the functions further comprising determining an additional pose of the robotic device relative to the sensor based on the portion of the robotic device within the second image.

18. The non-transitory computer-readable medium of claim 16, wherein the functions further comprise:

identifying the first fiducial marker on the base of the robotic device within a second received image;

determining, based on the second received image, a pose of the first fiducial marker relative to the sensor; and

determining an additional pose of the robotic device relative to the sensor based on the pose of the first fiducial marker relative to the sensor.

19. The non-transitory computer-readable medium of claim 16, the functions further comprising:

receiving the plurality of trajectory points from a trajectory point list, wherein the trajectory point list includes a plurality of semantic labels corresponding to the plurality of trajectory points; and

providing for display of the plurality of semantic labels at the determined respective positions of the plurality of corresponding trajectory points.

20. The non-transitory computer-readable medium of claim 16, wherein the fixed object containing the second fiducial marker comprises a supporting surface for one or more objects to be moved by the robotic device.