Invisible fiducial markers on a robot to visualize the robot in augmented reality

• US 9,916,506 B1
• Filed: 07/25/2015
• Issued: 03/13/2018
• Est. Priority Date: 07/25/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: X Development LLC (Alphabet Inc.)
• Inventors: Davis, Benjamin
• Primary Examiner(s): Patel, Nirav G
• Application Number:
  US14/809,181
• Time in Patent Office: 962 Days
• CPC Class Codes: B25J 9/1664 characterised by motion, path, trajectory planning
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G02B 27/34 illuminated
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 2210/21 Collision detection, intersection
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/22 by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

An example system includes a robotic device, at least one invisible fiducial marker on the robotic device that is detectable within a particular band of invisible light, a camera that is configured to detect the particular band of invisible light, and a control system. The control system may be configured to receive an image of the robotic device within an environment, identify at least one position within the image of the at least one invisible fiducial marker on the robotic device, determine a position of the robotic device relative to the camera, determine one or more positions within the image to overlay one or more virtual annotations, and provide for display of the image of the robotic device within the environment with the one or more virtual annotations overlaid at the one or more determined positions within the image.

Claims

1. A system, comprising:

a robotic device;

a plurality of invisible fiducial markers positioned on a corresponding plurality of links of the robotic device, wherein the plurality of invisible fiducial markers are detectable within a particular band of invisible light;

a camera that is configured to detect the particular band of invisible light; and

a control system configured to;

receive, from the camera, an image of the robotic device within an environment;

identify respective positions within the image of the plurality of invisible fiducial marker on the robotic device;

based on the the respective positions within the image of the plurality of invisible fiducial markers on the robotic device, determine respective positions of the corresponding plurality of links of the robotic device relative to the camera;

based on the respective positions of the corresponding plurality of links of the robotic device relative to the camera, determine one or more positions within the image to overlay one or more virtual annotations; and

provide for display of the image of the robotic device within the environment with the one or more virtual annotations overlaid at the one or more determined positions within the image.

2. The system of claim 1, wherein the plurality of invisible fiducial markers are made up of ink visible within a particular band of ultraviolet light, and

wherein the camera is configured to detect the particular band of ultraviolet light.

3. The system of claim 1, wherein the plurality of invisible fiducial markers are made up of ink visible within a particular band of infrared light, and

wherein the camera is configured to detect the particular band of infrared light.

4. The system of claim 1, wherein the one or more virtual annotations comprise one or more virtual trajectory points that correspond to one or more trajectory points for the robotic device.

5. The system of claim 4, further comprising an invisible fiducial marker is positioned on a base of the robotic device, and wherein the control system is further configured to:

receive one or more positions of the one or more trajectory points for the robotic device relative to the base of the robotic device;

based on a position within the image of the invisible fiducial marker on the base of the robotic device, determine a position of the base of the robotic device relative to the camera; and

determine the one or more positions within the image to overlay the one or more virtual trajectory points based on the received one or more positions of the one or more trajectory points relative to the base of the robotic device and further based on the position of the base of the robotic device relative to the camera.

6. The system of claim 1, wherein the control system is further configured to:

based on the plurality of positions within the image of the plurality of invisible fiducial markers on the robotic device, determine a pose of the robotic device relative to the camera; and

based on the pose of the robotic device relative to the camera, determine the one or more positions within the image to overlay the one or more virtual annotations.

7. The system of claim 6, wherein the plurality of invisible fiducial markers comprise a first invisible fiducial marker positioned on a base of the robotic device and a second invisible fiducial marker positioned on an arm of the robotic device, wherein the control system is further configured to:

determine an angle from the camera between the first invisible fiducial marker and the second invisible fiducial marker; and

determine the pose of the robotic device relative to the camera based at least in part on the determined angle.

8. The system of claim 6, wherein the control system is further configured to:

determine a plurality of shapes within the image of the plurality of invisible fiducial markers on the robotic device; and

determine the pose of the robotic device relative to the camera based at least in part on the determined plurality of shapes within the image of the plurality of invisible fiducial markers on the robotic device.

9. The system of claim 1, wherein the control system is further configured to:

receive encoded joint angles of a plurality of joints of the robotic device that connect a plurality of links of the robotic device;

based on the position of the robotic device relative to the camera and further based on the encoded joint angles of the plurality of joints of the robotic device, determine respective positions of the plurality of links of the robotic device relative to the camera; and

based on the respective positions of the plurality of links of the robotic device relative to the camera, determine the one or more positions within the image to overlay the one or more virtual annotations.

10. A method, comprising:

receiving, from a camera that is configured to detect a particular band of invisible light, an image of a robotic device within an environment;

identifying at least one position within the image of at least one invisible fiducial marker on the robotic device, wherein the at least one invisible fiducial marker is detectable within the particular band of invisible light;

based on the at least one position within the image of the at least one invisible fiducial marker on the robotic device, determining a position of the robotic device relative to the camera;

based on the position of the robotic device relative to the camera, determining one or more positions within the image to overlay one or more virtual annotations, wherein the one or more virtual annotations comprise one or more virtual trajectory points that correspond to one or more trajectory points for the robotic device; and

providing for display of the image of the robotic device within the environment with the one or more virtual annotations overlaid at the one or more determined positions within the image.

11. The method of claim 10, wherein the at least one invisible fiducial marker is positioned on a base of the robotic device, the method further comprising:

receiving one or more positions of the one or more trajectory points for the robotic device relative to the base of the robotic device;

based on the at least one position within the image of the at least one invisible fiducial marker on the base of the robotic device, determining a position of the base of the robotic device relative to the camera; and

determining the one or more positions within the image to overlay the one or more virtual trajectory points based on the received one or more positions of the one or more trajectory points relative to the base of the robotic device and further based on the position of the base of the robotic device relative to the camera.

12. The method of claim 10, wherein the at least one invisible fiducial marker comprises a plurality of invisible fiducial markers, the method further comprising:

identifying a plurality of positions within the image of the plurality of invisible fiducial markers on the robotic device;

based on the plurality of positions within the image of the plurality of invisible fiducial markers on the robotic device, determining a pose of the robotic device relative to the camera; and

based on the pose of the robotic device relative to the camera, determining the one or more positions within the image to overlay the one or more virtual annotations.

13. The method of claim 10, wherein the at least one invisible fiducial marker comprises a plurality of invisible fiducial markers positioned on a corresponding plurality of links of the robotic device, the method further comprising:

identifying respective positions within the image of the plurality of invisible fiducial markers;

based on the respective positions within the image of the plurality of invisible fiducial markers, determining respective positions of the corresponding plurality of links of the robotic device relative to the camera; and

based on the respective positions of the corresponding plurality of links of the robotic device relative to the camera, determining the one or more positions within the image to overlay the one or more virtual annotations.

14. A non-transitory computer-readable medium having stored therein instructions, that when executed by a computing system, cause the computing system to perform functions comprising:

receiving, from a camera that is configured to detect a particular band of invisible light, an image of a robotic device within an environment;

identifying at least one position within the image of at least one invisible fiducial marker on the robotic device, wherein the at least one invisible fiducial marker is detectable within the particular band of invisible light;

based on the at least one position within the image of the at least one invisible fiducial marker on the robotic device, determining a position of the robotic device relative to the camera;

receiving encoded joint angles of a plurality of joints of the robotic device that connect a plurality of links of the robotic device;

based on the position of the robotic device relative to the camera and further based on the encoded joint angles of the plurality of joints of the robotic device, determining respective positions of the plurality of links of the robotic device relative to the camera;

based on the position of the robotic device relative to the camera and the respective positions of the plurality of links of the robotic device relative to the camera, determining one or more positions within the image to overlay one or more virtual annotations; and

providing for display of the image of the robotic device within the environment with the one or more virtual annotations overlaid at the one or more determined positions within the image.

15. The non-transitory computer-readable medium of claim 14, wherein the one or more virtual annotations comprise one or more virtual trajectory points that correspond to one or more trajectory points for the robotic device.

16. The non-transitory computer-readable medium of claim 15, wherein the at least one invisible fiducial marker is positioned on a base of the robotic device, the functions further comprising:

receiving one or more positions of the one or more trajectory points for the robotic device relative to the base of the robotic device;

based on the at least one position within the image of the at least one invisible fiducial marker on the base of the robotic device, determining a position of the base of the robotic device relative to the camera; and

determining the one or more positions within the image to overlay the one or more virtual trajectory points based on the received one or more positions of the one or more trajectory points relative to the base of the robotic device and further based on the position of the base of the robotic device relative to the camera.

17. The non-transitory computer-readable medium of claim 14, wherein the at least one invisible fiducial marker comprises a plurality of invisible fiducial markers, the functions further comprising:

identifying a plurality of positions within the image of the plurality of invisible fiducial markers on the robotic device;

based on the plurality of positions within the image of the plurality of invisible fiducial markers on the robotic device, determining a pose of the robotic device relative to the camera; and

based on the pose of the robotic device relative to the camera, determining the one or more positions within the image to overlay the one or more virtual annotations.