VIRTUAL DISPLAY OF THE REAL-TIME POSITION OF A ROBOTIC DEVICE TO A HUMAN OPERATOR POSITIONED ON AN OPPOSING SIDE OF AN OBJECT

• US 20170075116A1
• Filed: 09/11/2015
• Published: 03/16/2017
• Est. Priority Date: 09/11/2015
• Status: Active Grant

• Current Assignee: The Boeing Co.
• Original Assignee: The Boeing Co.
• Inventors: Gardiner, Jim
• Granted Patent: US 9,964,765 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: Y10S 901/47 Optical

Abstract

Systems and methods for virtually displaying a robotic device to a human operator. An exemplary system includes an augmented reality headset for use by the human operator. The system includes a first imaging device having a field of view of a first side of an object, and tracks coordinate data for a first marker on the robotic device positioned on the first side. The system includes a second imaging device having a field of view of a second side of the object, and tracks coordinate data for a second marker on the augmented reality headset positioned on the second side. The system includes a controller that generates a virtual image of the robotic device in a coordinate system based on the coordinate data for the first and second markers, and provides the virtual image of the robotic device to the augmented reality headset for display to the human operator.

Claims

1. A system comprising:

an augmented reality headset for use by a human operator;

a first imaging device having a field of view of a first side of an object, and configured to track coordinate data for a first marker on a robotic device positioned on the first side of the object;

a second imaging device having a field of view of a second side of the object that opposes the first side, and configured to track coordinate data for a second marker on the augmented reality headset positioned on the second side of the object; and

a controller configured to generate a virtual image of the robotic device in a coordinate system based on the coordinate data for the first marker and the coordinate data for the second marker, and to provide the virtual image of the robotic device to the augmented reality headset for display to the human operator.

2. The system of claim 1 wherein:

the first marker comprises a first fiducial marker attached to an end effector of the robotic device; and

the first imaging device is configured to track the coordinate data for the first fiducial marker.

3. The system of claim 2 wherein:

the second marker comprises a second fiducial marker attached to the augmented reality headset; and

the second imaging device is configured to track the coordinate data for the second fiducial marker.

4. The system of claim 3 wherein:

the first fiducial marker and the second fiducial marker each include a plurality of references points having a Light Emitting Diode (LED).

5. The system of claim 1 wherein:

the controller includes;

a virtual data storage configured to store virtual data for different types of tooling on an end effector of the robotic device; and

a virtual image generator configured to identify tooling on the end effector, to retrieve the virtual data for the tooling from the virtual data storage, and to generate the virtual image of the robotic device based on the virtual data for the tooling.

6. The system of claim 5 wherein:

the virtual image generator is configured to read a Radio Frequency Identification (RFID) tag to identify the tooling on the end effector.

7. The system of claim 1 wherein:

the first imaging device includes;

at least one camera; and

a processor that uses photogrammetry to track the coordinate data for the first marker.

8. The system of claim 1 wherein:

the controller is configured to provide the coordinate data for the second marker on the augmented reality headset to the robotic device.

9. The system of claim 1 wherein:

the object comprises a component of an aircraft.

10. A method of virtually displaying a robotic device to a human operator, wherein the robotic device is positioned on a first side of an object and the human operator is positioned on a second side of the object which blocks the view of the human operator, the method comprising:

tracking, with a first imaging device, coordinate data for a first marker on the robotic device, wherein the first imaging device is positioned with a field of view of the first side of the object;

tracking, with a second imaging device, coordinate data for a second marker on an augmented reality headset worn by the human operator, wherein the second imaging device is positioned with a field of view of the second side of the object;

generating a virtual image of the robotic device in the coordinate system based on the coordinate data for the first marker and the coordinate data for the second marker; and

providing the virtual image of the robotic device to an augmented reality headset for display to the human operator.

11. The method of claim 10 wherein:

the first marker comprises a first fiducial marker attached to an end effector of the robotic device; and

tracking the coordinate data for the first marker comprises tracking the coordinate data for the first fiducial marker.

12. The method of claim 11 wherein:

the second marker comprises a second fiducial marker attached to the augmented reality headset; and

tracking the coordinate data for the second marker comprises tracking the coordinate data for the second fiducial marker.

13. The method of claim 12 wherein:

the first fiducial marker and the second fiducial marker each include a plurality of references points having a Light Emitting Diode (LED).

14. The method of claim 10 further comprising:

storing virtual data for different types of tooling on an end effector of the robotic device;

identifying tooling on the end effector;

retrieving the virtual data for the tooling; and

generating the virtual image based on the virtual data for the tooling.

15. The method of claim 14 wherein identifying tooling on the end effector comprises:

reading a Radio Frequency Identification (RFID) tag to identify the tooling on the end effector.

16. The method of claim 10 further comprising:

providing the coordinate data for the second marker on the augmented reality headset to the robotic device.

17. The method of claim 10 wherein:

the object comprises a component of an aircraft.

18. A positioning system for virtually displaying a robotic device to a human operator, the positioning system comprising:

an augmented reality headset for use by the human operator;

a first imaging device having a field of view of a first side of an object, and configured to track coordinate data for a first fiducial marker attached to an end effector of the robotic device that is positioned on the first side of the object;

a second imaging device having a field of view of a second side of the object that opposes the first side, and configured to track coordinate data for a second fiducial marker attached to the augmented reality headset, wherein the augmented reality headset is positioned on the second side of the object; and

a controller configured to generate a virtual image of the end effector of the robotic device in a coordinate system based on the coordinate data for the first fiducial marker and the coordinate data for the second fiducial marker, and to provide the virtual image of the end effector to the augmented reality headset for display to the human operator.

19. The positioning system of claim 18 wherein:

the controller includes;

a virtual data storage configured to store virtual data for different types of tooling on the end effector; and

a virtual image generator configured to identify tooling on the end effector, to retrieve the virtual data for the tooling from the virtual data storage, and to generate the virtual image based on the virtual data for the tooling.

20. The positioning system of claim 19 wherein:

the virtual image generator is configured to read a Radio Frequency Identification (RFID) tag to identify the tooling on the end effector.