ROBOT CONTROL, TRAINING AND COLLABORATION IN AN IMMERSIVE VIRTUAL REALITY ENVIRONMENT

• US 20160257000A1
• Filed: 03/04/2015
• Published: 09/08/2016
• Est. Priority Date: 03/04/2015
• Status: Active Grant

• Current Assignee: Johns Hopkins University
• Original Assignee: Johns Hopkins University
• Inventors: Guerin, Kelleher, Hager, Gregory D.
• Granted Patent: US 9,643,314 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: B25J 9/1605 Simulation of manipulator lay-out, design, modelling of manipulator
• CPC Class Codes: B25J 9/1671 characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems
• CPC Class Codes: G05B 2219/40131 Virtual reality control, programming of manipulator
• CPC Class Codes: G05B 2219/45097 Cable harnessing robot
• CPC Class Codes: Y10S 901/05 Machine driven lead through

Abstract

System and methods to create an immersive virtual environment using a virtual reality system that receives parameters corresponding to a real-world robot. The real-world robot may be simulated to create a virtual robot based on the received parameters. The immersive virtual environment may be transmitted to a user. The user may supply input and interact with the virtual robot. Feedback such as the current state of the virtual robot or the real-world robot may be provided to the user. The user may train the virtual robot. The real-world robot may be programmed based on the virtual robot training.

Claims

1. A method for programming a robot comprising:

creating an immersive virtual environment (IVE) using a virtual reality system (VRS);

receiving, by the VRS, parameters corresponding to a real-world robot;

creating, by the VRS within said IVE, a virtual robot, wherein the virtual robot is a simulation of the real-world robot based on the received parameters;

transmitting, by the VRS, a representation of said IVE to a user;

receiving, by the VRS, input from the user, wherein said VRE is configured to allow the user to interact with the virtual robot using said user input;

providing, by the VRS within said IVE, robot feedback to the user, wherein said robot feedback includes a current state of the virtual robot;

training, in the VRS, the virtual robot in the IVE by the user; and

programming, by the VRS, the real-world robot based on the virtual robot training.

2. The method of claim 1, wherein said user interaction includes an egocentric perspective, wherein the user operates the virtual robot from a point of view of the robot, and a robot'"'"'s end effector directly follows a motion of a hand of the user.

3. The method of claim 2, wherein an image of an arm of the robot is overlaid where an arm of the user would be in real life.

4. The method of claim 1, wherein said user interaction includes an exocentric perspective, where the user views the virtual robot inside the IVE from a point of view external to the robot.

5. The method of claim 1, wherein said user interaction includes controlling the real-world robot in real-time.

6. The method of claim 1, wherein said IVE includes an augmented environment.

7. The method of claim 1, wherein said IVE includes a virtual reality environment.

8. The method of claim 1, wherein the virtual robot is programmed with a series of gestures.

9. The method of claim 1, wherein the virtual robot is programmed by using a virtual user interface.

10. The method of claim 1, wherein the virtual robot is programmed with a series of motions.

11. The method of claim 10, wherein the virtual robot replays the series of motions before the series of motions are programmed into the real-world robot.

12. The method of claim 10, wherein a plurality of real-world robots are programmed with the series of motions.

13. The method of claim 1, wherein said robot feedback is supplied via a virtual information display (VID), wherein the virtual information display is one of:

a 2D sprite or a 3D mesh, and the VID is configurable to be textured or shaded.

14. The method of claim 13, wherein the VID is configurable to be at least one of:

locked to the virtual robot, locked to a view of the user, or locked to an avatar of the user.

15. The method of claim 1, wherein the IVE includes mixed reality user interfaces (UIs), wherein the UIs are configured to be at least one of:

adjusted by the user, locked to an avatar of the user, locked to the virtual robot, or locked to a view of the user.

16. The method of claim 1, wherein the user interacts with the IVE using at least one of:

a stereoscopic 3D display, a virtual reality headset, or an augmented reality device.

17. The method of claim 1, wherein the real-world robot includes one of:

an industrial robot, a domestic robot, an articulated welding robot, an autonomous robot, a military robot, or a medical robot.

18. The method of claim 1, wherein the real-world robot is one of:

remotely operated a distance from the user, located in a hazardous environment, or dangerous for the user to be in close proximity.

19. The method of claim 1, wherein said virtual robot controls a plurality of real-world robots.

20. The method of claim 1, wherein said IVE is configurable to define geometric primitives or detailed 3D mesh objects as overlays of real time sensing data to define geometric regions for interaction for the real-world robot.

21. The method of claim 1, wherein said IVE is configurable to define virtual constraints for the virtual robot, wherein said virtual constraints are translated physical constraints imposed on the real-world robot.

22. The method of claim 1, wherein said IVE is configurable to display real-time 2D or 3D video or depth information on a virtual scene overlaid in a geometrically correct manner on any 3D representation in the IVE.

23. The method of claim 1, wherein said IVE is configurable to display forces applied by the robot, or other objects, as sensed by the real-world robot.

24. The method of claim 1, wherein said IVE is configurable to display a real time representation of a real life environment with an ability to seamlessly transition between the real life environment and a virtual environment.

25. The method of claim 1, wherein a plurality of users can interact with the IVE.

26. The method of claim 1, further comprising:

training said user using IVE on said real-world robot using said virtual robot.

27. The method of claim 26, wherein said IVE is used as a sandbox for learning robot programming or task plan specification using said virtual robot.

28. A programming system comprising:

a dexterous machine with at least one articulating arm;

a processor operable to perform instructions to;

create an immersive virtual environment (IVE);

receive parameters corresponding to the dexterous machine;

create in said IVE a virtual representation of said dexterous machine based on the parameters;

transmit said IVE to a user;

receive input from the user, wherein said input includes interactions of the user with objects within the IVE;

providing in said IVE, feedback from said dexterous machine, wherein said feedback includes a current state of said dexterous machine; and

transmit programming instructions to said dexterous machine.