Robotic control system using virtual reality input

• US 9,579,799 B2
• Filed: 04/29/2015
• Issued: 02/28/2017
• Est. Priority Date: 04/30/2014
• Status: Active Grant

• Current Assignee: Coleman P. Parker
• Original Assignee: Coleman P. Parker
• Inventors: Parker, Coleman P.
• Primary Examiner(s): Sample, Jonathan L
• Application Number:
  US14/699,715
• Publication Number:
  US 20150314440A1
• Time in Patent Office: 671 Days
• US Class Current: 1/1
• CPC Class Codes: B25J 13/08 by means of sensing devices, e.g. viewing or touching devices
• CPC Class Codes: B25J 9/1612 characterised by the hand, wrist, grip control
• CPC Class Codes: B25J 9/163 learning, adaptive, model based, rule based expert control
• CPC Class Codes: B25J 9/1689 Teleoperation
• CPC Class Codes: G05B 2219/39543 Recognize object and plan hand shapes in grasping movements
• CPC Class Codes: G05B 2219/40053 Pick 3-D object from pile of objects
• CPC Class Codes: G05B 2219/40131 Virtual reality control, programming of manipulator

Abstract

A robotic system and method performs tasks autonomously on real-world objects (RWOs) upon receipt of inputs from a virtual reality environment. A robotic actuator has tools to manipulate the RWOs, and a set of robot-specific instructions for actuating the tools. A first set of rules governs the RWOs, and sensors detect the presence of the RWOs. A virtual reality (VR) computer uses sensor data to generate a virtual world (VW) including virtual world objects (VWOs) representative of RWOs. A second set of rules governs the VWOs. A user manipulates the VWOs to generate a modified virtual world (MVW). A transformation engine captures differences between the VW and MVW, transforms the differences into inputs for the robotic actuator. An autonomous instruction engine receives the inputs to identify RWOs and combines the inputs with the first set of rules to generate combined instructions usable by the robotic actuator to manipulate RWOs autonomously.

Claims

1. A robotic system for performing tasks autonomously on real-world objects (RWOs) upon receipt of inputs from a virtual reality environment, the system comprising:

a processor-controlled robotic actuator having tools configured to manipulate real-world objects (RWOs), the robotic actuator including a memory having a set of robot-specific instructions stored thereon, the robotic actuator further including a processor configured for executing the robot-specific instructions for actuating the tools;

one or more data stores including a first set of rules governing the RWOs;

one or more sensors configured to detect presence of the RWOs and to generate data corresponding to said presence;

a virtual reality (VR) computer communicably coupled to the robotic actuator and to the one or more sensors, the VR computer configured to capture the data and to use the data to generate a virtual world (VW) including virtual world objects (VWOs) representative of the RWOs;

the one or more data stores including a second set of rules governing the VWOs;

the VR computer including a user interface permitting a user to interact with the VW in accordance with the second set of rules, to manipulate the VWOs to achieve desired VW objectives, to generate a modified virtual world (MVW);

a transformation engine configured to capture differences between the VW and the MVW, to transform the differences into inputs usable by the robotic actuator, and to transmit the inputs to the robotic actuator; and

the robotic actuator including an autonomous instruction engine configured to receive the inputs to identify RWOs of interest and to combine the inputs with the first set of rules to generate a set of combined instructions usable by the robotic actuator to selectively execute individual ones of the robot-specific instructions to interact with the particular ones of the RWOs autonomously to achieve desired real-world (RW) objectives, wherein the VW objectives are independent of the RW objectives.

2. The system of claim 1, wherein the transformation engine is configured to capture differences between the VW and the MVW to identify VWOs selected by the user, and to transform the identified VWOs into inputs usable by the robotic actuator.

3. The system of claim 2, wherein the transformation engine is configured to transform the differences into robotic control instructions in the form of pick up item of a particular type at a particular location, x,y.

4. The system of claim 2, wherein the autonomous instruction engine is configured to receive the inputs in the form of the user identified VWOs to identify corresponding RWOs of interest.

5. The system of claim 4, wherein the autonomous instruction engine is configured to combine the inputs with the first set of rules to specify desired end-results of manipulation of the RWOs, to generate a set of combined instructions usable by the robotic actuator.

6. The system of claim 1, wherein the first set of rules comprise mathematical laws, physical laws, bio-chemical laws, and combinations thereof.

7. The system of claim 6, wherein the first set of rules comprise:

desired locations/bins for particular RWOs;

physical properties of the RWOs, including materials from which the RWOs are constructed;

safety considerations associated with handling of the RWOs, including hazards;

value of the RWO object, including rules governing the value of the object based on color, material type, quality, soil level;

harm induced by the RWO if it were burned, buried, submerged, e.g. (Original), in pollute produced in ppm (parts per million);

size of the RWOs, optimal pickup location for the RWOs; and

mixtures and combinations thereof.

8. The system of claim 1, wherein the second set of rules comprise colorization of objects, sounds and music associated with objects or actions, animations or videos associated with objects or actions, and combinations thereof.

9. The system of claim 8, wherein the second set of rules comprise:

colorization of objects based on value, material type, material grade, relative difficulty in grasping with the robot, rules about ppm of pollutants allowed to be un recovered;

rules about the percent (%) of material by type to be recovered to meet different goals;

rules about the value of materials recovered and the goal of the value of the recovered materials;

sounds and music associated with the appearance of certain objects, such as with the collected objects or of desired objects that were “

missed”

by the user;

sounds associated with certain actions which are being “

rewarded” and

sounds associated with actions being discouraged, such as an incomplete identification of an object, or failure to identify a predetermined number of objects per minute, as determined by the rules governing the goals;

animations or videos associated with new actions for the user to learn, examples of items the user missed in previous sessions, new items of particular value or harm, or significant changes in the value or harm of a previously known item;

instructions for how to solve a problem when presented with broken VWOs or too many items on the screen at one time;

goals for the player, such as remove as much as you can, remove the most hazardous VWOs first; and

combinations thereof.

10. The system of claim 1, wherein the robot-specific instructions include rules governing type and/or size of tool to use, the force with which the tool should grasp the particular RWO, the speed at which the tool should be moved in order to intercept the RWO, how much to close the tool to successfully pickup different sized RWOs, and combinations thereof.

11. The system of claim 1, wherein the autonomous instruction engine is communicably coupled to the robotic actuator.

12. The system of claim 1, wherein the robot is configured to interact with the RWOs autonomously by inserting the combined instructions into one or more of a plurality of preprogrammed instruction sets to synchronize the MVW with desired impacts on the RWOs.

13. The system of claim 1, wherein the set of robot-specific instructions are disposed in firmware.

14. The system of claim 1, wherein the sensor comprises an electromagnetic sensor.

15. The system of claim 14, wherein the electromagnetic sensor comprises an image capture device.

16. The system of claim 1, further comprising a data store including a third set of rules governing physical aspects of the VW, including light, visual and non-visual bands, sounds, temperature, weight, relative movement, and combinations thereof.

17. The system of claim 1, wherein the robotic actuator is configured to selectively execute the individual ones of the robot-specific instructions to interact with the particular ones of the RWOs autonomously and asynchronously relative to user interaction with the VW.

18. A method for performing tasks autonomously on real-world objects (RWOs) upon receipt of inputs from a virtual reality environment using the system of claim 1, the method comprising:

(a) detecting, with the one or more sensors, presence of the RWOs and generating data corresponding to said presence;

(b) capturing and using the data, with the VR computer, to generate the VW including VWOs representative of the RWOs;

(c) enabling a user, via the user interface, to manipulate the VWOs to generate a modified virtual world (MVW);

(d) capturing, with the transformation engine, differences between the VW and the MVW to transform the differences into inputs usable by the robotic actuator, and transmitting the inputs from the transformation engine to the robotic actuator;

(e) receiving inputs at the autonomous instruction engine of the robotic actuator to identify RWOs of interest and with the autonomous instruction engine, combining the inputs with the first set of rules to generate a set of combined instructions usable by the robotic actuator to selectively execute individual ones of the robot-specific instructions to interact with the particular ones of the RWOs autonomously; and

(f) manipulating the RWOs with the robotic actuator.