Drone Assisted Adaptive Robot Control

• US 20160271796A1
• Filed: 03/18/2016
• Published: 09/22/2016
• Est. Priority Date: 03/19/2015
• Status: Active Grant

• Current Assignee: Rahul Babu
• Original Assignee: Rahul Babu
• Inventors: Babu, Rahul
• Granted Patent: US 9,855,658 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: B05D 1/02 performed by spraying
• CPC Class Codes: B25J 11/0075 Manipulators for painting or coating
• CPC Class Codes: B25J 9/1664 characterised by motion, path, trajectory planning
• CPC Class Codes: B25J 9/1666 Avoiding collision or forbidden zones
• CPC Class Codes: B25J 9/1674 characterised by safety, monitoring, diagnostic
• CPC Class Codes: B25J 9/1694 characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G01B 21/04 by measuring coordinates of points
• CPC Class Codes: G05B 2219/37555 Camera detects orientation, position workpiece, points of workpiece
• CPC Class Codes: G05B 2219/39016 Simultaneous calibration of manipulator and camera
• CPC Class Codes: G05B 2219/40005 Vision, analyse image at one station during manipulation at next station
• CPC Class Codes: G05B 2219/40425 Sensing, vision based motion planning
• CPC Class Codes: G05B 2219/40504 Simultaneous trajectory and camera planning
• CPC Class Codes: G05B 2219/40604 Two camera, global vision camera, end effector neighbourhood vision camera
• CPC Class Codes: G05B 2219/40609 Camera to monitor end effector as well as object to be handled
• CPC Class Codes: G05B 2219/40617 Agile eye, control position of camera, active vision, pan-tilt camera, follow object
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target

Abstract

A method, a drone device, and an adaptive robot control system (ARCS) for adaptively controlling a programmable robot are provided. The ARCS receives environmental parameters of a work environment where the drone device operates and geometrical information of a target object to be operated on by the programmable robot. The ARCS dynamically receives a calibrated spatial location of the target object in the work environment based on the environmental parameters and a discernment of the target object from the drone device. The ARCS determines control information including parts geometry of the target object, a task trajectory of a task to be performed on the target object, and a collision-free robotic motion trajectory for the programmable robot, and dynamically transmits the control information to the programmable robot via a communication network to adaptively control the programmable robot while accounting for misalignments of the target object in the work environment.

Claims

1. A method for adaptively controlling a programmable robot, in communication with a drone device, the method employing an adaptive robot control system executable by at least one processor configured to execute computer program instructions for performing the method comprising:

receiving a plurality of environmental parameters of a work environment where the drone device operates, by the adaptive robot control system;

receiving geometrical information of a target object to be operated on by the programmable robot positioned in the work environment, by the adaptive robot control system;

dynamically receiving a calibrated spatial location of the target object in the work environment based on the environmental parameters of the work environment where the target object is positioned and a discernment of the target object from the drone device by the adaptive robot control system, wherein the drone device is configured to dynamically record and calibrate a spatial location of the target object in the work environment by navigating independently in a plurality of directions around the target object, accessing inaccessible areas with respect to the target object in the work environment, and capturing one or more images of the target object in each of the directions;

determining parts geometry of the target object by the adaptive robot control system based on the dynamically received calibrated spatial location of the target object, the received geometrical information of the target object, and predetermined dimensions of a plurality of objects stored in a parts geometry database;

determining, by the adaptive robot control system, a task trajectory of a task to be performed on the target object by the programmable robot based on the determined parts geometry of the target object, the predetermined dimensions of the objects stored in the parts geometry database, and predetermined task trajectories of the target objects stored in a procedure database;

determining a collision-free robotic motion trajectory for the programmable robot by the adaptive robot control system based on the determined task trajectory, a model of the target object extracted from the determined parts geometry, and the environmental parameters of the work environment; and

dynamically transmitting the dynamically received calibrated spatial location of the target object, the determined parts geometry, the determined task trajectory, and the determined collision-free robotic motion trajectory to the programmable robot by the adaptive robot control system via a communication network to adaptively control the programmable robot to perform the task on the target object while accounting for misalignments of the target object in the work environment.

2. The method of claim 1, further comprising identifying contours of the target object from the determined parts geometry of the target object and determining configurable positions for a component of the programmable robot with respect to the identified contours of the target object by the adaptive robot control system for adaptively controlling the programmable robot to move the component of the programmable robot to the determined configurable positions with respect to the identified contours of the target object to perform the task on the identified contours of the target object.

3. The method of claim 1, further comprising determining volumetric data associated with the task to be performed on the target object, by the adaptive robot control system based on the determined parts geometry of the target object.

4. The method of claim 1, further comprising determining a variance in one of the dynamically received calibrated spatial location of the target object, the determined parts geometry of the target object, and a combination thereof by the adaptive robot control system based on the predetermined dimensions of the objects stored in the parts geometry database for offsetting the misalignments of the target object in the work environment.

5. The method of claim 1, wherein the environmental parameters comprise a boundary, an area, and dimensions of the work environment.

6. The method of claim 1, wherein the adaptive robot control system is configured to receive the environmental parameters of the work environment and the geometrical information of the target object from one of a direct entry through a graphical user interface and one or more of three-dimensional cameras and a plurality of sensors deployed in the drone device via the communication network, wherein the sensors comprise image sensors, optical sensors, position sensors, inertial sensors, and motion sensors.

7. A method for adaptively controlling a programmable robot, the method employing a drone device comprising at least one processor configured to execute computer program instructions for performing the method comprising:

dynamically capturing a plurality of images of a target object to be operated on in a work environment from a plurality of directions by the drone device using one or more three-dimensional cameras configured in the drone device, while navigating independently in the directions around the target object in the work environment and accessing inaccessible areas with respect to the target object in the work environment;

recording a spatial location of the target object by the drone device using the dynamically captured images of the target object in the work environment and sensor data obtained from one or more of a plurality of sensors configured in the drone device;

dynamically calibrating the recorded spatial location of the target object in communication with the one or more three-dimensional cameras and the one or more of the sensors by the drone device based on environmental parameters of the work environment and a discernment of the target object;

determining parts geometry of the target object by the drone device based on the dynamically calibrated spatial location of the target object, geometrical information of the target object, and predetermined dimensions of a plurality of objects stored in a parts geometry database;

determining, by the drone device, a task trajectory of a task to be performed on the target object by the programmable robot based on the determined parts geometry of the target object, the predetermined dimensions of the objects stored in the parts geometry database, and predetermined task trajectories of the target objects stored in a procedure database;

determining a collision-free robotic motion trajectory for the programmable robot by the drone device based on the determined task trajectory, a model of the target object extracted from the determined parts geometry, and the environmental parameters of the work environment; and

dynamically transmitting the dynamically calibrated spatial location of the target object, the determined parts geometry, the determined task trajectory, and the determined collision-free robotic motion trajectory to the programmable robot by the drone device via a communication network to adaptively control the programmable robot to perform the task on the target object while accounting for misalignments of the target object in the work environment.

8. The method of claim 7, further comprising preparing one or more maps of the work environment by the drone device using the one or more three-dimensional cameras and the one or more of the sensors configured in the drone device.

9. The method of claim 7, further comprising obtaining the environmental parameters of the work environment by the drone device from the one or more three-dimensional cameras and the one or more of the sensors deployed in the drone device, wherein the environmental parameters comprise a boundary, an area, and dimensions of the work environment, and wherein the sensors comprise image sensors, optical sensors, position sensors, inertial sensors, and motion sensors.

10. The method of claim 7, further comprising obtaining the geometrical information of the target object by the drone device from the dynamically captured images of the target object.

11. The method of claim 7, further comprising comparing, by the drone device, the target object encountered during the navigation of the drone device with known objects stored in the drone device to obtain the discernment of the target object.

12. The method of claim 7, further comprising identifying contours of the target object from the determined parts geometry of the target object and determining configurable positions for a component of the programmable robot with respect to the identified contours of the target object by the drone device for adaptively controlling the programmable robot to move the component of the programmable robot to the determined configurable positions with respect to the identified contours of the target object to perform the task on the identified contours of the target object.

13. The method of claim 7, further comprising determining volumetric data associated with the task to be performed on the target object, by the drone device based on the determined parts geometry of the target object.

14. The method of claim 7, further comprising determining a variance in one of the dynamically calibrated spatial location of the target object, the determined parts geometry of the target object, and a combination thereof by the drone device based on the predetermined dimensions of the objects stored in the parts geometry database for offsetting the misalignments of the target object in the work environment by the drone device.

15. A system for adaptively controlling a programmable robot, the system comprising:

a drone device comprising;

one or more three-dimensional cameras mounted thereon for dynamically capturing a plurality of images of a target object to be operated on in a work environment from a plurality of directions, while the drone device navigates independently in the directions around the target object in the work environment and accesses inaccessible areas with respect to the target object in the work environment; and

one or more sensors for detecting a configuration of the work environment and characteristics, positioning, and movements of the target object; and

at least one processor configured to execute computer program instructions defined by modules deployed in one of the drone device and in an electronic device positioned external to and in operable communication with the drone device via a communication network, the modules comprising;

a location recorder deployed in the drone device, the location recorder configured to record a spatial location of the target object using the dynamically captured images of the target object in the work environment and sensor data obtained from the one or more sensors configured in the drone device;

a spatial location calibration module deployed in the drone device, the spatial location calibration module configured to dynamically calibrate the recorded spatial location of the target object in communication with the one or more three-dimensional cameras and the one or more sensors based on environmental parameters of the work environment and a discernment of the target object;

a parts geometry determination module deployed in the one of the drone device and the electronic device, the parts geometry determination module configured to determine parts geometry of the target object based on the dynamically calibrated spatial location of the target object, geometrical information of the target object, and predetermined dimensions of a plurality of objects stored in a parts geometry database;

a task trajectory determination module deployed in the one of the drone device and the electronic device, the task trajectory determination module configured to determine a task trajectory of a task to be performed on the target object by the programmable robot based on the determined parts geometry of the target object, the predetermined dimensions of the objects stored in the parts geometry database, and predetermined task trajectories of the target objects stored in a procedure database;

a motion trajectory determination module deployed in the one of the drone device and the electronic device, the motion trajectory determination module configured to determine a collision-free robotic motion trajectory for the programmable robot based on the determined task trajectory, a model of the target object extracted from the determined parts geometry, and the environmental parameters of the work environment; and

a data communication module deployed in the one of the drone device and the electronic device, the data communication module configured to dynamically transmit the dynamically calibrated spatial location of the target object, the determined parts geometry, the determined task trajectory, and the determined collision-free robotic motion trajectory to the programmable robot via the communication network to adaptively control the programmable robot to perform the task on the target object while accounting for misalignments of the target object in the work environment.

16. The system of claim 15, wherein the location recorder is further configured to prepare one or more maps of the work environment using the one or more three-dimensional cameras and the one or more sensors configured in the drone device.

17. The system of claim 15, wherein the spatial location calibration module and the data communication module are further configured to obtain the environmental parameters of the work environment from the one or more sensors deployed in the drone device, wherein the environmental parameters comprise a boundary, an area, and dimensions of the work environment, and wherein the sensors comprise image sensors, optical sensors, position sensors, inertial sensors, and motion sensors.

18. The system of claim 15, wherein the parts geometry determination module is further configured to obtain the geometrical information of the target object from the dynamically captured images of the target object.

19. The system of claim 15, wherein the spatial location calibration module is further configured to compare the target object encountered during the navigation of the drone device with known objects stored in the drone device to obtain the discernment of the target object.

20. The system of claim 15, wherein the parts geometry determination module is further configured to identify contours of the target object from the determined parts geometry of the target object, and wherein the task trajectory determination module is further configured to determine configurable positions for a component of the programmable robot with respect to the identified contours of the target object for adaptively controlling the programmable robot to move the component of the programmable robot to the determined configurable positions with respect to the identified contours of the target object to perform the task on the identified contours of the target object.

21. The system of claim 15, wherein the parts geometry determination module is further configured to determine volumetric data associated with the task to be performed on the target object, based on the determined parts geometry of the target object.

22. The system of claim 15, wherein the parts geometry determination module is further configured to determine a variance in one of the dynamically calibrated spatial location of the target object, the determined parts geometry of the target object, and a combination thereof based on the predetermined dimensions of the objects stored in the parts geometry database for offsetting the misalignments of the target object in the work environment.

23. A non-transitory computer readable storage medium having embodied thereon, computer program codes comprising instructions executable by at least one processor for adaptively controlling a programmable robot, the computer program codes comprising:

a first computer program code for processing a plurality of images of a target object to be operated on in a work environment, the images of the target object dynamically captured by a drone device from a plurality of directions using one or more three-dimensional cameras configured in the drone device, while navigating independently in the directions around the target object in the work environment and accessing inaccessible areas with respect to the target object in the work environment;

a second computer program code for recording a spatial location of the target object using the dynamically captured images of the target object in the work environment and sensor data obtained from one or more of a plurality of sensors configured in the drone device;

a third computer program code for dynamically calibrating the recorded spatial location of the target object in communication with the one or more three-dimensional cameras and the one or more of the sensors based on environmental parameters of the work environment and a discernment of the target object;

a fourth computer program code for determining parts geometry of the target object based on the dynamically calibrated spatial location of the target object, geometrical information of the target object, and predetermined dimensions of a plurality of objects stored in a parts geometry database;

a fifth computer program code for determining a task trajectory of a task to be performed on the target object by the programmable robot based on the determined parts geometry of the target object, the predetermined dimensions of the objects stored in the parts geometry database, and predetermined task trajectories of the target objects stored in a procedure database;

a sixth computer program code for determining a collision-free robotic motion trajectory for the programmable robot based on the determined task trajectory, a model of the target object extracted from the determined parts geometry, and the environmental parameters of the work environment; and

a seventh computer program code for dynamically transmitting the dynamically calibrated spatial location of the target object, the determined parts geometry, the determined task trajectory, and the determined collision-free robotic motion trajectory to the programmable robot via a communication network to adaptively control the programmable robot to perform the task on the target object while accounting for misalignments of the target object in the work environment.

24. The non-transitory computer readable storage medium of claim 23, wherein the computer program codes further comprise:

an eighth computer program code for identifying contours of the target object from the determined parts geometry of the target object; and

a ninth computer program code for determining configurable positions for a component of the programmable robot with respect to the identified contours of the target object for adaptively controlling the programmable robot to move the component of the programmable robot to the determined configurable positions with respect to the identified contours of the target object to perform the task on the identified contours of the target object.

25. The non-transitory computer readable storage medium of claim 23, wherein the computer program codes further comprise a tenth computer program code for determining volumetric data associated with the task to be performed on the target object based on the determined parts geometry of the target object.