Robotic system and method for operating on a workpiece

• US 10,625,419 B2
• Filed: 03/21/2018
• Issued: 04/21/2020
• Est. Priority Date: 03/21/2018
• Status: Active Grant

• Current Assignee: The Boeing Co.
• Original Assignee: The Boeing Co.
• Inventors: Szarski, Martin A., Wierzbicki, Dominic, Haeusler, Phillip James
• Primary Examiner(s): Marc, McDieunel
• Application Number:
  US15/927,994
• Publication Number:
  US 20190291275A1
• Time in Patent Office: 762 Days
• CPC Class Codes: B25J 11/0085 Cleaning
• CPC Class Codes: B25J 9/162 Mobile manipulator, movable base with manipulator arm mounted on it
• CPC Class Codes: B25J 9/1666 Avoiding collision or forbidden zones
• CPC Class Codes: B25J 9/1669 characterised by special application, e.g. multi-arm co-operation, assembly, grasping
• CPC Class Codes: B25J 9/1671 characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 19/4097 characterised by using design data to control NC machines, e.g. CAD/CAM G05B19/4093 takes precedence
• CPC Class Codes: G05D 1/0214 in accordance with safety or protection criteria, e.g. avoiding hazardous areas monitoring the location of vehicles within a certain area, e.g. forbidden or allowed areas, in traffic control systems for road vehicles G08G1/13
• CPC Class Codes: Y10S 901/01 Mobile robot
• CPC Class Codes: Y10S 901/09 Closed loop, sensor feedback controls arm movement
• CPC Class Codes: Y10S 901/47 Optical
• CPC Class Codes: Y10S 901/49 Protective device

Abstract

A robotic system includes a robot and a processor configured to autonomously generate a master plan for movement of an end effector of the robot for operating on an operating surface of a workpiece in a physical environment. The master plan is based on a computer aided design (CAD) model of the workpiece and defines movement paths of the end effector based on a reach distance of a robotic arm of the robot at each of a plurality of working waypoints defined in the master plan and corresponding to fiducial markers on the workpiece. The processor registers the robot to the workpiece, iteratively adjusts an approximate workpiece position in a world map until matching an actual workpiece position in the physical environment, and constructs and executes a robotic arm trajectory causing the end effector to operate on the operating surface along the movement paths defined in the master plan.

Claims

1. A robotic system for autonomously operating on a workpiece, comprising:

at least one robot communicatively coupled to a data processing system including a processor, and having a robotic arm and an end effector coupled to the robotic arm;

the processor configured to autonomously generate a master plan for movement of the end effector for operating on an operating surface of a workpiece positioned in a physical environment, the master plan being based on a computer aided design (CAD) model of the workpiece and defining movement paths of the end effector based at least in part on a reach distance of the robotic arm of the robot at each of a plurality of working waypoints included in the master plan to correspond to a plurality of fiducial markers mounted on the workpiece and represented in the CAD model;

the processor configured to autonomously perform the following sequence of steps for each one of the fiducial markers;

register the robot to the workpiece in the physical environment when the robot is driving toward and/or is at an approach waypoint in front of a fiducial marker, and iteratively adjust a position of a tessellated workpiece collision geometry within a world map of the physical environment unless or until an approximate workpiece position of the tessellated workpiece collision geometry within the world map substantially matches an actual workpiece position in the physical environment when the robot is driving toward and/or is at the working waypoint corresponding to the fiducial marker;

construct, with the robot at the working waypoint, a robotic arm trajectory for performing a sequence of movements of the end effector along the movement paths associated with the working waypoint; and

execute the robotic arm trajectory and cause the end effector to operate on a portion of the operating surface of the workpiece corresponding to the portion of the master plan defined for the robot at the working waypoint.

2. The robotic system of claim 1, wherein the processor, in autonomously generating the master plan, is configured to sequentially perform the following:

construct, based on a geometry file of the CAD model, a plurality of working waypoints respectively at a distance from the fiducial markers based on the reach distance of the robotic arm;

construct at each one of the working waypoints a reach sphere sized in correspondence to the reach distance;

define a plurality of surface segments based on an intersection of each reach sphere with the operating surface

determine for each surface segment a cut direction defined as a direction of a largest dimension of the surface segment;

form the movement paths for each surface segment by intersecting each surface segment with a series of vertical planes oriented parallel to the cut direction;

designate, for each working waypoint, alternating directions of movement of the end effector along adjacent movement paths, alternating directions of movement defining a sequence of movements for the end effector; and

save the master plan containing the movement paths and the sequence of movements of the end effector at all of the working waypoints.

3. The robotic system of claim 2, wherein the processor is configured to:

define the operating surface of the CAD model of the workpiece, the CAD model having collision geometry;

define the plurality of fiducial markers of the CAD model;

tessellate the collision geometry of the CAD model to form tessellated workpiece collision geometry; and

save the geometry file containing at least the definition of the operating surface, the fiducial markers, and the tessellated workpiece collision geometry of the workpiece.

4. The robotic system of claim 1, wherein:

the robot includes a sensor system; and

the processor is configured to construct the world map based on imaging data of environment boundaries of the physical environment and objects within the environment boundaries recorded by the sensor system while the robot is piloted along a floor surface of the physical environment.

5. The robotic system of claim 4, wherein the processor is configured to:

position within the world map a tessellated workpiece collision geometry of the workpiece;

create a convex hull of the workpiece based on the tessellated workpiece collision geometry; and

set the convex hull as a keep-out region for the robot.

6. The robotic system of claim 1, wherein the processor is configured to:

load a kinematic model of the robot into the world map; and

determine, based on the kinematic model, whether the end effector is capable of reaching all locations along the movement paths associated with each working waypoint prior to registering the robot to the workpiece.

7. The robotic system of claim 1, wherein, in registering the robot to the workpiece and adjusting the position of the tessellated workpiece collision geometry within the world map, the processor is configured to sequentially perform the following:

drive the robot to the approach waypoint;

confirm, when the robot is driving toward and/or is at the approach waypoint, that an identification pattern of the fiducial marker on the workpiece matches the identification pattern of the fiducial marker represented in the CAD model;

adjust, when the robot is driving toward and/or is at the approach waypoint, an approximate workpiece position of the tessellated workpiece collision geometry within the world map unless or until the approximate workpiece position is within a predetermined initial tolerance of an actual workpiece position of the workpiece in the physical environment;

move the robot from the approach waypoint to the working waypoint; and

iteratively adjust, with the robot at the working waypoint, the position of the tessellated workpiece collision geometry within the world map unless or until the position of the tessellated workpiece collision geometry within the world map substantially matches the actual workpiece position in the physical environment.

8. The robotic system of claim 1, wherein, in constructing the robotic arm trajectory, the processor is configured to sequentially perform the following:

sample a plurality of poses of the end effector at spaced intervals along each movement path;

sample a plurality of z-axis rotations for each pose of the end effector;

solve all inverse kinematic solutions for each sampled pose;

construct a graph connecting all poses and solving for minimum joint movement between each pose; and

construct the robotic arm trajectory for performing the sequence of movements of the end effector along the movement paths according to the minimum joint movement.

9. A robotic system for autonomously operating on a workpiece, comprising:

a plurality of robots each having a robotic arm and an end effector coupled to the robotic arm;

at least one processor configured to autonomously generate a master plan for movement of each end effector for operating on an operating surface of a workpiece positioned in a physical environment, the master plan being based on a computer aided design (CAD) model of the workpiece and comprising a plurality of child plans for distribution to the plurality of robots and each defining movement paths of an end effector of one of the robots, the movement paths being based at least in part on a reach distance of a robotic arm of the robot assigned by the child plan to one of more of a unique set of working waypoints included in the master plan, the working waypoints corresponding to a plurality of fiducial markers mounted on the workpiece and represented in the CAD model;

the processor configured to autonomously perform the following;

for each one of the robots, register to the workpiece in the physical environment when driving toward and/or at an approach waypoint in front of a working waypoint of the set of working waypoints assigned to the robot;

for only one of the robots when driving toward and/or at a first one of the working waypoints in the set of working waypoints, iteratively adjust a position of a tessellated workpiece collision geometry within a world map of the physical environment unless or until an approximate workpiece position of the tessellated workpiece collision geometry within the world map substantially matches an actual workpiece position in the physical environment when the robot is driving toward and/or is at the working waypoint;

for each one of the robots when at each working waypoint in the set of working waypoints assigned to the robot;

construct a robotic arm trajectory for performing a sequence of movements of the end effector along the movement paths associated with the working waypoint; and

execute the robotic arm trajectory in a manner causing the end effector to operate on a portion of the operating surface corresponding to the portion of the child plan defined for the robot at the working waypoint.

10. The robotic system of claim 9, wherein the processor is configured to assign a set of working waypoints to each one of the plurality of robots based on at least one of the following:

waypoint location of the working waypoints relative to one another;

end effector capability of the end effectors relative to one another; and

robot workload of the robots relative to one another.

11. A method of operating on a workpiece:

autonomously generating, using a processor, a master plan for movement of at least one end effector correspondingly of at least one robot for operating on an operating surface of a workpiece in a physical environment, the master plan being based on a computer aided design (CAD) model of the workpiece and defining movement paths of the end effector based at least in part on a reach distance of a robotic arm of the robot at each of a plurality of working waypoints included in the master plan, the plurality of working waypoints corresponding to a plurality of fiducial markers mounted on the workpiece and represented in the CAD model;

autonomously performing, using the processor, the following sequence of steps for the robot at each one of the plurality of working waypoints;

registering the robot to the workpiece in the physical environment when the robot is driving toward and/or is at an approach waypoint in front of a working waypoint, and iteratively adjusting a position of a tessellated workpiece collision geometry within a world map of the physical environment unless or until an approximate workpiece position of the tessellated workpiece collision geometry within the world map substantially matches an actual workpiece position in the physical environment when the robot is driving toward and/or is at the working waypoint;

constructing, with the robot at the working waypoint, a robotic arm trajectory for performing a sequence of movements of the end effector along the movement paths associated with the working waypoint; and

executing the robotic arm trajectory in a manner causing the end effector to operate on a portion of the operating surface corresponding to the portion of the master plan defined for the robot at the working waypoint.

12. The method of claim 11, wherein the step of autonomously generating the master plan comprises:

constructing, based on a geometry file of the CAD model, the plurality of working waypoints respectively at a distance from the fiducial markers based on the reach distance of the robotic arm;

constructing at each one of the working waypoints a reach sphere sized in correspondence to the reach distance;

defining a plurality of surface segments based on an intersection of each reach sphere with the operating surface;

determining for each surface segment a cut direction defined as a direction of a largest dimension of the surface segment;

forming the movement paths for each surface segment by intersecting each surface segment with a series of vertical planes oriented approximately parallel to the cut direction;

designating, for each working waypoint, alternating directions of movement of the end effector along adjacent movement paths, alternating directions of movement defining a sequence of movements for the end effector at each working waypoint; and

saving the master plan containing the movement paths and the sequence of movements of the end effector at all of the working waypoints.

13. The method of claim 12, further including:

defining the operating surface of the CAD model of the workpiece, the CAD model having collision geometry;

defining the plurality of fiducial markers of the CAD model;

tessellating the collision geometry of the CAD model to form tessellated workpiece collision geometry; and

generating the geometry file by saving a definition of the operating surface, the fiducial markers, and the tessellated workpiece collision geometry of the workpiece.

14. The method of claim 11, further comprising constructing the world map by performing the steps of:

piloting the robot along a floor surface of the physical environment while recording, using a sensor system of the robot, imaging data of environment boundaries of the physical environment and objects within the environment boundaries; and

constructing, using the processor, the world map based on the imaging data.

15. The method of claim 14, wherein after constructing the world map, the method further includes:

positioning, within the world map, tessellated workpiece collision geometry of the workpiece;

creating a convex hull of the workpiece based on the tessellated workpiece collision geometry; and

setting the convex hull as a keep-out region for the robot.

16. The method of claim 11, wherein after generating the master plan and prior to registering the robot to the workpiece, the method includes:

loading a kinematic model of the robot into the world map; and

determining, based on the kinematic model, whether the end effector is capable of reaching all locations along the movement paths associated with each working waypoint.

17. The method of claim 11, wherein the step of registering the robot to the workpiece and adjusting the position of the tessellated workpiece collision geometry within the world map comprises sequentially performing the following:

driving the robot to the approach waypoint;

confirming, when the robot is driving toward and/or is at the approach waypoint, that an identification pattern of the fiducial marker on the workpiece matches the identification pattern of the fiducial marker represented in the CAD model;

adjusting, when the robot is driving toward and/or is at the approach waypoint, an approximate workpiece position of the tessellated workpiece collision geometry within the world map until the approximate workpiece position is within a predetermined initial tolerance of an actual workpiece position in the physical environment;

moving the robot from the approach waypoint to the working waypoint; and

iteratively adjusting, when the robot is driving toward and/or is at the working waypoint, the approximate workpiece position of the tessellated workpiece collision geometry within the world map until the approximate workpiece position within the world map substantially matches the actual workpiece approximate workpiece position in the physical environment.

18. The method of claim 11, wherein the step of constructing the robotic arm trajectory comprises:

sampling a plurality of poses of the end effector at spaced intervals along each movement path;

sampling a plurality of z-axis rotations for each pose of the end effector;

solving all inverse kinematic solutions for each sampled pose;

constructing a graph connecting all poses and solving for minimum joint movement between each pose; and

constructing the robotic arm trajectory for performing the sequence of movements of the end effector along the movement paths according to the minimum joint movement.

19. A method of autonomously operating on a workpiece, comprising:

autonomously generating, using a processor, a master plan for movement of a plurality of end effectors of a corresponding plurality of robots configured to operate on different surface segments of an operating surface of a workpiece positioned in a physical environment, the master plan being based on a computer aided design (CAD) model of the workpiece and comprising a plurality of child plans for distribution correspondingly to the plurality of robots and each defining movement paths of an end effector of one of the robots, the movement paths being based at least in part on a reach distance of a robotic arm of the robot assigned by the child plan to operate from a unique set of working waypoints included in the master plan, the working waypoints corresponding to a plurality of fiducial markers mounted on the workpiece and represented in the CAD model;

autonomously performing, using the processor, the following;

registering each one of the robots to the workpiece in the physical environment when the robot is driving toward and/or is at an approach waypoint in front of a working waypoint of the set of working waypoints assigned to the robot,

iteratively adjusting, for only one of the robots when driving toward and/or at a first one of the working waypoints in the set of working waypoints, an approximate workpiece position of a tessellated workpiece collision geometry within a world map of the physical environment unless or until the approximate workpiece position within the world map substantially matches an actual workpiece position in the physical environment when the robot is driving toward and/or is at the working waypoint;

constructing, for each one of the robots when at each working waypoint in the set of working waypoints assigned to the robot, a robotic arm trajectory for performing a sequence of movements of the end effector along the movement paths associated with the working waypoint; and

executing, for each one of the robots when at each working waypoint in the set of working waypoints assigned to the robot, the robotic arm trajectory in a manner causing the end effector to operate on a portion of the operating surface corresponding to the portion of the child plan defined for the robot at the working waypoint.

20. The method of claim 19, further including the step of:

assigning a unique set of working waypoints to each one of the plurality of robots based on at least one of the following;

a waypoint location of the working waypoints relative to one another;

an end effector capability of the end effectors relative to one another; and

a robot workload of the robots relative to one another.