Vision system for training an assembly system through virtual assembly of objects

• US 10,223,589 B2
• Filed: 03/01/2016
• Issued: 03/05/2019
• Est. Priority Date: 03/03/2015
• Status: Active Grant

• Current Assignee: Cognex Corporation
• Original Assignee: Cognex Corporation
• Inventors: Shivaram, Guruprasad, Foster, Willard
• Primary Examiner(s): Philippe, Gims S
• Application Number:
  US15/058,118
• Publication Number:
  US 20170024613A1
• Time in Patent Office: 1,099 Days
• Field of Search: 21912434, 355 72, 348187, 382141, 382152
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 2219/31044 Assembly of modular products, variant configurability
• CPC Class Codes: G05B 2219/31066 Virtual assembly disassembly planning
• CPC Class Codes: G05B 2219/39045 Camera on end effector detects reference pattern
• CPC Class Codes: G05B 2219/40499 Reinforcement learning algorithm
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20092 Interactive image processing based on input by user
• CPC Class Codes: G06T 2207/30164 Workpiece; Machine component
• CPC Class Codes: G06T 7/0006 using a design-rule based approach
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: Y02P 90/02 Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]

Abstract

This invention provides a system and method for guiding the workpieces to optimal positions to train an assembly system that is generally free of the use of a CMM or similar metrology device. The system and method expresses the image features of the workpieces, when they are in their respective stations, in a common coordinate system. This ability allows a user to visualize the result of assembling the workpieces without actually assembling them, in a “virtual assembly”. The virtual assembly assists guiding placement of workpieces in respective stations into a desired relative alignment. The system and method illustratively generates a composite image using the images from cameras used in guiding the workpieces that helps the user visualize how the part would appear following assembly. The user can reposition the images of workpieces in their respective stations until the composite image has a desired appearance.

Claims

1. A system for virtual assembly of an object in an assembly system having at least a first station containing a first workpiece, the assembly system assembling a second workpiece to the first workpiece in alignment therebetween, the system comprising:

one or more vision system cameras and a vision system processor assembly arranged to image the first workpiece and the second workpiece, the plurality of cameras being calibrated to a common coordinate system;

an alignment process that generates at least one of images and features of the first workpiece and the second workpiece in the common coordinate system;

a manipulation process, having vision tools that locate a first set of features on the first workpiece and a second set of features on the second workpiece, that automatically aligns the first set of features with respect to the second set of features so as to align at least one of images and features of the first workpiece and the second workpiece; and

an assembly error computation module that computes an assembly error based on a degree of manipulation of the first workpiece and the second workpiece by the manipulation process during the virtual assembly process.

2. The system as set forth in claim 1 wherein the second workpiece is imaged at a second station remote from the first station.

3. The system as set forth in claim 2 further comprising at least a third workpiece, the vision system cameras imaging the third workpiece, the alignment process generating at least one of images and features of the third workpiece in the common coordinate system, the manipulation process being arranged to allow manipulation of the third workpiece with respect to at least one of the first workpiece and the second workpiece so as to align at least one of images and features of the third workpiece with at least one of images and features of at least one of the first workpiece and the second workpiece, and wherein the assembly error computation module computes an assembly error based on a degree of manipulation of the third workpiece with respect to at least one of the first workpiece and the second workpiece.

4. The system as set forth in claim 3 wherein the first workpiece is imaged by the vision system cameras at a first station and each of the second workpiece and the third workpiece are imaged by the vision system cameras at a second station.

5. The system as set forth in claim 4 further comprising at least a fourth workpiece imaged by the vision system cameras at a fourth station.

6. The system as set forth in claim 2 further comprising;

(a) a first calibration process that calibrates all the vision system cameras imaging the first station to a first coordinate system, in which all or a subset of the calibrated vision system cameras are adapted for use during runtime operation with the assembly system,

(b) a second calibration process that calibrates all the vision system cameras imaging the second station to a second coordinate system, in which all or subset of the calibrated cameras are adapted for use during runtime operation with the assembly system,

(c) a third calibration process that ties the first coordinate system and the second coordinate system into the common coordinate system, that employs a different discrete subset of the calibrated vision system cameras to tie the first coordinate system and the second coordinate system into the common coordinate system, and

(d) wherein all, or a subset of, the calibrated vision system cameras from step (a) and step (b) at each of the first station and the second station are employed in virtual assembly based upon the common coordinate system established based upon the first calibration process in step (a) and the second calibration process in step (b) and the third calibration process in step (c).

7. The system as set forth in claim 2 wherein one of the first station and the second station is an assembly platform and another of the first station and the second station is a pick location.

8. The system as set forth in claim 7 wherein the assembly platform defines a motion stage and the assembly system has a repeatable manipulator that moves the second workpiece from the pick station to the motion stage.

9. The system as set forth in claim 1 wherein the assembly error is defined in at least three degrees of freedom.

10. The system as set forth in claim 1, further comprising a display and user interface for manual alignment of at least one image of the first workpiece and at least one image of the second workpiece.

11. The system as set forth in claim 1 further comprising at least one vision system camera images at least one of the first workpiece and the second workpiece, the at least one vision system camera being constructed and arranged to acquire images used in at least one of training and runtime operation of the assembly system and being free of use with respect to the alignment process.

12. The system as set forth in claim 1 manipulation process employs at least one of (a) images of the first workpiece that are stitched and (b) images of the second workpiece that are stitched.

13. The system as set forth in claim 12 wherein the images of the first workpiece and the second workpiece are composed into a composite image with respect to the interface.

14. The system as set forth in claim 1 wherein at least one of the first workpiece and the second workpiece defines a planar surface imaged by the vision system cameras.

15. A method for virtual assembly of an object using an assembly system having at least a first station containing a first workpiece, the assembly system assembling a second workpiece to the first workpiece in alignment therebetween, the method comprising the steps of:

acquiring an image, with one or more vision system cameras and a vision system processor assembly, of the first workpiece and the second workpiece, the plurality of cameras being calibrated to a common coordinate system;

generating at least one of images and features of the first workpiece and the second workpiece in the common coordinate system;

automatically manipulating the second workpiece with respect to the first workpiece so as to align at least one of images and features of the first workpiece and the second workpiece; and

computing an assembly error based on a degree of manipulation of the first workpiece and the second workpiece during the virtual assembly process.

16. The method as set forth in claim 15 wherein the step of acquiring an image comprises acquiring an image of the second workpiece at a second station remote from the first station.

17. The method as set forth in claim 16 wherein one of the first station and the second station is an assembly platform and another of the first station and the second station is a pick location.

18. The method as set forth in claim 15 wherein the assembly error is defined in at least three degrees of freedom.

19. The method as set forth in claim 15, further comprising employing a display and user interface for manual alignment of at least one image of the first workpiece and at least one image of the second workpiece.

20. A system for virtual assembly of an object in an assembly system having at least a first station containing a first workpiece, the assembly system assembling a second workpiece to the first workpiece in alignment therebetween, the system comprising:

one or more vision system cameras and a vision system processor assembly arranged to image the first workpiece and the second workpiece, the plurality of cameras being calibrated to a common coordinate system, wherein the second workpiece is imaged at a second station remote from the first station;

an alignment process that generates at least one of images and features of the first workpiece and the second workpiece in the common coordinate system;

a manipulation process, having vision tools that locate a first set of features on the first workpiece and a second set of features on the second workpiece, that automatically aligns the first set of features with respect to the second set of features so as to align at least one of images and features of the first workpiece and the second workpiece; and

an assembly error computation module that computes an assembly error based on a degree of manipulation of the first workpiece and the second workpiece by the manipulation process during the virtual assembly process; and

at least a third workpiece, the vision system cameras imaging the third workpiece, the alignment process generating at least one of images and features of the third workpiece in the common coordinate system, the manipulation process being arranged to allow manipulation of the third workpiece with respect to at least one of the first workpiece and the second workpiece so as to align at least one of images and features of the third workpiece with at least one of images and features of at least one of the first workpiece and the second workpiece, and wherein the assembly error computation module computes an assembly error based on a degree of manipulation of the third workpiece with respect to at least one of the first workpiece and the second workpiece.