System and method for servoing robots based upon workpieces with fiducial marks using machine vision

• US 6,681,151 B1
• Filed: 12/15/2000
• Issued: 01/20/2004
• Est. Priority Date: 12/15/2000
• Status: Active Grant

• Current Assignee: Cognex Technology & Investment Corporation (Cognex Corporation)
• Original Assignee: Cognex Technology & Investment Corporation (Cognex Corporation)
• Inventors: Weinzimmer, Russ, Wallack, Aaron
• Primary Examiner(s): Cuchlinski, Jr., William A.
• Assistant Examiner(s): Marc, McDieunel, Marc, McDieunel
• Application Number:
  US09/738,542
• Time in Patent Office: 1,131 Days
• Field of Search: 700/254, 700/245, 700/259, 700/159, 700/83, 700/85, 382/146, 382/151, 382/154, 382/145, 382/103, 382/199, 382/291, 382/294, 382/295, 382/287, 382/152, 717/104, 717/109, 219/124.34, 219/121.85, 356/243.1, 356/237.6, 356/237.1, 348/94, 348/95
• US Class Current: 700/259
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 19/128 the workpiece itself serves as a record carrier, e.g. by its form, by marks or codes on it
• CPC Class Codes: G05B 2219/39397 Map image error directly to robot movement, position with relation to world, base not needed, image based visual servoing
• CPC Class Codes: G05B 2219/40565 Detect features of object, not position or orientation
• CPC Class Codes: G05B 2219/40607 Fixed camera to observe workspace, object, workpiece, global

Abstract

A system and method for servoing robot marks using fiducial marks and machine vision provides a machine vision system having a machine vision search tool that is adapted to register a pattern, namely a trained fiducial mark, that is transformed by at least two translational degrees and at least one mon-translational degree of freedom. The fiducial is provided to workpiece carried by an end effector of a robot operating within a work area. When the workpiece enters an area of interest within a field of view of a camera of the machine vision system, the fiducial is recognized by the tool based upon a previously trained and calibrated stored image within the tool. The location of the work-piece is derived by the machine vision system based upon the viewed location of the fiducial. The location of the found fiducial is compared with that of a desired location for the fiducial. The desired location can be based upon a standard or desired position of the workpiece. If a difference between location of the found fiducial and the desired location exists, the difference is calculated with respect to each of the translational axes and the rotation. The difference can then be further transformed into robot-based coordinates to the robot controller, and workpiece movement is adjusted based upon the difference. Fiducial location and adjustment continues until the workpiece is located the desired position with minimum error.

Claims

1. A system for guiding a robot, carrying a workpiece, operating in a work area comprising:

a machine vision system having a camera located so as to acquire an image of the work area, the machine vision system including a machine vision search tool for locating an instance of a trained pattern in the image, the tool registering the trained pattern transformed by at least two translational degrees of freedom and at least one non-translational degree of freedom with respect to an image plane acquired by the machine vision search tool;

a fiducial, on the workpiece, the fiducial matching the trained pattern;

wherein the search tool is for determining, based upon the image, a relative location of the workpiece with respect to a predetermined reference point in the machine vision search tool at which the workpiece is employed; and

means for mapping between the predetermined reference point in the machine vision search tool and a predetermined reference point in a reference system of the robot.

2. The system as set forth in claim 1 wherein the predetermined reference point in the machine vision search tool comprises a desired location of the fiducial that corresponds to a target location for the work piece in the reference system of the robot.

3. The system as set forth in claim 1 wherein the predetermined reference point in the machine vision search tool comprises a desired location of the fiducial, and further comprising software for computing a difference between the viewed location of the fiducial and the desired location of the fiducial.

4. The system as set forth in claim 1 further comprising a robot controller that directs the workpiece so that the fiducial is positioned at the predetermined reference point based upon the means for mapping.

5. The system as set forth in claim 1 further comprising software for computing a difference between the viewed location and the predetermined reference point and wherein the means for mapping transforms the difference into a displacement vector with respect to a movement reference system of the robot controller.

6. The system as set forth in claim 5 wherein the robot controller directs the robot to move the workpiece in four degrees of freedom, including two degrees defining movement axes substantially coincident with the image plane, one degree defining a movement axis substantially perpendicular to the image plane and one degree defining a rotation of the workpiece substantially within the image plane about the axis substantially perpendicular to the image plane.

7. The system as set forth in claim 5 wherein the robot controller is interconnected to the machine vision search tool so as to receive the displacement vector, the robot controller being for moving the robot to, thereby, move the workpiece based upon the displacement vectors.

8. The system as set forth in claim 7 wherein the means for mapping is for providing an under-correction to the displacement vector so that the robot, based upon the robot controller, moves the workpiece less than an amount needed to locate the fiducial at the desired location.

9. The system as set forth in claim 1 wherein the predetermined reference point in the machine vision search tool comprises a desired location for the fiducial, and further comprising software for computing a difference between the viewed location of the fiducial and the desired location of the fiducial in the machine vision search tool, and wherein the difference is resolved into the non-translational degrees of freedom including two orthonagl axes within the image plane.

10. The system as set forth in claim 9 wherein the difference is resolved into the non-translational degree of freedom that comprises one of either a rotation within the image plane or a scale value along an axis substantially perpendicular to the image plane.

11. The system as set forth in claim 1 wherein the search tool comprises a rotation/scale-invariant search (RSIS) tool for determining the relative location of the fiducial in each at least four degrees of freedom including each of the two translational degrees and one rotational degree about an axis transverse to the image plane defined by the two translational degrees and one degree along the axis transverse to the image plane.

12. The system as set forth in claim 1 wherein the fiducial comprises a structure formed on the workpiece.

13. The system as set forth in claim 1 wherein the workpiece comprises one of either a part for positioning at a target location or a tool operating at the target location.

14. A method for guiding a robot, carrying a workpiece, operating in a work area comprising:

acquiring an image in the work area with a machine vision search tool for locating an instance of a trained pattern in the image, the tool registering the trained pattern transformed by at least two translational degrees of freedom and at least one non-translational degree of freedom with respect to an image plane acquired by the machine vision search tool;

positioning a workpiece having fiducial matching the trained pattern in an end effector of the robot and with respect to a field of view of the machine vision search tool;

determining, based upon the image, a relative location of the workpiece with respect to a predetermined reference point in the machine vision search tool at which the workpiece is employed; and

mapping between the predetermined reference point in the machine vision search tool and a predetermined reference point in a reference system of the robot.

15. The method as set forth in claim 14 wherein the predetermined reference point in the machine vision search tool comprises a desired location of the fiducial that is mapped by the step of mapping to a target location for the workpiece at which the workpiece is employed.

16. The method as set forth in claim 14 wherein the predetermined reference point in the machine vision search tool comprises a desired location of the fiducial and further comprising computing a difference between the viewed location of the fiducial and the desired location.

17. The method as set forth in claim 14 further comprising directing, with a robot controller, the robot to move the workpiece so that the fiducial is positioned at the predetermined reference point.

18. The method as set forth in claim 17 wherein the step of directing robot includes moving the workpiece in four degrees of freedom, including two degrees defining movement axes substantially coincident with the image plane, one degree defining a movement axis substantially perpendicular to the image plane and one degree defining a rotation of the workpiece substantially within the image plane about the axis substantially perpendicular to the image plane.

19. The method as set forth in claim 14 wherein the predetermined reference point in the machine vision search tool comprises a desired location of the fiducial in the machine vision search tool, and further comprising computing a difference between the viewed location of the fiducial and the desired location, directing, with a robot controller, the robot so as to move the workpiece so that the fiducial is positioned at the predetermined reference point in the machine vision search tool and transforming the difference into a displacement vector with respect to a movement reference system of the robot controller.

20. The method as set forth in claim 19 further comprising transmitting, to the robot controller from the machine vision search tool, a displacement vector, and commanding the robot to move the workpiece based upon the displacement vector.

21. The method as set forth in claim 20 wherein the step of transforming includes providing an under-correction to the displacement vectors so that the robot, based upon the robot controller, moves the workpiece less than an amount needed to locate the fiducial at the desired location.

22. The method as set forth in claim 19 further comprising performing an operation at a target area with a workpiece that comprises a tool after the step of directing.

23. The method as set forth in claim 14 wherein the predetermined reference point in the machine vision search tool comprises a desired location of the fiducial and further comprising computing a difference between the viewed location of the fiducial and the desired location of the fiducial, and wherein the step of computing includes resolving the difference into the non-translational degrees of freedom including two orthogonal axes within the image plane.

24. The method as set forth in claim 23 wherein the step of computing includes resolving the difference into the non-translational degree of freedom that comprises one of either a rotation within the image plane or a scale value along an axis substantially perpendicular to the image plane.

25. The method as set forth in claim 14 wherein the step of determining includes deriving the relative location of the fiducial with a rotation/scale-invariant search (RSIS) tool for determining the relative location of the fiducial in each of the four degrees of freedom including each of the two translational degrees and one rotational degree about an axis transverse to the image plane defined by the two translational degrees and one scale degree along the axis transverse to the image plane.

26. The method as set forth in claim 14 further comprising training the search tool to recognize the fiducial including exposing the search tool to an image containing the fiducial and selecting a window that contains the fiducial so as to cause the search tool to store a pattern of the selected window with the fiducial therein.

27. The method as set forth in claim 14 further comprising training the machine vision search tool to recognize the fiducial based upon loading a stored image file including a pattern substantially matching the fiducial therein.

28. The method as set forth in claim 14 further comprising calibrating the search tool to recognize a predetermined displacement of the fiducial including moving the workpiece for a predetermined displacement from a base location in each of a plurality of degrees of freedom and locating the fiducial at a respective displaced location after moving for each predetermined displacement so as to compute a difference between the base location and each respective displaced location.

29. The method as set forth in claim 28 wherein the step of calibrating further comprises mapping the difference with respect to predetermined known displacement values for the workpiece in a robot reference system and providing a displacement vector based upon the mapping to the robot so as to enable adjustment of the robot to a corrected location relative to the predetermined reference point in the machine vision search tool.

30. A computer-readable medium executed on a computer for guiding a robot carrying a workpiece operating in a work area, the computer-readable medium comprising program instructions for performing the steps of:

acquiring an image in the work area with a machine vision search tool that locates an instance of a trained pattern in the image, the tool registering the trained pattern transformed by at least two translational degrees of freedom and at least one non-translational degree of freedom with respect to an image plane acquired by the machine vision search tool;

positioning a workpiece having fiducial matching the trained pattern; and

determining, based upon the image, a relative location of the workpiece with respect to a predetermined reference point in the machine vision search tool at which the workpiece is employed; and

mapping between the predetermined reference point in the machine vision search tool and a predetermined reference point in a reference system of the robot.

31. The computer-readable medium as set forth in claim 30 wherein the predetermined reference point in the machine vision search tool comprises a desired location of the fiducial and further comprising computing a difference between the viewed location of the fiducial and the desired location of the fiducial, and wherein the step of computing includes resolving the difference into the non-translational degrees of freedom including two orthogonal axes within the image plane.

32. The computer-readable medium as set forth in claim 31 wherein the step of computing includes resolving the difference into the non-translational degree of freedom that comprises one of either a rotation within the image plane or a scale value along an axis substantially perpendicular to the image plane.

33. The computer-readable medium as set forth in claim 30 wherein the step of determining includes deriving the relative location of the fiducial with a rotation/scale-invariant search (RSIS) tool for determining the relative location of the fiducial in each of the four degrees of freedom including each of the two translational degrees and one rotational degree about an axis transverse to the image plane defined by the two translational degrees and one scale degree along the axis transverse to the image plane.

34. The computer-readable medium as set forth in claim 30 further comprising training the search tool to recognize the fiducial including exposing the search tool to an image containing the fiducial and selecting a window that contains the fiducial so as to cause the search tool to store a pattern of the selected window with the fiducial therein.

35. The computer-readable medium as set forth in claim 30 further comprising training the search tool to recognize the fiducial based upon loading a stored image file including a pattern substantially matching the fiducial therein.

36. The computer-readable medium as set forth in claim 30 further comprising calibrating the search tool to recognize a predetermined displacement of the fiducial including moving the workpiece for a predetermined displacement from a base location in each of a plurality of degrees of freedom and locating the fiducial at a respective displaced location after moving for each predetermined displacement so as to compute a difference between the base location and each respective displaced location.

37. The computer-readable medium as set forth in claim 36 wherein the step of calibrating further comprises mapping the difference with respect to predetermined known displacement values for the workpiece in a robot reference system and providing a displacement vector based upon the mapping to the robot so as to enable adjustment of the robot to a corrected location relative to the predetermined reference point in the machine vision search tool.