SYSTEM AND METHOD FOR TRAINING A MODEL IN A PLURALITY OF NON-PERSPECTIVE CAMERAS AND DETERMINING 3D POSE OF AN OBJECT AT RUNTIME WITH THE SAME

• US 20120147149A1
• Filed: 12/08/2010
• Published: 06/14/2012
• Est. Priority Date: 12/08/2010
• Status: Active Grant

• Current Assignee: Cognex Corporation
• Original Assignee: Cognex Corporation
• Inventors: Wallack, Aaron S., Marrion, Cyril C. Jr., Liu, Lifeng
• Granted Patent: US 11,488,322 B2
• Time in Patent Office: Days
• US Class Current: 348/50
• CPC Class Codes: G06T 7/75 involving models

Abstract

This invention provides a system and method for training and performing runtime 3D pose determination of an object using a plurality of camera assemblies in a 3D vision system. The cameras are arranged at different orientations with respect to a scene, so as to acquire contemporaneous images of an object, both at training and runtime. Each of the camera assemblies includes a non-perspective lens that acquires a respective non-perspective image for use in the process. The searched object features in one of the acquired non-perspective image can be used to define the expected location of object features in the second (or subsequent) non-perspective images based upon an affine transform, which is computed based upon at least a subset of the intrinsics and extrinsics of each camera. The locations of features in the second, and subsequent, non-perspective images can be refined by searching within the expected location of those images. This approach can be used in training, to generate the training model, and in runtime operating on acquired images of runtime objects. The non-perspective cameras can employ telecentric lenses.

Claims

1. A method for determining a 3D pose of an object during runtime operation of a 3D vision system comprising the steps of:

orienting at least a first non-perspective camera assembly and a second non-perspective camera assembly with respect to the object so that a first non-perspective image is acquired by the first non-perspective camera assembly and a second non-perspective image is contemporaneously acquired by the second non-perspective camera;

searching for 2D model features in the first non-perspective image based upon a first model;

searching for 2D model features in the second non-perspective image based upon a second model in which the second model is a trained descendant of the first model; and

determining 3D pose of the object based upon locations of the 2D model features in the first non-perspective image and the second non-perspective image.

2. The method as set forth in claim 1 further comprising training the second model at training time based upon an affine transform applied to the first model.

3. The method as set forth in claim 2 further comprising distorting at least one of the first non-perspective image and the second non-perspective image, according to a predetermined relationship, and removing lens distortion from at least one of the first non-perspective image and the second non-perspective image in order to perform the searching step respectively therein.

4. The method as set forth in claim 2 further comprising computing the affine transform based upon at least a subset of intrinsics and extrinsics of the first non-perspective camera assembly and intrinsics and extrinsics of the second camera non-perspective camera assembly.

5. The method as set forth in claim 1 further comprising performing an operation on the object based upon the 3D pose.

6. The method as set forth in claim 5 wherein the operation includes manipulating a robot with respect to the object.

7. A 3D vision system for determining a 3D pose of an object during runtime operation comprising:

a first non-perspective camera assembly and a second non-perspective camera assembly that respectively acquire a first non-perspective image and a second non-perspective image of the object contemporaneously; and

a searching tool that searches for 2D model features in the first non-perspective image based upon a first model and that searches for 2D model features in the second non-perspective image based upon a second model in which the second model is a trained descendant of the first model, the 3D pose being based upon locations of the searched 2D model features in each of the first non-perspective image and the second-non-perspective image.

8. The 3D vision system as set forth in claim 7 the second model is trained at training time based upon an affine transform applied to the first model.

9. The 3D vision system as set forth in claim 8 wherein at least one of the first non-perspective image and the second non-perspective image is distorted according to a predetermined relationship upon acquisition, and further comprising an undistorting process that undistorts the at least one of the first non-perspective image and the second non-perspective image for searching by the searching tool.

10. The system as set forth in claim 8 wherein the affine transform is based upon at least a subset of intrinsics and extrinsics of the first non-perspective camera assembly and intrinsics and extrinsics of the second camera non-perspective camera assembly.

12. The system as set forth in claim 7 wherein the first non-perspective camera assembly includes a non-perspective lens and the second non-perspective camera assembly includes a non-perspective lens.

13. The system as set forth in claim 12 wherein at least one non-perspective lens comprises a telecentric lens.

14. The system as set forth in claim 7 further comprising a device that moves with respect to the object based upon the pose data.

15. The system as set forth in claim 14 wherein the device comprises a robot.

16. A method for training a model of an object for use during runtime operation of a 3D vision system comprising the steps of:

orienting at least a first non-perspective camera assembly and a second non-perspective camera assembly with respect to the object so that a first non-perspective image is acquired by the first non-perspective camera assembly and a second non-perspective image is contemporaneously acquired by the second non-perspective camera;

providing an affine transform between the first non-perspective camera assembly and the second non-perspective camera assembly based on at least a subset of intriniscs and extrinsics of the first non-perspective camera assembly and intrinsics and extrinsics of the second non-perspective camera assembly;

defining a first model with a reference point in the first non-perspective image; and

generating a second model with the reference point based upon the affine transform.

17. The method as set forth in claim 16 further comprising refining the second model based upon a search for the reference point in the second non-perspective image within a search range limited based upon the second model generated by the affine transform.

18. The method as set forth in claim 17 further comprising, during runtime, determining a 3D pose of the object based upon at least the first model and the second model.

19. The method as set forth in claim 18 further comprising, based upon the 3D pose, performing an operation upon the object.

20. The method as set forth in claim 17 wherein the step of performing includes directing movement of a robot with respect to the object.

21. A 3D vision system comprising:

a first non-perspective camera assembly and a second non-perspective camera assembly that respectively acquire a first non-perspective image and a second non-perspective image of the object contemporaneously; and

a training process that includes,

(a) a search process that locates a first model pattern in the first non perspective image,

(b) an affine transform process that generates an affine-transformed pattern from the first pattern based upon at least a subset of intrinsics and extrinsics of each of the first non-perspective camera assembly and the second non-perspective camera assembly, and

(c) a registration process that registers the affine-transformed pattern with respect to the second non-perspective image, determines a locale in the second non-perspective image of a pattern associated with the affine-transformed pattern, and defines a second model pattern from the associated pattern.

22. The 3D vision system as set forth in claim 21 wherein, during runtime, the first non-perspective camera assembly and the second non-perspective camera assembly are constructed and arranged to respectively acquire a first non-perspective runtime image and a second non-perspective runtime image of a runtime object, contemporaneously, the runtime object having features similar to the object, and further comprising a runtime process that includes a registration process that registers the first model pattern with respect to the first runtime non-perspective image and the second model pattern with respect to the second runtime non-perspective image so as to determine a 3D pose of the object.

23. The method as set forth in claim 22 further comprising a device that, based upon the 3D pose, performs an operation upon the object.

24. The method as set forth in claim 23 wherein the device comprises a robot that moves with respect to the object.