2D/3D Localization and Pose Estimation of Harness Cables Using A Configurable Structure Representation for Robot Operations

• US 20140267614A1
• Filed: 03/15/2013
• Published: 09/18/2014
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Seiko Epson Corporation (Seiko Group)
• Original Assignee: Seiko Epson Corporation (Seiko Group)
• Inventors: Ding, Yuanyuan, Xiao, Jing, Fukasawa, Kenji
• Granted Patent: US 9,154,773 B2
• Time in Patent Office: Days
• US Class Current: 348/46
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30108 Industrial image inspection
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/187 involving region growing; involving region merging; involving connected component labelling
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/757 Matching configurations of points or features
• CPC Class Codes: G06V 20/653 by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 2201/06 Recognition of objects for industrial automation
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals

Abstract

A robot is made to recognize and manipulate different types of cable harnesses in an assembly line. This is achieved by using a stereo camera system to define a 3D cloud of a given cable harness. Pose information of specific parts of the cable harness are determined from the 3D point cloud, and the cable harness is then re-presented as a collection of primitive geometric shapes of known dimensions, whose positions and orientations follow the spatial position of the represented cable harness. The robot can then manipulate the cable harness by using the simplified representation as a reference.

Claims

1. A cable harness visualization system, comprising:

a three-dimensional, i.e., 3D, imaging system imaging at least one physical cable harness and creating a 3D cable harness model of the imaged physical cable harness;

a cable structure representation module representing said 3D cable harness model in terms of sets of predefined primitive 3D shapes and node markers, each primitive 3D geometric shape being of known dimensions, the imaged physical cable harness having connectors and non-rigid cable segments, said node markers defining the opposing ends of said non-rigid cable segments, each non-rigid cable segment being joined to a connector or to another non-rigid cable segment at a node marker;

wherein;

each non-rigid cable segment is represented by a corresponding one of said sets of predefined primitive 3D shapes;

the corresponding set of predefined primitive 3D shapes consisting of a train of primitive 3D geometric shapes that spatially track the path of its corresponding non-rigid cable segment, each non-rigid cable segment extends into the interior of its corresponding train of geometric shapes, and the individual geometric shapes within a train of geometric shapes turn in accord with twists and turns of the non-rigid cable segment they represent.

2. The cable harness visualization system of claim 1, wherein in said cable structure representation module,

IF a non-rigid cable segment consists of a plurality of wires substantially coupled linearly side-by-side each other to a connector, THEN its corresponding train of primitive 3D geometric shapes consists of a train of rectangular prisms;

ELSE its corresponding train of primitive 3D geometric shapes consists of a train of cylinders.

3. The cable harness visualization system of claim 2, wherein the widths of said rectangular prisms substantially span across the side-by-side wires they represent, and the diameters of said cylinders substantially encompass the portion of the non-rigid cable segment they represent.

4. The cable harness visualization system of claim 2, wherein all rectangular prisms within the train of rectangular prisms have a common first length, and all cylinders within train of cylinders have a common second length.

5. The cable harness visualization system of claim 2, wherein each non-rigid cable segment is characterized by a lack of joints or rigid portions between its opposing ends.

6. The cable harness visualization system of claim 1, wherein said 3D imaging system includes at least one of a time-of-flight device, a 3D laser scanner, a KINECT sensor, and a range camera.

7. The cable harness visualization system of claim 1, wherein said 3D cable harness model is based on a point cloud.

8. The cable harness visualization system of claim 1, wherein said 3D imaging system includes:

a stereo imaging device producing stereoscopic image pairs of said physical cable harnesses, each stereoscopic image pair including a first image and a second image;

a two-dimensional, i.e., 2D, cable localization module localizing commonly imaged physical cable harnesses in said first and second images, said 2D cable localization module further identifying corresponding pixel pairs in said first and second images, each corresponding pixel pair including a first pixel from said first image and a second pixel from said second image, both first and second pixels corresponding to a commonly imaged point on a commonly imaged physical cable harness;

a 3D cable localization module creating a 3D point cloud representation of each commonly imaged physical cable harness in 3D space in accordance with perspective constraints and the commonly imaged physical cable harness'"'"' corresponding pixel pairs;

a 3D pose estimator determining cable pose orientations of commonly imaged physical cable harnesses in said 3D space in accordance with their corresponding 3D point clouds, said 3D pose estimator determining a surface normal direction for selected points within said 3D point clouds relative to a neighborhood-of-points of predefined size surrounding each selected point, said 3D cable harness model being defined by a corresponding 3D point cloud and corresponding cable pose orientations.

9. The cable harness visualization system of claim 8, wherein said 3D pose estimator determines the surface normal direction for all points within all 3D point clouds.

10. The cable harness visualization system of claim 8, wherein said 2D cable localization module segments said first and second image to define image segments, and each image segment defines a silhouette of an imaged cable harness.

11. The cable harness visualization system of claim 10, wherein each image segment is defined by:

(I) for each unsegmented part of an image, selecting a seed point within the unselected part, and for each selected point iteratively applying the following steps;

(i) determine similarity measures for the color similarities between the seed point and its nearest neighbor pixels, said nearest neighbor pixels being candidate pixels;

(ii) join the seed point and the candidate pixels if the similarity measures are higher than a predefined threshold, and IF any of the joined pixels are a part of an existing image segment, THEN the existing image segment is grown to include the joined pixels, ELSE the joined pixels define a new image segment;

(iii) determine a region color distribution for the image segment of step (ii) and calculate its principle component;

(II) returning to step (I) until all points within an image have been selected, in turn.

12. The cable harness visualization system of claim 11, further including after step (II):

(III) joining together any proximate image segments having a region color distribution similarity within a predefined first threshold and a geometric properties similarity within a predefined second threshold.

13. The cable harness visualization system of claim 12, wherein proximate image segments are defined as image segments separated by not more than 60 pixels.

14. The cable harness visualization system of claim 12, wherein the region color distribution similarity and the geometric properties similarity between image segments are determined from their respective principle components, within predefined thresholds.

15. The cable harness visualization system of claim 11, further including after step (II):

(a) determining a first set of feature descriptors for the image segments in said first image and a second set of feature descriptors for image segments in said second image; and

(b) defining said corresponding pixel pairs by matching pairs of corresponding feature descriptors between the first and second sets of feature descriptors.

16. The cable harness visualization system of claim 15, wherein step (b) includes:

identifying as a candidate matching descriptor, a feature descriptor in said second set that matches a given feature descriptor in said first set; and

IF its relative position within said second image differs from the relative position of the given feature descriptor in said first image by more than a predefined margin,

THEN discarding said candidate matching descriptor,

ELSE deeming said candidate matching descriptor as a corresponding pixel pair to the given feature descriptor.

17. The cable harness visualization system of claim 15, wherein step (b) feature descriptors in said first and second sets of feature descriptors are matched by means of a tree-based feature matching scheme.

18. The cable harness visualization system of claim 8, wherein said 3D pose estimator implements the following steps:

(a) applying global homography to said first and second images to reject corresponding pixel pairs in said first and second images that do not satisfy global homography constraints; and

(b) for each corresponding pixel pair not rejected in step (a), applying local homography in accordance to its corresponding image segments to further remove from each image segment any points that meeting local homography constraints.

19. The cable harness visualization system of claim 8, wherein said 3D pose estimator determines a surface normal direction for a selected point by;

defining a local window around the selected point;

identifying the 3D points within the defined local window;

fitting a 2D plane on to the identified 3D points, and estimating the local 3D normal direction of the fitted 2D plane.

20. A robotic system for manipulating cable harnesses, said robotic system implementing the cable harness visualization system of claim 1.