Geodesic Distance Based Primitive Segmentation and Fitting for 3D Modeling of Non-Rigid Objects from 2D Images

• US 20150317821A1
• Filed: 04/30/2014
• Published: 11/05/2015
• Est. Priority Date: 04/30/2014
• Status: Active Grant

• Current Assignee: Seiko Epson Corporation (Seiko Group)
• Original Assignee: Seiko Epson Corporation (Seiko Group)
• Inventors: Xiao, Jing, Ding, Yuanyuan
• Granted Patent: US 9,436,987 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 15/10 Geometric effects
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/00 Image analysis
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/187 involving region growing; involving region merging; involving connected component labelling
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes

Abstract

A stereo camera system produces a stereo image pair of a cable harness, which is used to define a 3D point cloud of the cable harness at its current pose position. Pose information of specific parts of the cable harness are determined from the 3D point cloud, and the cable harness is then re-presented as a collection of primitive geometric shapes of known dimensions, whose positions and orientations follow the spatial position and orientation of the imaged cable harness. The length, position and number of geometric shapes are atomically determined from a 2D image segmentation of one of the images in the stereo image pair.

Claims

1. A method of generating a 3D (i.e. three dimensional) geometric primitive representation of a physical target object, said method comprising:

using a data processing device to implement the following steps;

accessing a 3D point cloud representation of the target object at a given pose position;

accessing a 2D (i.e. two dimensional) image segmentation of the target object, wherein the 2D image segmentation is generated from a 2D image of the target object at said given pose position;

identifying a working region on said 2D image segmentation, said working region having a first pair of opposing boundaries traversing the 2D image segmentation across its principle direction, said first working region being sized to enclose all parts of the 2D image segmentation that lie between its opposing boundaries and being smaller than the total area of 2D image segmentation; and

starting with said working region and repeating for a plurality of said working regions distributed along said 2D segmentation until a majority of said 2D segmentation is processed by working regions, implementing the following steps for each working region;

(i) defining the portion of the 2D image segmentation enclosed by the working region as a 2D image sub-segment;

(ii) defining as a cable-segment direction, a linear direction determined from the gradient of edge pixels within the 2D image sub-segment;

(iii) within the 2D image sub-segment, identifying as first and second key points the pixels having the largest distance perpendicular to the cable-segment direction and still bound by the 2D image sub-segment;

(iv) identifying first and second key 3D points that respectively correspond to the first and second key points on the 2D image sub-segment;

(v) identifying a working 3D sub-cloud comprised of 3D points from the subset of the 3D point cloud that corresponds to the 2D image sub-segment;

(vi) fitting a tangent plane to the working 3D sub-cloud;

(vii) projecting 3D points of the working 3D sub-cloud that are between the first and second key 3D points onto the tangent plane in the normal direction of the tangent plane;

(viii) defining an axis line parallel to the principle direction of the projected 3D points on the tangent plane;

(xi) projecting 3D points of the working 3D sub-cloud that are between the first and second key 3D points to the axis line;

(x) using the first and second key 3D points to define a length for a most-current 3D geometric primitive shape oriented along the axis line and encompassing at least a portion of the working 3D sub-cloud; and

(xi) using the projected 3D points within the working 3D sub-cloud to the axis line to define at least one of a width, height and radius of the most-current 3D geometric primitive shape.

2. The method of claim 1, wherein:

after the step of accessing the 2D image segmentation of the target object,

identifying a first end-point and a second end-point of the 2D image segmentation, said first end-point being at an opposite end of the 2D image segmentation as the second end-point;

determining the principle direction of the 2D image segmentation from the first end-point to the second end-point;

the first of said plurality of working regions encompasses the first end-point, said plurality of working regions are distributed along said 2D segmentation along the determined principle direction of the 2D image segmentation toward the second-end point, and the last of said plurality of working regions encompasses the second-end point; and

after step (xi), each additional working region is defined by;

(xii) IF the second end-point is not within the part of the 2D image segmentation that corresponds to the 3D points encompassed by the most-current 3D geometric primitive shape,

THEN defining a new working region on said 2D image segmentation along its principle direction toward the second end-point and excluding the part of the 2D image segmentation that corresponds to the 3D points encompassed by the most-current 3D geometric primitive shape, said new region having a first pair of opposing boundaries traversing the 2D image segmentation across its principle direction, said new region being sized to enclose all parts of the 2D image segmentation that lie between its opposing boundaries, and redefining said new region as a working region and proceeding to step (i);

ELSE defining the chain of previously created 3D geometric primitive shapes as the generated 3D geometric primitive representation of the physical target object.

3. The method of claim 1, wherein the two pixels on the 2D image segmentation having the longest geodesic distance between them are identified as said first and second end-points, respectively.

4. The method of claim 1, wherein in step (v), the identified working 3D sub-cloud is comprised of all the 3D points within the subset of the 3D point cloud that corresponds to the 2D image sub-segment.

5. The method of claim 1, wherein in step (v), the identified working 3D sub-cloud is comprised of only the 3D points that are between the first and second key 3D points.

6. The method of claim 1, wherein said working region is a geometric shape.

7. The method of claim 1, wherein said working region is a parallelogram.

8. The method of claim 7, wherein said working region is a square.

9. The method of claim 7, wherein the parallelogram is angled parallel to the cable-segment direction of step (ii). and the principle direction is determined from the gradient of the edge pixels in the 2D image sub-segmentation.

10. The method of claim 1, wherein in step (ii), the cable-segment direction is defined as the principle direction of the edge pixels within the 2D image sub-segment.

11. The method of claim 1, wherein step (viii) includes determining the principle direction of projected 3D points on the tangent plane.

12. The method of claim 1, wherein in step (viii) the axis line traverses the working 3D sub-cloud.

13. The method of claim 12, wherein axis line is concentric to the working 3D sub-cloud.

14. The method of claim 1, wherein in step (viii), the principle direction of the projected 3D points on the tangent plane defines the axis line.

15. The method of claim 1, wherein in step (ix), the 3D points within the working 3D sub-cloud are projected perpendicularly to the axis line.

16. The method of claim 1, wherein step (x) includes defining first and second boundary lines perpendicular to the axis line and respectively traversing the first and second key 3D points, wherein the distance between the first and second boundary lines defines the length of most-current 3D geometric primitive shape.

17. The method of claim 1, wherein in step (xi), if the most current 3D geometric primitive is a cylinder, then the radius of the cylinder is determined to be not smaller than the longest projection distance among the projected 3D points.

18. The method of claim 1, wherein in step (xi), if the most current 3D geometric primitive is a cuboid with its length determined from the first and second key 3D points, then its height and width are defined to not extend beyond necessary to fully encompass all the projected 3D points.

19. The method of claim 1, wherein in step (xii), the a new region is selected to be defined if the part of the 2D image segmentation that corresponds to the 3D points encompassed by the most-current 3D geometric primitive shape is not within a predefined geodesic distance from second end-point.

20. The method of claim 2, wherein the new region of step (xii) is of the same shape and size as the most previous working region.