3-D reconstruction engine

• US 20030091227A1
• Filed: 11/09/2001
• Published: 05/15/2003
• Est. Priority Date: 11/09/2001
• Status: Active Grant

• Current Assignee: ArcSoft, Inc.
• Original Assignee: ArcSoft, Inc.
• Inventors: Fan, Xing, Jin, Yiqing, Wu, Donghui, Chang, Chu-Fei, Sun, Jie
• Granted Patent: US 7,046,840 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/593 from stereo images

Abstract

Systems and methods for generating three-dimensional models of an object use images having unmeasured camera parameters. Camera calibration determines the perspective of the camera from the content of the images. A background having a pattern with a known marks in each image can facilitate determination of the camera parameters. One background pattern includes separated marks having rectangular sections where corners of the rectangular sections provide calibrations points for the camera parameters. The camera parameters can also be determined by matching features of the object in different images and determining differences in perspective from differences in the appearance of the matched features in different images. A combination of projective and metric reconstructions provides robust reconstruction.

Claims

1. An image processing method comprising:

analyzing a plurality of image of an object, wherein for each image, analyzing comprises;

analyzing the image of an object to identify a mark from among a plurality of marks in a background of the object;

identifying locations of a plurality of calibration points on the mark; and

using the calibration points in determining camera parameters for the image; and

generating a three-dimensional model of the object from the images and the camera parameters determined from the images.

2. The method of claim 1, wherein the mark comprises a first rectangular segment having corners that are among the calibration points.

3. The method of claim 2, wherein the mark further comprises a second rectangular segment.

4. The method of claim 3, wherein the first and second rectangular segments make the mark L-shaped.

5. The method of claim 1, further comprising setting the object in a position relative to a sheet containing the background so that the background appears in the images of the object.

6. The method of claim 5, wherein setting the object comprises setting the object on the sheet.

7. The method of claim 1, further comprising taking the images of the object using a camera having an unmeasured orientation relative to the object.

8. The method of claim 1, wherein for each image, the identified mark is separated from the object in the image.

9. The method of claim 1, wherein determining the camera parameters for an image comprises determining a transform from three-dimensional coordinates to two-dimensional image coordinates.

10. The method of claim 9, wherein generating the three dimensional model comprises:

identifying a silhouette of the object in a selected one of the images;

applying the transform for the selected image to three-dimensional coordinates of points from a candidate volume; and

constructing an approximate volume of the object from the points that the transform maps from the candidate volume on to the silhouette of the object in the selected image.

11. The method of claim 10, further comprising:

determining whether a three-dimensional model constructed from the approximate volume is suitable; and

in response to the three-dimensional model being unsuitable,

determining a transform from three-dimensional coordinates to two-dimensional image coordinates in an added image;

identifying a silhouette of the object in the added image;

applying the transform for the added image to three-dimensional coordinates of points in the approximate volume; and

refining the approximate volume of the object to contain only points that the transform for the added image maps to the silhouette of the object in the added image.

12. The method of claim 11, further comprising repeating steps listed in claim 11 until the three-dimensional model is suitable.

13. The method of claim 10, wherein identifying the silhouetted of the object in a selected image comprises:

analyzing the image using software that distinguishes the silhouette of the object from the background; and

providing a user with a method for modifying of the silhouette as distinguished by the software.

14. The method of claim 1, wherein generating a three-dimensional model of the object constructing a mesh of polygons that approximate the surface of the object.

15. The method of claim 14, wherein generating a three-dimensional model of the object further comprises providing a user with a method for modifying the mesh.

16. An image processing method comprising:

analyzing a set of images containing an object and a background to identify for each image a silhouette of an object and locations in the image of pattern points in the background;

for each image, using the locations of the pattern points to determine a transform from three-dimensional coordinates to two-dimensional coordinates in the image;

transforming three-dimensional coordinates of a set of points in a candidate volume to the two-dimensional coordinates of a first of the images; and

identify an approximate volume of the object as containing the points that the transform maps onto the silhouette of the object in the first of the images.

17. The method of claim 16, further comprising using the points in the approximate volume to construct a mesh of polygons that collectively approximate the surface of the object.

18. The method of claim 17, further comprising mapping a texture from the images to the polygons.

19. The method of claim 16, further generating the candidate volume using the transform and silhouette for a second of the images.

20. A texture mapping process comprising:

for each polygon in a mesh representing a surface of an object, constructing a list of images wherein images in the list are ranked as sources for texture for the polygon;

identifying groups of the polygons that are contiguous and have matching highest rank image in respective lists; and

identifying a first of the groups, which has a first image as the highest ranking image, and a second group, which has a second image as the highest ranking image;

selecting the second image as a source for texture of the first group of polygons provided that the second image is in respective lists for the polygons in the first group.

21. The process of claim 20, wherein constructing the list for a polygon comprises:

listing candidate images with rank in the list according to a direction of the camera axis of the image relative to the normal vector of the polygon;

determining a projection of the polygon into each of the images in the list; and

removing from the list any candidate images wherein the projection of the polygon in the image overlaps a projection of another polygon that is in the mesh and closes to a camera that took the image.

22. The process of claim 20, wherein constructing the list for a polygon comprises:

listing candidate images with rank in the list according to a direction of the camera axis of the image relative to the normal vector of the polygon;

determining a projection of the polygon into each of the images in the list; and

removing from the list any candidate images wherein the projection of the polygon in the image extends outside a silhouette of the object.

23. The process of claim 20, further comprising changing colors of pixels in one or more of images to make contrast more consistent in the images.

24. The process of claim 20, further comprising:

identifying a first image that has highest rank in fewer than a threshold number of lists for the polygons; and

removing the first image from all of the lists.

25. The process of claim 20, further comprising constructing a texture image from multiple images.

26. The process of claim 20, wherein for each polygon, constructing the list of images comprises providing a user with a method for selecting one of images as the sources for texture for the polygon.

27. A method for constructing a three-dimensional model of an object, comprising:

analyzing a plurality of images to identify features in the images;

analyzing pairs of the images to determine for each pair a fundamental matrix that maps identified features in one image of the pair to identified features in another image of the pair;

analyzing triplets of the images to determine for each triplet a trifocal tensor;

performing a projective reconstruction using the fundamental matrices and the trifocal tensors to construct a first model of the object;

performing a metric reconstruction using the fundamental matrices to construct a second model of the object; and

determining the three-dimensional model of the object from the first and second models.

28. A reconstruction engine comprising:

a silhouette extraction unit capable of extracting a silhouette of an object and locations of pattern marks from an image;

a volume generator that receives the location of the pattern marks in an image and determines a transform for the image and determines three-dimensional coordinates of points on a surface of the object from the transform and the silhouettes; and

a reconstruction unit that constructs a three-dimensional model of the object from the three-dimensional coordinates of the points on the surface of the object.

29. The reconstruction engine of claim 28, wherein the silhouette extraction unit and the volume generator are software.