3-D MODEL GENERATION

• US 20150381968A1
• Filed: 06/27/2014
• Published: 12/31/2015
• Est. Priority Date: 06/27/2014
• Status: Active Grant

• Current Assignee: A9.com Incorporated (Amazon.com, Inc.)
• Original Assignee: A9.com Incorporated (Amazon.com, Inc.)
• Inventors: Dhua, Arnab Sanat Kumar, Ramesh, Sunil, Arora, Himanshu, Fang, Chen
• Granted Patent: US 10,574,974 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems

Abstract

Various embodiments provide for the generation of 3D models of objects. For example, depth data and color image data can be captured from viewpoints around an object using a sensor. A camera having a higher resolution can simultaneously capture image data of the object. Features between images captured by the image sensor and the camera can be extracted and compared to determine a mapping between the camera and the image. Once the mapping between the camera and the image sensor is determined, a second mapping between adjacent viewpoints can be determined for each image around the object. In this example, each viewpoint overlaps with an adjacent viewpoint and features extracted from two overlapping viewpoints are matched to determine their relative alignment. Accordingly, a 3D point cloud can be generated and the images captured by the camera can be projected on the surface of the 3D point cloud to generate the 3D model.

Claims

1. A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause a computing device to:

capture, using an image sensor, depth data of an object from a plurality of viewpoints;

capture, using a first camera, first image data of the object from each viewpoint of the plurality of viewpoints, a preregistered process aligning two-dimensional (2D) coordinates of the first image data for the first camera and three-dimensional (3D) coordinates of the depth data for the image sensor;

capture, using a second camera, second image data of the object from each viewpoint of the plurality of viewpoints, the second image data being a higher resolution relative to the first image data;

extract, using a feature extraction algorithm, first features from the first image data of each viewpoint captured by the first camera;

extract, using the feature extraction algorithm, second features from the second image data of each viewpoint captured by the second camera;

determine matching features between the first features and the second features;

determine, using a projective mapping algorithm, a first mapping between the first image data and the second image data for each viewpoint using the matching features, the first mapping providing 3D coordinates for the second features of the second image data captured by the second camera;

determine, for second imaged data of each viewpoint, matching second features between adjacent viewpoints, a first viewpoint having a first field of view at least partially overlapping a second field of view of an adjacent second viewpoint;

determine, for the second imaged data of each viewpoint, a second mapping between the second image data of adjacent viewpoints using a Euclidean mapping algorithm;

generate, using the depth data, a 3D point cloud for the object;

generate, using a mesh reconstruction algorithm, a triangular mesh of the object from the 3D point cloud; and

generate a 3D model of the object by projecting, based at least in part on the 3D coordinates for the second features from first mapping, the second image data onto the triangular mesh for each viewpoint of the plurality of viewpoints using the second mapping.

2. The non-transitory computer-readable storage medium of claim 1, wherein the instructions that, when executed by at least one processor, further cause the computing device to:

select a reference viewpoint;

determine a difference between the reference viewpoint and an estimated mapping for the reference viewpoint; and

adjust one or more viewpoints of the plurality of viewpoints by distributing the difference between the reference viewpoint and the estimated mapping of the reference viewpoint.

3. The non-transitory computer-readable storage medium of claim 1, wherein the instructions that, when executed by at least one processor, further cause the computing device to:

determine, for a set of points of the 3D point cloud, a normal direction by fitting one or more planes to the set of points;

estimate, for the one or more planes, one or more intercepts between planes;

refine the 3D point cloud by uniformly sampling points of the one or more planes;

estimate an error-in-fit between the 3D point cloud and the refined 3D point cloud by comparing at least a portion of the points of the refined 3D point cloud to corresponding points of the 3D point cloud; and

determine, in response to the error-in-fit being greater than a threshold, the object belongs to a cuboid shape class.

4. The non-transitory computer-readable storage medium of claim 1, wherein the instructions that, when executed by at least one processor, further cause the computing device to:

estimate, for each height of the 3D point cloud, a circle, the estimated circle providing a center and a radius;

refine the 3D point cloud by uniformly sampling points of the 3D point cloud, the center of each circle for each height corresponding to a z-axis of the object;

estimate an error-in-fit between the 3D point cloud and the refined 3D point cloud by comparing at least a portion of the points of the refined 3D point cloud to corresponding points of the 3D point cloud; and

determine, in response to the error-in-fit being greater than a threshold, the object belongs to a bottle shape class.

5. A computer-implemented method, comprising:

capturing, using an image sensor, depth information of an object;

capturing, using a first camera, a first image of the object, three-dimensional (3D) coordinates of the depth information being aligned with two-dimensional (2D) coordinates of the first image;

capturing, using a second camera, a second image of the object;

detecting first features in the first image captured by the first camera;

detecting second features in the second image captured by the camera;

determining matching features between the first features and the second features; and

determining 3D coordinates for the second features of the second image based at least in part on the matching features between the first features and the second features.

6. The computer-implemented method of claim 5, further comprising:

capturing, using the image sensor, second depth information of the object;

capturing, using the first camera, a third image of the object;

capturing, using the second camera, a fourth image of the object;

detecting third features from the third image captured by the first camera;

detecting fourth features from the second image captured by the second camera;

determining matching features between the third features and the fourth features; and

determining second 3D coordinates for the fourth features of the fourth image based at least in part on the matching features between the third features and the fourth features.

7. The computer-implemented method of claim 6, wherein.

determining matching second features between the second features of the second image and the fourth features of the fourth image;

determining a mapping between the second image and the fourth image based at least in part on the mapping;

generating, using the depth information, a 3D point cloud for the object; and

generating a 3D model of the object by projecting, based at least in part on the 3D coordinates for the second features and the mapping, the second image and the fourth image onto the 3D point cloud.

8. The computer-implemented method of claim 7, further comprising:

determining a difference between a viewpoint corresponding to the fourth image and the mapping of the fourth image to the second image; and

adjusting at least one of one or more features of the second features of each viewpoint or one or more viewpoints of a plurality of viewpoints by distributing the difference between the viewpoint corresponding to the fourth image and the mapping of the fourth image to the second image.

9. The computer-implemented method of claim 7, further comprising:

estimating, for each height of the 3D point cloud, a circle, the estimated circle providing a center and a radius;

refining the 3D point cloud by uniformly sampling points of the 3D point cloud, the center of each circle for each height corresponding to a z-axis of the object;

estimating an error-in-fit between the 3D point cloud and the refined 3D point cloud by comparing at least a portion of the points of the refined 3D point cloud to corresponding points of the 3D point cloud; and

determining, in response to the error-in-fit being greater than a threshold, the object belongs to a bottle shape class.

10. The computer-implemented method of claim 7, further comprising:

determining, for a set of points of the 3D point cloud, a normal direction by fitting one or more planes to the set of points;

estimating, for the one or more planes, one or more intercepts between planes;

refining the 3D point cloud by uniformly sampling points of the one or more planes;

estimating an error-in-fit between the 3D point cloud and the refined 3D point cloud by comparing at least a portion of the points of the refined 3D point cloud to corresponding points of the 3D point cloud; and

determining, in response to the error-in-fit being greater than a threshold, the object belongs to a cuboid shape class.

11. The computer-implemented method of claim 5, wherein the features are extracted using a feature extraction algorithm, wherein the feature extraction is at least one of Scale-invariant feature transform (SIFT), Speeded Up Robust Features (SURF), or Accumulated Signed Gradient (ASG).

12. The computer-implemented method of claim 5, wherein the mapping is at least one of projective mapping or Euclidean mapping and the mapping is performed using Random Sampling Consensus (RANSAC).

13. A computing system, comprising:

a processor;

an image sensor;

a first camera

a second camera; and

memory including instructions that, when executed by the processor, cause the computing system to;

capture, using the image sensor, depth information of an object;

capture, using the first camera, a first image of the object, three-dimensional (3D) coordinates of the depth information being aligned with two-dimensional (2D) coordinates of the first image;

capture, using the second camera, a second image of the object;

detect first features in the first image captured by the first camera;

detect second features in the second image captured by the second camera;

determine matching features between the first features and the second features; and

determine 3D coordinates for the second features of the second image based at least in part on the matching features between the first features and the second features.

14. The computing system of claim 13, wherein the instructions, when executed by the processor, further enable the computing system to:

capture, using the image sensor, second depth information of the object;

capture, using the first camera, a third image of the object;

capture, using the second camera, a fourth image of the object;

detect third features from the third image captured by the first camera;

detect fourth features from the second image captured by the second camera;

determine matching features between the third features and the fourth features; and

determine second 3D coordinates for the fourth features of the fourth image based at least in part on the matching features between the third features and the fourth features.

15. The computing system of claim 14, wherein the instructions, when executed by the processor, further enable the computing system to:

determine matching second features between the second features of the second image and the fourth features of the fourth image;

determine a mapping between the second image and the fourth image based at least in part on the mapping;

generate, using the depth information, a 3D point cloud for the object; and

generate a 3D model of the object by projecting, based at least in part on the 3D coordinates for the second features and the mapping;

the second image and the fourth image onto the 3D point cloud.

16. The computing system of claim 15, wherein the instructions, when executed by the processor, further enable the computing system to:

determine a difference between a viewpoint corresponding to the fourth image and the mapping of the fourth image to the second image; and

adjust at least one of one or more features of the second features of each viewpoint or one or more viewpoints of a plurality of viewpoints by distributing the difference between the viewpoint corresponding to the fourth image and the mapping of the fourth image to the second image.

17. The computing system of claim 15, wherein the instructions, when executed by the processor, further enable the computing system to:

determine, for a set of points of the 3D point cloud, a normal direction by fitting one or more planes to the set of points;

estimate, for the one or more planes, one or more intercepts between planes;

refine the 3D point cloud by uniformly sampling points of the one or more planes;

estimate an error-in-fit between the 3D point cloud and the refined 3D point cloud by comparing at least a portion of the points of the refined 3D point cloud to corresponding points of the 3D point cloud; and

determine, in response to the error-in-fit being greater than a threshold, the object belongs to a cuboid shape class.

18. The computing system of claim 15, wherein the instructions, when executed by the processor, further enable the computing system to:

estimate, for each height of the 3D point cloud, a circle, the estimated circle providing a center and a radius;

refine the 3D point cloud by uniformly sampling points of the 3D point cloud, the center of each circle for each height corresponding to a z-axis of the object;

estimate an error-in-fit between the 3D point cloud and the refined 3D point cloud by comparing at least a portion of the points of the refined 3D point cloud to corresponding points of the 3D point cloud; and

determine, in response to the error-in-fit being greater than a threshold, the object belongs to a bottle shape class.

19. The computing system of claim 13, wherein the features are extracted using a feature extraction algorithm, wherein the feature extraction is at least one of Scale-invariant feature transform (SIFT), Speeded Up Robust Features (SURF), or Accumulated Signed Gradient (ASG).

20. The computing system of claim 13, wherein the mapping is at least one of projective mapping or Euclidean mapping and the mapping is performed using Random Sampling Consensus (RANSAC).