Model and sizing information from smartphone acquired image sequences

• US 9,619,933 B2
• Filed: 06/16/2015
• Issued: 04/11/2017
• Est. Priority Date: 06/16/2014
• Status: Active Grant

• Current Assignee: Occipital, Inc.
• Original Assignee: Occipital, Inc.
• Inventors: Spinella-Mamo, Vincent Paul, Minor, Brandon, Roberts, Isaac, Morrison, John
• Primary Examiner(s): Chin, Michelle
• Application Number:
  US14/740,323
• Publication Number:
  US 20150363970A1
• Time in Patent Office: 665 Days
• Field of Search: None
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/596 from three or more stereo images

Abstract

A method for providing a model file to a user. The user acquires a plurality of images using a device, such as a smart phone, with at least a monocular image capture device. The plurality of images are then processed to provide the user with a model file. The model file is created by a minimization of an energy which is done using a primal-dual hybrid gradient. The model file may be used in a printer capable of printing in three dimensions. The user is either charged a fee for uploading the plurality of images, downloading the model file, or both.

Claims

1. A system for providing a user with a 3D model including a non-transitory computer readable medium having instruction that, when executed, cause one or more processers to perform the following steps:

a. receive a plurality of images from an image capture device which was at least 15 cm from a surface, but not more than 1 m from the surface;

b. determine a pose of the image capture device for each image in the plurality of images, camera intrinsics for the capture device including a focal point and an image center, and one or more radial distortion factors;

c. rectify each image in the plurality of images using the one or more radial distortion factors;

d. create a buffer of 60 sequential images from the plurality of images;

e. select a reference image from the buffer;

f. select a subset of 20 comparison images from the buffer such that each comparison image is at least 0.5 cm to 2.0 cm from any other comparison image and the reference image;

g. generate a cost volume from the subset of comparison images and the reference image;

h. estimate a depth image from the cost volume;

i. fuse the depth image into a signed distance function (SDF);

j. shift the buffer by removing the first 20 frames and adding 20 more sequential images from the plurality of images to the end of the buffer;

k. repeat steps e through j until there are no more images in the plurality of images;

l. create a 3D model from the signed distance function; and

m. send the 3D model to an end user, wherein the 3D model generated is of an object placed on the surface.

2. The system of claim 1, wherein the cost volume is a volumetric representation, having a two dimensional pixel coordinate and a one dimensional inverse depth coordinate, each voxel of the volume storing an average reprojection error between each pixel in the reference image with each corresponding pixel of every image of the comparison images parameterized for a number of levels of inverse depth from a minimum inverse depth to a maximum inverse depth and further where the reprojection error is one of the set comprised of an absolute value of a difference in intensity values of each pixel located in the reference image and each corresponding pixel in every comparison image, a sum of absolute differences of a patch centered at each pixel in the reference image and each corresponding pixel in every comparison image, a sum of squared differences of a patch centered at each pixel in the reference image and a patch centered at each corresponding pixel in every comparison image, and a Hamming distance of a census value of each pixel in the reference image and each corresponding pixel in every comparison image, and further wherein the corresponding pixel is calculated via a warping function defined as follows:

3. The system of claim 2, wherein the number of levels of inverse depth is 64 and the minimum inverse depth is 0.4 m⁻

1 and the maximum inverse depth is 5 m<sup>−

1</sup>.

4. The system of claim 1, wherein the depth image is estimated from the cost volume by performing a search for a minimum reprojection error at each pixel coordinate along the inverse depth coordinate and further wherein the inverse of the inverse depth coordinate corresponding to the mimimum reprojection error represents the depth at each pixel coordinate and further wherein the linear search over the cost volume is performed in parallel on a graphics processing unit.

5. The system of claim 4, wherein the depth image is further refined by performing the steps comprising:

a. denoising the depth image by minimizing the energy of a total variation regularizer of depth and an L1 norm of the difference of the initial depth estimate and a denoised depth estimate using a primal-dual hybrid gradient descent in parallel on a graphics processing unit to create a denoised depth image;

b. using the denoised depth image as an initialization for a wide-baseline refinement by using four additional images from the buffer which are further from the reference image than the comparison images, but not more than 80 cm away from the reference image and performing a primal-dual hybrid gradient on an L1 norm of an error function representation of a first order linear expansion of a sum of a photometric error of the four additional images with respect to a depth coordinate in addition to a weighted Huber regularizer, performed on a graphics processing unit, the error function representation described mathematically as

E(x,d)=I_r(x)−

[I_k(w(x,k,d₀))+(d−

d₀)∇

_dI_k(w(x,k,d₀))].

6. The system of claim 1, wherein the pose of the image capture device associated with every image is determined by placing a grid consisting of a regular pattern of rows and columns of a first shape and a second shape of known sizes on the surface, such that any observed subset of the grid provides an initial estimate of a pose for every observation and subsequently performing a non-linear least squares optimization over a cost comprised of pose information and camera calibration information on the initial poses for every observation of the surface to refine the pose associated with every image, as well as determine the camera intrinsics and one or more radial distortion factors.

7. The system of claim 6, wherein the first shape is a circle and the second shape is a circle having a radius such that it can be differentiated from the first shape.

8. The system of claim 6, wherein each pose of an image capture device is further refined by performing the following steps:

a. performing a raycasting operation on the SDF from the pose of the reference frame to generate a reference depth image and a weighting image; and

b. performing a non-linear least squares optimization of the pose by minimizing a reprojected cost for every pixel in the reference image as compared with a comparison image as reprojected through the reference depth image and further wherein the optimization is initialized with the relative pose between the reference image and the comparison image and further where the reprojection cost of every pixel is weighted by a corresponding pixel in the weighting image.

9. The system of claim 1, wherein the pose and camera intrinsics of the image capture device for every image is determined by extracting features selected from the set comprised of FAST, FREAK, ORB, BRIEF, SIFT, SURF, and AKAZE for every image in the sequence of images, performing feature matching between features extracted on every images to determine a set of matched features, and implementing a non-linear least squares optimization for the poses and camera calibration and one or more radial distortion parameters using a cost function which is a function of reprojection error of the matched features.

10. The system of claim 9, wherein the cost function also comprises other sensor data received by the one or more processors and further wherein the other sensor data is data from the set comprised of laser range data, accelerometer data, gyroscopic data, magnetic field data, and global positioning system data.

11. The system of claim 1, wherein the fusing of depth estimates is done by projecting the depth values from the pose of the image capture device into the SDF, the SDF having 400×

400×

400 contiguous voxel elements, each voxel representing 1/400th of a meter on each side such that the SDF as a whole represents 1 m³, and further wherein the SDF is initialized at a location by performing a non-linear least squares optimization to determine a principle focus point, where the principle focus point is a point which is observed in a most amount of images from the image sequence, using an X and a Y coordinate of the principle point to center the SDF on an X-Y plane, and choosing a Z-coordinate of the SDF such that a lowest level of the SDF is located 5 cm below the surface.

12. The system of claim 1, wherein the 3D model is created from the SDF by assigning each voxel of the SDF an index, where the index is based on distance information contained in all neighboring voxels, and further wherein a mesh element is created at a voxel position based on the index.

13. The system of claim 1, wherein a desired resolution of the 3D model is selected by a user and, based on the resolution selected, the end user is charged a fee, and further wherein the 3D model is not transferred to the end user until the fee has been paid.

14. The system of claim 1, wherein the 3D model is of an object selected from the set comprised of a foot, a hand, a body, and a head, and further wherein the 3D model is used to suggest an optimal garment size.

15. The system of claim 1, wherein the 3D model is sent to a 3D printer.

16. The system of claim 1, wherein the image capture device and the one or more processers are not co-located on, or connected within, a single device.

17. A method for providing a user with a 3D model comprising the following steps:

a. sending a plurality of images from an image capture device which was at least 15 cm from a surface, but not more than 1 m from the surface to one or more processors;

b. determining a pose of the image capture device for each image in the plurality of images, camera intrinsics for the capture device including a focal point, an image center, and one or more radial distortion factors;

c. rectifying each image in the plurality of images using the one or more radial distortion factors;

d. creating a buffer of 60 sequential images from the plurality of images;

e. selecting a reference image from the buffer;

f. selecting a subset of 20 comparison images from the buffer such that each comparison image is at least 0.5 cm to 2 cm from any other comparison image or the reference image;

g. generating a cost volume from the subset of comparison images and the reference image;

h. estimating a depth image from the cost volume;

i. fusing the depth image into a signed distance function (SDF);

j. shifting the buffer by removing the first 20 frames and adding 20 more sequential images from the plurality of images to the end of the buffer;

k. repeating steps e through j until there are no more images in the plurality of images;

l. creating a 3D model from the signed distance function; and

m. sending the 3D model to an end user, wherein the 3D model generated is of an object placed on the surface.

18. The method of claim 17, wherein the pose of the image capture device associated with every image is determined by placing a grid consisting of a regular pattern of rows and columns of a first shape and a second shape of known sizes on the surface, such that any observed subset of the grid provides an initial estimate of a pose for every observation and subsequently performing a non-linear least squares optimization over a cost comprised of pose information and camera calibration information on the initial poses for every observation of the surface to refine the pose associated with every image, as well as determine the camera intrinsics and one or more radial distortion factors.

19. The method of claim 17, wherein the pose and camera intrinsics of the image capture device for every image is determined by extracting features selected from the set comprised of FAST, FREAK, ORB, BRIEF, SIFT, SURF, and AKAZE for every image in the sequence of images, performing feature matching between features extracted on every images to determine a set of matched features, and implementing a non-linear least squares optimization for the poses and camera calibration and one or more radial distortion parameters using a cost function which is a function of reprojection error of the matched features.

20. The method of claim 17, wherein a desired resolution of the 3D model is selected by the end user and, based on the desired resolution, the end user is charged a fee, and further wherein the 3D model is not transferred to the end user until the fee has been paid.