Systems and methods for generating three-dimensional models using sensed position data

• US 9,542,773 B2
• Filed: 05/23/2013
• Issued: 01/10/2017
• Est. Priority Date: 05/23/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Pan, Weibin, Hu, Liang
• Primary Examiner(s): Drennan, Barry
• Assistant Examiner(s): PRINGLE-PARKER, JASON A
• Application Number:
  US14/761,803
• Publication Number:
  US 20150363971A1
• Time in Patent Office: 1,328 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2200/21 involving computational photography
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2219/2016 Rotation, translation, scaling
• CPC Class Codes: G06T 3/60 Rotation of whole images or parts thereof
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: H04N 13/207 using a single 2D image sensor

Abstract

Embodiments include a computer-implemented method for generating a three-dimensional (3D) model. The method includes receiving a first and second sets of sensed position data indicative of a position of a camera device(s) at or near a time when it is used to acquire first and second images of an image pair, respectively, determining a sensed rotation matrix and/or a sensed translation vector for the image pair using the first and second sets of sensed position data, identifying a calculated transformation including a calculated translation vector and rotation matrix, generating a sensed camera transformation including the sensed rotation matrix and/or the sensed translation vector, and, if the sensed camera transformation is associated with a lower error than the calculated camera transformation, using it to generate a 3D model.

Claims

1. A computer-implemented method for generating a three-dimensional (3D) model, the method comprising:

receiving, by one or more computing devices, a first set of sensed position data indicative of an orientation of a camera device used to acquire a first two-dimensional (2D) image, wherein the first set of sensed position data is provided by a sensor of the camera device used to acquire the first 2D image;

receiving, by the one or more computing devices, a second set of sensed position data indicative of an orientation of a camera device used to acquire a second two-dimensional (2D) image, wherein the second set of sensed position data is provided by a sensor of the camera device used to acquire the second 2D Image;

determining, by the one or more computing devices, a sensed rotation matrix for an image pair comprising the first and second 2D images using the first and second sets of sensed position data;

identifying, by the one or more computing devices, a calculated camera transformation matrix for the image pair, the calculated transformation comprising a calculated translation vector and a calculated rotation matrix, wherein identifying a calculated camera transformation, comprises,

deriving a plurality of candidate calculated transformation matrices using a set of matching points of the first and second 2D images, wherein the candidate calculated transformation matrices each comprise a translation component and a calculated rotation matrix;

identifying a candidate calculated transformation matrix of the plurality of candidate calculated transformation matrices that is associated with the lowest transformation error; and

identifying the candidate calculated transformation matrix that is associated with the lowest transformation error as the calculated camera transformation;

generating, by the one or more computing devices, a sensed camera transformation matrix for the image pair, the sensed camera transformation comprising a translation vector and the sensed rotation matrix;

identifying, by the one or more computing devices, a set of matching points of the first and second 2D images;

determining, by the one or more computing devices, whether a first error associated with a transformation of the set of matching points using the sensed camera transformation is less than a second error associated with a transformation of the set of matching points using the calculated camera transformation; and

in response to determining, by the one or more computing devices, that a first error associated with a transformation of the set of matching points using the sensed camera transformation is less than a second error associated with a transformation of the set of matching points using the calculated camera transformation, generating a 3D model using the sensed camera transformation; and

storing, by the one or more computing devices, the 3D model in an 3D model repository.

2. The method of claim 1, wherein deriving a plurality of candidate calculated transformation matrices using a set of matching points comprises:

computing a fundamental matrix between the first and second 2D images using a set of matching points;

computing an essential matrix based on the fundamental matrix; and

decomposing the essential matrix into four candidate calculated transformation matrices via singular value decomposition.

3. The method of claim 1, wherein the set of matching points comprises a subset of a set of matching points of the first and second 2D images.

4. The method of claim 3, wherein the set of matching points comprises five matching points of a set of matching points of the first and second 2D images.

5. The method of claim 1, wherein determining whether a first error associated with a transformation of the set of matching points using the sensed camera transformation is less than a second error associated with a transformation of the set of matching points using the calculated camera transformation comprises:

selecting a subset of a set of matching points of the first and second 2D images;

determining a first error associated with a transformation between the subset of matching points using the calculated camera transformation;

determining a second error associated with a transformation between the subset of matching points using the sensed camera transformation; and

comparing the first error and the second error to determine whether the first error associated with a transformation of the subset of matching points using the sensed camera transformation is less than a second error associated with a transformation of the subset of matching points using the calculated camera transformation.

6. The method of claim 1, wherein generating a 3D model using the sensed camera transformation matrix comprises:

transforming matching points of the set of matching points to 3D coordinates; and

adding the 3D coordinates to the 3D model.

7. The method of claim 1, wherein the translation vector of the calculated camera transformation and the translation vector of the sensed camera transformation are the same.

8. The method of claim 1, wherein the first set of sensed position data is indicative of a location of a camera device at or near a time when it is used to acquire a first two-dimensional (2D) image, wherein the second set of sensed position data is indicative of a location of a camera device at or near a time when it is used to acquire a second two-dimensional (2D) image, the method further comprising:

determining a sensed rotation matrix for an image pair comprising the first and second 2D images using the first and second sets of sensed position data, wherein the translation vector of the sensed camera transformation comprises the sensed rotation matrix, and wherein the translation vector of the calculated camera transformation is calculated based on matching points of the first and second 2D image.

9. The method of claim 1, wherein the sensor comprises a gyroscope.

10. A non-transitory computer readable medium comprising program instructions stored thereon that are executable by a processor to cause the following steps for generating a three-dimensional (3D) model:

receiving a first set of sensed position data indicative of an orientation of a camera device used to acquire a first two-dimensional (2D) image, wherein the first set of sensed position data is provided by a sensor of the camera device used to acquire the first 2D image;

receiving a second set of sensed position data indicative of an orientation of a camera device used to acquire a second two-dimensional (2D) image, wherein the second set of sensed position data is provided by a sensor of the camera device used to acquire the second 2D Image;

determining a sensed rotation matrix for an image pair comprising the first and second 2D images using the first and second sets of sensed position data;

identifying a calculated camera transformation matrix for the image pair, the calculated transformation comprising a calculated translation vector and a calculated rotation matrix, wherein identifying a calculated camera transformation, comprises,

deriving a plurality of candidate calculated transformation matrices using a set of matching points of the first and second 2D images, wherein the candidate calculated transformation matrices each comprise a translation component and a calculated rotation matrix;

identifying a candidate calculated transformation matrix of the plurality of candidate calculated transformation matrices that is associated with the lowest transformation error; and

identifying the candidate calculated transformation matrix that is associated with the lowest transformation error as the calculated camera transformation;

generating a sensed camera transformation matrix for the image pair, the sensed camera transformation comprising a translation vector and the sensed rotation matrix;

identifying a set of matching points of the first and second 2D images;

determining whether a first error associated with a transformation of the set of matching points using the sensed camera transformation is less than a second error associated with a transformation of the set of matching points using the calculated camera transformation; and

in response to determining that a first error associated with a transformation of the set of matching points using the sensed camera transformation is less than a second error associated with a transformation of the set of matching points using the calculated camera transformation, generating a 3D model using the sensed camera transformation; and

storing the 3D model in an 3D model repository.

11. The medium of claim 10, wherein deriving a plurality of candidate calculated transformation matrices using a set of matching points comprises:

computing a fundamental matrix between the first and second 2D images using a set of matching points;

computing an essential matrix based on the fundamental matrix; and

decomposing the essential matrix into four candidate calculated transformation matrices via singular value decomposition.

12. The medium of claim 10, wherein the set of matching points comprises a subset of a set of matching points of the first and second 2D images.

13. The medium of claim 12, wherein the set of matching points comprises five matching points of a set of matching points of the first and second 2D images.

14. The medium of claim 10, wherein determining whether a first error associated with a transformation of the set of matching points using the sensed camera transformation is less than a second error associated with a transformation of the set of matching points using the calculated camera transformation comprises:

selecting a subset of a set of matching points of the first and second 2D images;

determining a first error associated with a transformation between the subset of matching points using the calculated camera transformation;

determining a second error associated with a transformation between the subset of matching points using the sensed camera transformation; and

comparing the first error and the second error to determine whether the first error associated with a transformation of the subset of matching points using the sensed camera transformation is less than a second error associated with a transformation of the subset of matching points using the calculated camera transformation.

15. The medium of claim 10, wherein generating a 3D model using the sensed camera transformation matrix comprises:

transforming matching points of the set of matching points to 3D coordinates; and

adding the 3D coordinates to the 3D model.

16. The medium of claim 10, wherein the translation vector of the calculated camera transformation and the translation vector of the sensed camera transformation are the same.

17. The medium of claim 10, wherein the first set of sensed position data is indicative of a location of a camera device at or near a time when it is used to acquire a first two-dimensional (2D) image, wherein the second set of sensed position data is indicative of a location of a camera device at or near a time when it is used to acquire a second two-dimensional (2D) image, the method further comprising:

determining a sensed rotation matrix for an image pair comprising the first and second 2D images using the first and second sets of sensed position data, wherein the translation vector of the sensed camera transformation comprises the sensed rotation matrix, and wherein the translation vector of the calculated camera transformation is calculated based on matching points of the first and second 2D image.

18. The medium of claim 10, wherein the sensor comprises a gyroscope.

19. A computer-implemented method for generating a three-dimensional (3D) model, the method comprising:

receiving, by one or more computing devices, a first set of sensed position data indicative of a position of a camera device used to acquire a first two-dimensional (2D) image, wherein the first set of sensed position data is provided by an integrated sensor of the camera device used to acquire the first 2D image;

receiving, by the one or more computing devices, a second set of positioning sensor data indicative of a position of a camera device used to acquire a second two-dimensional (2D) image, wherein the second set of sensed position data is provided by an integrated sensor of the camera device used to acquire the second 2D image;

determining, by the one or more computing devices, a sensed rotation matrix and/or a sensed translation vector between the first and second 2D images using the first and second sets of sensed position data;

identifying, by the one or more computing devices, a calculated camera transformation comprising a calculated translation vector and a calculated rotation matrix, wherein identifying a calculated camera transformation, comprises,

deriving a plurality of candidate calculated transformation matrices using a set of matching points of the first and second 2D images, wherein the candidate calculated transformation matrices each comprise a translation component and a calculated rotation matrix;

identifying a candidate calculated transformation matrix of the plurality of candidate calculated transformation matrices that is associated with the lowest transformation error; and

identifying the candidate calculated transformation matrix that is associated with the lowest transformation error as the calculated camera transformation;

generating, by the one or more computing devices, a sensed camera transformation comprising the sensed rotation matrix and/or the sensed translation vector;

identifying, by the one or more computing devices, a set of matching points of the first and second 2D images;

determining, by the one or more computing devices, whether a first error associated with a transformation of the set of matching points using the sensed camera transformation is less than a second error associated with a transformation of the set of matching points using the calculated camera transformation; and

in response to determining, by the one or more computing devices, that a first error associated with a transformation of the set of matching points using the sensed camera transformation is less than a second error associated with a transformation of the set of matching points using the calculated camera transformation, generating a 3D model using the sensed camera transformation; and

storing, by the one or more computing devices, the 3D model in an 3D model repository.