Photograph localization in a three-dimensional model

• US 9,741,121 B2
• Filed: 03/10/2016
• Issued: 08/22/2017
• Est. Priority Date: 10/21/2013
• Status: Active Grant

• Current Assignee: Autodesk, Inc.
• Original Assignee: Autodesk, Inc.
• Inventors: Schmidt, Ryan, Attar, Ramtin, Khan, Azam
• Primary Examiner(s): Desire, Gregory M
• Application Number:
  US15/067,077
• Publication Number:
  US 20160196655A1
• Time in Patent Office: 530 Days
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects

Abstract

A photo localization application is configured to determine the location that an image depicts relative to a 3D representation of a structure. The 3D representation may be a 3D model, color range scan, or gray scale range scan of the structure. The image depicts a particular section of the structure. The photo localization application extracts and stores features from the 3D representation in a database. The photo localization application then extracts features from the image and compares those features against the database to identify matching features. The matching features form a location fingerprint, from which the photo localization application determines the location that the image depicts, relative to the 3D representation. The location allows the user to better understand and communicate information captured by the image.

Claims

1. A computer-implemented method for determining a location within a structure, the method comprising:

identifying a first feature included in a first image depicting the structure;

rendering, via a processor, a second image depicting a first location within a three-dimensional (3D) representation of the structure; and

estimating a location in the structure depicted in the first image based on the first location depicted in the second image.

2. The method of claim 1, further comprising:

selecting one or more additional locations within the 3D representation of the structure; and

for each additional location;

extracting one or more features from the 3D representation; and

associating in a database the one or more features with the additional location.

3. The method of claim 2, wherein extracting the one or more features comprises:

positioning a virtual camera at one of the additional locations;

rendering a third image with the virtual camera that represents a portion of the 3D representation; and

extracting one or more image features from the third image.

4. The method of claim 2, wherein the one or more features correspond to physical features of the structure.

5. The method of claim 1, wherein the first image comprises a two-dimensional (2D) image.

6. The method of claim 1, wherein the first image comprises a stereo image.

7. The method of claim 6, wherein identifying the first feature included in the first image comprises:

creating a 3D depth map based on the first image; and

identifying one or more 3D features included in the 3D depth map.

8. The method of claim 1, further comprising storing the location in the structure depicted in the first image as numeric coordinates within the 3D representation or as a text description of the location.

9. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the steps of:

identifying a first feature included in a first image depicting the structure;

rendering, via the processor, a second image depicting a first location within a three-dimensional (3D) representation of the structure; and

estimating a location in the structure depicted in the first image based on the first location depicted in the second image.

10. The non-transitory computer-readable medium of claim 9, further comprising:

selecting one or more additional locations within the 3D representation of the structure; and

for each additional location;

extracting one or more features from the 3D representation; and

associating in a database the one or more features with the additional location.

11. The non-transitory computer-readable medium of claim 10, wherein extracting the one or more features comprises:

positioning a virtual camera at one of the additional locations;

rendering a third image with the virtual camera that represents a portion of the 3D representation; and

extracting one or more image features from the third image.

12. The non-transitory computer-readable medium of claim 10, wherein the one or more features correspond to physical features of the structure.

13. The non-transitory computer-readable medium of claim 9, wherein the first image comprises a two-dimensional (2D) image.

14. The non-transitory computer-readable medium of claim 9, wherein the first image comprises a stereo image.

15. The non-transitory computer-readable medium of claim 14, wherein identifying the first feature included in the first image comprises:

creating a 3D depth map based on the first image; and

identifying one or more 3D features included in the 3D depth map.

16. The non-transitory computer-readable medium of claim 9, further comprising storing the location in the structure depicted in the first image as numeric coordinates within the 3D representation or as a text description of the location.

17. A computer system, comprising:

a memory storing instructions; and

a processor that is coupled to the memory and, when executing the instructions, is configured to perform the steps of;

identifying a first feature included in a first image depicting the structure;

rendering, via the processor, a second image depicting a first location within a three-dimensional (3D) representation of the structure; and

estimating a location in the structure depicted in the first image based on the first location depicted in the second image.

18. The computer system of claim 17, wherein the processor is further configured to perform the steps of:

selecting one or more additional locations within the 3D representation of the structure; and

for each additional location;

extracting one or more features from the 3D representation; and

associating in a database the one or more features with the additional location.

19. The computer system of claim 18, wherein extracting the one or more features comprises:

positioning a virtual camera at one of the additional locations;

rendering a third image with the virtual camera that represents a portion of the 3D representation; and

extracting one or more image features from the third image.

20. The computer system of claim 18, wherein the extracted features correspond to physical features of the structure.