Automatic method for photo texturing geolocated 3-D models from geolocated imagery

• US 8,339,394 B1
• Filed: 09/26/2011
• Issued: 12/25/2012
• Est. Priority Date: 08/12/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Lininger, Scott
• Primary Examiner(s): HAJNIK, DANIEL F
• Application Number:
  US13/244,787
• Time in Patent Office: 456 Days
• Field of Search: None
• US Class Current: 345/419
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 7/49 based on structural texture description, e.g. using primitives or placement rules
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 30/422 Technical drawings; Geographical maps
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

A method and system for applying photo texture to geolocated 3D models operates within a 3D modeling system. The modeling system includes a modeling application operating on a workstation and a database of geotagged imagery. A 3D model created or edited within the 3D modeling system is geolocated such that every point in the 3D modeling space corresponds to a real world location. For a selected surface, the method and system search the database of imagery to identify in the database one or more images depicting the selected surface of the 3D model. The method and system identify the boundaries of the selected surface within the image by transforming two or more sets of coordinates from the 3D modeling space to a coordinate space corresponding to the image. The portion of the image corresponding to the selected surface is copied and mapped to the selected surface of the 3D model.

Claims

1. A computer-implemented method for applying photo texturing to a 3D model of a real-world object, the method comprising:

receiving geolocation information for a point in a model space in which the 3D model is located;

receiving a selection of a surface of the 3D model;

receiving a request to automatically apply to the selected surface a photo texture;

determining a location of a center point of the selected surface;

determining a first point at a fixed distance from the center point along the normal of the selected surface;

projecting the first point to a ground plane, wherein the projected point on the ground plane becomes a default viewing position;

selecting, according to the received geolocation information, an image to use to photo texture the selected surface, the image selected from among one or more images determined to be located nearby the default viewing position;

selecting an area of the image to apply to the selected surface according to the received geolocation information; and

applying the area of the selected image to the selected surface.

2. The method of claim 1, wherein selecting an image to use to photo texture the selected surface comprises selecting a first image of a plurality of geolocated images.

3. The method of claim 2, wherein selecting a first image comprises:

determining a real-world location from which a real-world surface corresponding to the selected surface could be viewed;

determining one or more geolocated images captured at the determined real-world location; and

selecting as the first image one of the one or more geolocated images captured at the determined real-world location.

4. The method of claim 3, wherein selecting as the first image one of the one or more geolocated images captured at the determined real-world location comprises selecting an image that depicts the real-world surface.

5. The method of claim 1, further comprising determining a location of the vertices of the selected surface.

6. The method of claim 1, wherein selecting an image to use to photo texture the selected surface comprises:

determining one or more images captured from a nearest position to the default viewing position; and

determining which of the one or more images was captured from an angle most closely pointed at the center point of the selected surface.

7. The method of claim 1, wherein selecting an image to use to photo texture the selected surface comprises:

determining an actual viewpoint nearest to the default viewing position, the actual viewpoint being a location associated with one or more geolocated images; and

determining one of the one or more geolocated images having a focal point closest to the center point of the selected surface.

8. The method of claim 1, further comprising:

determining the location of two or more vertices of the selected surface; and

performing a plurality of transformations for each of the two or more vertices to transform each of the vertices through a plurality of coordinate spaces.

9. The method of claim 8, wherein the plurality of coordinate spaces includes at least a first coordinate space defined within a modeling program.

10. The method of claim 8, wherein the plurality of transformations comprises:

a first transformation from a 3-dimensional program space to a 3-dimensional vehicle space;

a second transformation from the 3-dimensional vehicle space to a 2-dimensional panorama space; and

a third transformation from a 2-dimensional viewport space to a UV coordinate space.

11. The method of claim 8, wherein a plurality of images are captured by an image capture device mounted on a vehicle, and wherein the plurality of coordinate spaces comprises:

a first coordinate space in which an origin is specified by a value of geographical latitude and a value of geographical longitude;

a second coordinate space in which the 3D model is specified, relative to the origin, by a set of Cartesian coordinates;

a third coordinate space in which each of the two or more vertices is specified by a 3-dimensional set of Cartesian coordinates relative to the image capture device and vehicle direction at a time when the plurality of images were captured;

a fourth coordinate space in which each of the two or more vertices is specified by a 2-dimensional set of coordinates in a panorama space, the panorama space relative to the direction that the vehicle is traveling at the time when the plurality of images were captured;

a fifth coordinate space in which each of the two or more vertices is specified by a 2-dimensional set of coordinates in a compass space in which the panorama space is rotated such that it is relative to magnetic north instead of vehicle direction;

a sixth coordinate space in which each of the two or more vertices is specified by a 2-dimensional set of coordinates in a viewport space such that each of the two or more vertices is specified within the selected image; and

a seventh coordinate space in which the selected area of the image is mapped to the selected surface of the 3D model.

12. The method of claim 1, wherein applying the area of the selected image to the selected surface comprises:

creating a second image of a bounded area of the first image, the bounded area defined by the two or more vertices; and

UV mapping the created second image onto the selected surface of the 3D model.

13. The method of claim 1, wherein the steps of (a) selecting an image to use to photo texture the selected surface, (b) selecting an area of the image to apply to the selected surface, and (c) applying the area of the selected image to the selected surface are performed without intervention from a user upon receiving a request to automatically apply a photo texture to the selected surface.

14. A system comprising:

a network;

a database communicatively coupled to the network, the database storing a plurality of images, each image having associated with it a set of information specifying at least the location at which the image was captured;

a display device;

a processor communicatively coupled to the database via the network and to the display device; and

a memory device storing a set of machine readable instructions executable by the processor, the instructions operable to cause the processor to;

display on the display device a modeling space for modeling a 3-dimensional object;

receive a command to geolocate a point in the modeling space;

create a 3-dimensional model of the 3-dimensional object;

receive a selection of a surface of the 3D model; and

receive a request to photo texture the selected surface and, in response to the request to photo texture the selected surface, use received geolocation information to;

calculating a center point of the selected surface;

selecting a normal point at a predetermined distance from the center point along the surface normal;

project the first point to a ground plane to calculate a likely viewing point of the selected surface;

selecting from the database one or more images captured from a location near the likely viewing point;

select automatically from the database an image to use to photo texture the selected surface from among the selected one or more images captured from the location near the likely viewing point;

select an area of the image to apply to the selected surface; and

apply the area of the selected image to the selected surface.

15. The system of claim 14, wherein geolocating a point in the modeling space comprises:

receiving a selection of a geographical location; and

assigning the selected geographical location to a point within the modeling space.

16. The system of claim 14, wherein selecting an area of the image to apply to the selected surface comprises:

determining coordinates of two or more vertices of the selected surface, the coordinates in a first coordinate space;

transforming the determined coordinates for each of the two or more vertices from the first coordinate space to at least a second coordinate space, the determined coordinates in the second coordinate space corresponding to the respective positions in the selected image of the two or more vertices.

17. The system of claim 16, wherein selecting an area of the image to apply to the selected surface further comprises transforming the determined coordinates through at least a third coordinate space between the first coordinate space and the second coordinate space.

18. The system of claim 16, wherein selecting an area of the image to apply to the selected surface further comprises transforming the determined coordinates through at least each of a third and fourth coordinate space between the first coordinate space and the second coordinate space.

19. The system of claim 16, wherein selecting an area of the image to apply to the selected surface further comprises transforming the determined coordinates through at least each of a third, a fourth, and a fifth coordinate space between the first coordinate space and the second coordinate space.

20. The system of claim 14, wherein applying the area of the selected image to the selected surface comprises performing UV mapping of the selected area of the image to the selected surface of the 3D model.

21. A computer system for applying photo texturing to a 3D model of an object, the system comprising:

a display device;

a processor coupled to the display device; and

a memory device storing a set of machine readable instructions executable by the processor, the instructions operable to cause the processor to;

display on the display device a modeling space for modeling a 3-dimensional object;

geolocate a point in the modeling space;

create a 3-dimensional model of the 3-dimensional object;

receive a selection of a surface of the 3D model; and

receive a request to photo texture the selected surface and, in response to the request to photo texture the selected surface, use the geolocated point in the modeling space to;

determine a location of a center point of the selected surface;

determine a first point at a fixed distance from the center point along the normal of the selected surface;

project the first point to a ground plane, wherein the projected point on the ground plane becomes a default viewing position;

select automatically from a database of geolocated images an image to use to photo texture the selected surface wherein the automatic selection is from one or more images determined to be located near the default viewing position;

select an area of the image to apply to the selected surface; and

apply the area of the selected image to the selected surface.

22. A tangible, non-transitory computer-readable medium storing instructions, the instructions when executed by a processor, operable to cause the processor to:

create a 3-dimensional (3D) digital model of a real-world object;

geolocate the digital space in which the 3D model is created;

receive a selection of a surface of the 3D model; and

receive a request to photo texture the selected surface and, in response to the request;

determine coordinates of two or more points on the selected surface, the two or more points including at least a center point of the selected surface;

determining a first point at a fixed distance from the center point along the normal of the selected surface;

projecting the first point to a ground plane to determine a likely viewing point of the selected surface;

determine automatically from a plurality of stored, geolocated photographic images an image that depicts the two or more points on the selected surface wherein the automatic determination is based upon selecting geolocated photographic images near the position of the likely viewing point;

transform the coordinates of the two or more points from a first coordinate system to a second coordinate system, the second coordinate system specifying points on the image;

select an area of the image according to at least the transformed coordinates; and

texture map the selected area of the image to the selected surface.

23. The computer-readable medium of claim 22, wherein the plurality of stored photographic images are panoramic images, and wherein determining an image that depicts the two or more points on the selected surface comprises determining one or more point of view parameters for a panoramic image.

24. The computer-readable medium of claim 22, wherein determining the coordinates of two or more points on the selected surface comprises determining the coordinates of two or more vertices of the selected surface.

25. The computer-readable medium of claim 22, wherein the first coordinate system corresponds to a 3D space within a modeling program.

26. The computer-readable medium of claim 22, wherein the first coordinate system specifies within a modeling program the location of the two or more points on the selected surface, and further comprising:

transforming the coordinates of the two or more points from the second coordinate system to a third coordinate system.