Augmentation of a dynamic terrain surface

• US 9,824,490 B1
• Filed: 06/08/2015
• Issued: 11/21/2017
• Est. Priority Date: 06/08/2015
• Status: Active Grant

• Current Assignee: Bentley Systems Incorporated
• Original Assignee: Bentley Systems Incorporated
• Inventors: Ct, Stphane, Ltourneau, Ian, Marcoux-Ouellet, Jade
• Primary Examiner(s): Tryder, Gregory J
• Assistant Examiner(s): Wills, Diane
• Application Number:
  US14/733,660
• Time in Patent Office: 897 Days
• CPC Class Codes: E02F 9/245 for preventing damage to underground objects during excavation, e.g. indicating buried pipes or the like detection of pipes in the ground F16L1/11
• CPC Class Codes: E02F 9/261 Surveying the work-site to be treated
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

In one embodiment, an augmented view is generated that accounts for dynamically changing terrain surface at a site. A sensor captures live georeferenced terrain surface topography for the site. A camera captures an image of the site. Further, a tracking system determines a georeferenced camera pose of the camera. An augmented reality application aligns a georeferenced three-dimensional (3-D) model for the site with the live georeferenced terrain surface topography. Then, using at least the captured image, the georeferenced camera pose, the georeferenced 3-D model and live georeferenced terrain surface topography, the augmented reality application creates an augmented view of the site that shows graphical representations of subsurface features. At least a portion of the graphical representations are dynamically conformed to the contours of the terrain surface in the image based on the live georeferenced terrain surface topography. The graphical representations may include virtual excavation and/or virtual paint markings.

Claims

1. A method for providing augmented reality for a site having a dynamically changing terrain surface comprising:

capturing a live georeferenced terrain surface topography for the site using a sensor;

capturing an image of the site using a camera;

determining a georeferenced camera pose of the camera using a tracking system;

aligning a georeferenced three-dimensional (3-D) model for the site with the live georeferenced terrain surface topography; and

using at least the captured image, the georeferenced camera pose, the georeferenced 3-D model and the live georeferenced terrain surface topography, to create an augmented view of the site that shows graphical representations of subsurface features, wherein a least a portion of the graphical representations are dynamically conformed to the contours of the terrain surface based on the live georeferenced terrain surface topography.

2. The method of claim 1, wherein the graphical representation comprises a virtual excavation that surrounds a textured 3-D element representing a subsurface feature imposed upon the captured image, wherein the virtual excavation includes one or more boundary surfaces that are conformed to contours of the terrain surface based on the live georeferenced terrain surface topography.

3. The method of claim 1, wherein the graphical representation comprises virtual paint markings including a texture imposed at locations on the terrain surface in the captured image, wherein the locations are dynamically conformed to contours of the terrain surface based on the live georeferenced terrain surface topography.

4. The method of claim 1, wherein the graphical representations comprise both a virtual excavation and virtual paint markings.

5. The method of claim 1, wherein the augmented view is displayed as part of a hybrid display that includes at least one vertical section corresponding to a slicing plane.

6. The method of claim 1, wherein the site is a site of an ongoing excavation, and the capturing the live georeferenced terrain surface topography captures a change to the terrain surface that results from the ongoing excavation.

7. The method of claim 1, wherein the displaying further comprises:

showing the augmented view within the cab of a piece of heavy construction equipment.

8. The method of claim 7, wherein the showing further comprises:

projecting the augmented view onto a windshield of the piece of heavy construction equipment to provide a heads up display (HUD) to an operator.

9. The method of claim 7, wherein the piece of heavy construction equipment is an excavator.

10. The method of claim 1, wherein the live georeferenced terrain surface topography is a point cloud, and the method further comprises:

converting the point cloud to a polygon mesh,

wherein the aligning aligns the georeferenced 3-D model with the polygon mesh, and the using conforms at least a portion of the graphical representations to contours of the terrain surface based on the polygon mesh.

11. The method of claim 1, wherein the sensor includes a depth-sensing camera or laser scanning system, and the tracking system includes at least one global positioning system (GPS) receiver and orientation sensor, or at least one robotic total station theodolite (TST) station.

12. An apparatus comprising:

a sensor configured to capture a live terrain surface topography for a site;

a camera configured to capture an image of the site, the camera having a pose; and

a computing device configured to execute an augmented reality application that is operable to;

align a three-dimensional (3-D) model for the site with the live terrain surface topography, and

use at least the captured image, the camera pose, the 3-D model and the live terrain surface topography, to create an augmented view of the site that shows graphical representations of subsurface features, wherein at least a portion of the graphical representations are dynamically conformed to the contours of the terrain surface based on the live terrain surface topography.

13. The apparatus of claim 12, further comprising

a tracking system configured to determine the pose of the camera.

14. The apparatus of claim 12, wherein the graphical representations comprise a virtual excavation that surrounds a textured 3-D element representing a subsurface feature imposed upon the captured image, wherein the virtual excavation includes one or more boundary surfaces that are conformed to contours of the terrain surface based on the live terrain surface topography.

15. The apparatus of claim 12, wherein the graphical representations comprise virtual paint markings including a texture imposed at locations on the terrain surface in the captured image, wherein the locations are dynamically conformed to contours of the terrain surface based on the live terrain surface topography.

16. The apparatus of claim 12, wherein the graphical representations comprise both a virtual excavation and virtual paint markings.

17. A non-transitory electronic device-readable medium that includes instructions executable on the electronic-device that, when executed, are operable to:

receive a captured image of a site and a terrain surface topography for the site;

align a three-dimensional (3-D) model for the site with the terrain surface topography;

create an augmented view of the site that shows graphical representations of subsurface features, wherein a least a portion of the graphical representations are dynamically conformed to the contours of the terrain surface based on the terrain surface topography, wherein the graphical representations include at least one of

a virtual excavation that surrounds a textured 3-D element representing a subsurface feature, wherein the virtual excavation includes one or more boundary surfaces that are conformed to contours of the terrain surface based on the terrain surface topography, or

virtual paint markings including a texture imposed at locations on the terrain surface in the captured image, wherein the locations are dynamically conformed to contours of the terrain surface based on the terrain surface topography.

18. The non-transitory electronic device-readable medium of claim 17, wherein the graphical representations comprise both the virtual excavation and the virtual paint markings.

19. The non-transitory electronic device-readable medium of claim 17, wherein the site is a site of an ongoing excavation, and the capturing the georeferenced terrain surface topography captures changes to the terrain surface that result from the ongoing excavation.

20. The non-transitory electronic device-readable medium of claim 17, wherein the instructions when executed are further operable to

display the augmented view as a heads up display (HUD) onto a windshield of a piece of heavy construction equipment.