Hybrid 2D/3D data in a virtual environment

• US 10,438,413 B2
• Filed: 11/07/2017
• Issued: 10/08/2019
• Est. Priority Date: 11/07/2017
• Status: Active Grant

• Current Assignee: the united states of america as represented by the secretary of the navy
• Original Assignee: the united states of america as represented by the secretary of the navy
• Inventors: Bilinski, Mark, Greunke, Larry Clay
• Primary Examiner(s): Harrison, Chante E
• Application Number:
  US15/806,125
• Publication Number:
  US 20190139306A1
• Time in Patent Office: 700 Days
• CPC Class Codes: G01S 17/00 Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2210/04 Architectural design, interior design
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays

Abstract

A method for using a virtual reality (VR) headset to view a two-dimensional (2D) technical drawing of a physical object of a real-world, real-world environment in three dimensions (3D), the method comprising: using LiDAR to produce a 3D point cloud of the real-world environment; scaling and aligning the 2D technical drawing to match the size and orientation of the physical object as depicted in the 3D point cloud; overlaying the 2D technical drawing (including all labels and dimensions) over the physical object as depicted in the 3D point cloud; and visually comparing the 3D point cloud representation of the physical object to the 2D technical drawing by simultaneously displaying the 3D point cloud of the real-world environment and the overlaid 2D technical drawing to a user with the VR headset.

Claims

1. A method for using a virtual reality (VR) headset to view a two-dimensional (2D) technical drawing of a real-world, physical object in three dimensions (3D), the method comprising:

using LiDAR to produce a 3D point cloud of a real-world environment that comprises the physical object by performing the following substeps;

scanning the real-world environment with a laser scanner to produce scan data;

registering the scan data with a computer to stitch data from multiple scans together to create a data-intensive point cloud;

degrading the data-intensive point cloud by reducing the number of data points in the data-intensive point cloud to create a degraded point cloud by using an octree filter to subdivide the data-intensive point cloud and preserving only one point from each nonempty subdivision;

creating the 3D point cloud by formatting the degraded point cloud into a game-engine-compatible format; and

importing the 3D point cloud into a software framework used for creating and developing video games;

scaling and aligning the 2D technical drawing to match the size and orientation of the physical object as depicted in the 3D point cloud;

overlaying the 2D technical drawing (including all labels and dimensions) on the physical object depicted in the 3D point cloud; and

visually comparing the 3D point cloud representation of the physical object to the 2D technical drawing by simultaneously displaying the 3D point cloud of the real-world environment and the overlaid 2D technical drawing to a user with the VR headset.

2. The method of claim 1, wherein the game-engine-compatible format is ASCII.

3. The method of claim 1, wherein the game-engine-compatible format is polygonal file format (PLY).

4. The method of claim 1, wherein the degraded point cloud comprises fewer than two million data points.

5. The method of claim 1, further comprising removing a background from the 2D technical drawing leaving only lines, labels, and dimensions such that when the 3D point cloud of the real-world environment and the overlaid 2D technical drawing are simultaneously displayed to the user with the VR headset the user only sees the lines, labels, and dimensions of the 2D technical drawing overlaid on the physical object as depicted in the 3D point cloud.

6. The method of claim 1, wherein when the 3D point cloud of the real-world environment and the overlaid 2D technical drawing are simultaneously displayed to the user with the VR headset the physical object as depicted in the 3D point cloud is obscured from the view of the user by the overlaid 2D technical drawing.

7. The method of claim 1, wherein the physical object is a floor and the 2D technical drawing is a floor plan.

8. The method of claim 1, wherein the physical object is a wall.

9. The method of claim 1, wherein the 2D technical drawing pre-exists the production of the 3D point cloud.

10. A method for using a virtual reality (VR) headset to view a two-dimensional (2D) technical drawing of a real-world physical object in three dimensions (3D), the method comprising:

scanning a real-world environment that comprises the physical object to produce scan data;

creating with a computer a 3D point cloud of the real-world environment based on the scan data comprising the substeps of;

registering the scan data with the computer such that scan data from multiple scans are stitched together to create a data-intensive point cloud;

degrading the data-intensive point cloud by reducing the number of data points in the data-intensive point cloud to create a degraded point cloud by using an octree filter to subdivide the data-intensive point cloud and preserving only one point from each nonempty subdivision;

creating the 3D point cloud by formatting the degraded point cloud into a game-engine-compatible format; and

importing the 3D point cloud into a software framework used for creating and developing video games;

overlaying the 2D technical drawing (including all lines, labels, and dimensions) on the physical object as depicted in the 3D point cloud; and

visually comparing the 3D point cloud representation of the physical object to the 2D technical drawing by simultaneously displaying the 3D point cloud of the real-world environment and the overlaid 2D technical drawing to a user with the VR headset.

11. The method of claim 10, wherein prior to the overlaying step, the 2D technical drawing is scaled and aligned to match the size and orientation of the physical object as depicted in the 3D point cloud.

12. The method of claim 11, wherein the scanning step is performed with a LiDAR laser scanner.

13. The method of claim 10, wherein a background of the overlaid 2D technical drawing is sufficiently transparent such that when the 3D point cloud of the real-world environment and the overlaid 2D technical drawing are simultaneously displayed to the user with the VR headset the lines, labels, and dimensions of the overlaid 2D technical drawing as well as the physical object as depicted in the 3D point cloud are visible to the user in the VR headset.