Apparatus and method of generating electronic three-dimensional walkthrough environment

• US 10,616,483 B1
• Filed: 02/27/2019
• Issued: 04/07/2020
• Est. Priority Date: 02/27/2019
• Status: Active Grant

• Current Assignee: Hong Kong Applied Science And Technology Research Institute Co., Ltd.
• Original Assignee: Hong Kong Applied Science And Technology Research Institute Co., Ltd.
• Inventors: Ngai, Yat Cheung, Chan, Yin Yee
• Primary Examiner(s): Rahaman, Mohammed S
• Application Number:
  US16/288,075
• Time in Patent Office: 405 Days
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30168 Image quality inspection
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 3/00 Geometric image transformations in the plane of the image
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 20/00 Scenes; Scene-specific elements control of digital cameras H04N23/60
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

A method of generating an electronic 3D walkthrough environment of a targeted environment, comprising: capturing a panorama at a first location point in the targeted environment; measuring an initial orientation of the camera whenever a panorama is captured; moving through the targeted environment and at each location point, capturing a live image of surrounding, and measuring an orientation of the camera; determining whether a new panorama is required to be captured at the current location point at where the camera is with the current orientation of the camera, comprising: identifying a sub-image in the panorama last captured that matches the live image captured; and if the sub-image cannot be identified, then a new panorama is required; and generating the electronic 3D walkthrough environment by connecting the panoramas captured.

Claims

1. A method of generating an electronic three-dimensional (3D) walkthrough environment of a targeted environment, comprising:

at a first location point in the targeted environment;

capturing, by an optical sensor, a first panorama;

measuring and recording, by a positional tracking system, an initial orientation of the optical sensor; and

tracking and recording, by the positional tracking system, a first coordinate of the optical sensor;

moving the optical sensor through the targeted environment and at each location point in the targeted environment;

capturing, by the optical sensor, a live image of surrounding;

measuring, by the positional tracking system, a relative orientation of the optical sensor, wherein the relative orientation is measured in relative to the initial orientation;

tracking and recording, by the positional tracking system, a coordinate of the optical sensor;

determining whether a new panorama is required to be captured, comprising;

identifying, by a first processor, a sub-image in the panorama last captured that matches the live image captured; and

if the sub-image cannot be identified, then a new panorama is required to be captured at the current location point with the current relative orientation of the optical sensor; and

if it is determined a new panorama is required to be captured at the current location point, then;

capturing, by the optical sensor, a new panorama at the current location point; and

measuring and recording, by the positional tracking system, a new initial orientation of the optical sensor.

2. The method of claim 1, further comprising:

generating, by a second processor, the electronic 3D walkthrough environment by connecting the panoramas captured to create seamless scenes of the electronic 3D walkthrough environment, and by creating a two-dimensional map of the electronic 3D walkthrough environment from the coordinates of the optical sensor recorded at the location points.

3. The method of claim 1, wherein the identification of a sub-image in the panorama last captured that matches the live image captured comprises:

estimating a scene depth from the live image captured;

determining a zoom factor from a displacement vector and the relative orientation of the optical sensor at the current location point where the live image is captured, and the scene depth; and

identifying the sub-image using the zoom factor and one or more configurable setting parameters.

4. The method of claim 3, wherein the estimation of a scene depth comprises:

defining a point of interest (POI) of the live image captured;

identifying one or more feature points in the live image captured;

determining a feature point depth of each of one or more feature points;

determining a distance between the POI and each of one or more feature points;

estimating the scene depth by obtaining computation result of;

5. The method of claim 3, wherein the determination of a zoom factor comprises:

determining a displacement vector of movement of the optical sensor from the recorded coordinate of the optical sensor at the location point where the last panorama is captured and the recorded coordinate of the optical sensor at the current location point where the live image is captured;

determining a depth vector using the scene depth estimated from the live image and the relative orientation by obtaining computation result of;

D cos θ

î

+D sin θ

ĵ

;

wherein;

D is the scene depth estimated from the live image; and

θ

is relative orientation;

î

is the unit vector in the x direction; and

ĵ

is the unit vector in the y direction; and

determining the zoom factor by obtaining computation result of;

6. The method of claim 3, wherein the identification of the sub-image using the zoom factor and one or more configurable setting parameters comprises:

determining a zoom factor threshold based on one or more of the configurable setting parameters, the configurable setting parameters comprise;

one or more intrinsic characteristics of the optical sensor,

image size of the panoramas captured,

the electronic 3D walkthrough environment'"'"'s image display window size, and

the electronic 3D walkthrough environment'"'"'s maximum-zoom-in resolution,

comparing the zoom factor to the zoom factor threshold, and if the zoom factor does not meet the zoom factor threshold, the sub-image is unidentifiable in the panorama last captured that matches the live image captured.

7. The method of claim 1, wherein the identification of a sub-image in the panorama last captured that matches the live image captured comprises:

defining a point of interest (POI) of the live image captured;

obtaining an absolute angular difference between a vertical central line of horizontal view angle of the POI in the last panorama captured and the optical sensor'"'"'s direct view of the POI at the current location point; and

if the absolute angular difference is larger than one half of an angular horizontal field of view of the live image captured, then the sub-image is unidentifiable from the last panorama captured.

8. The method of claim 1, wherein the determination of whether a new panorama is required to be captured at the current location point where the optical sensor is with the current relative orientation of the optical sensor further comprises:

determining a matching quality of the sub-image identified; and

if the sub-image identified does not meet a quality threshold in matching the live image captured, then a new panorama is required to be captured at the current location point with the current relative orientation of the optical sensor.

9. The method of claim 1, wherein the optical sensor is a 360 camera.

10. An apparatus for generating an electronic three-dimensional (3D) walkthrough environment of a targeted environment, comprising:

one or more optical sensors configured;

to capture a first panorama at a first location point in the targeted environment;

to capture panoramas at selected location points in the targeted environment; and

to capture a live image of surrounding at each location point as the optical sensors are moved through the targeted environment;

a positional tracking system configured;

to measure an initial orientation of the optical sensors at each location point where a panorama is captured;

to measure a relative orientation of the optical sensors at each location point where the live image is captured, wherein the relative orientation is measured in relative to the initial orientation; and

to track a coordinate of the optical sensors at each location point; and

a first processor configured;

to determine whether a new panorama is required to be captured at the current location point where the optical sensors are with the current relative orientation of the optical sensors, comprising;

identifying a sub-image in the panorama last captured that matches the live image captured; and

if the sub-image cannot be identified, then a new panorama is required to be captured at the current location point with the current relative orientation of the optical sensors;

to command the optical sensors to capture a new panorama if it is determined a new panorama is required to be captured at the current location point;

to record the initial orientation of the optical sensors at each location point where a panorama is captured; and

to record the coordinate of the optical sensors at each location point.

11. The apparatus of claim 10, further comprising a second processor configured to generate the electronic 3D walkthrough environment by connecting the panoramas captured to create seamless scenes of the electronic 3D walkthrough environment, and by creating a two-dimensional map of the electronic 3D walkthrough environment from the coordinates of the optical sensors recorded at the location points.

12. The apparatus of claim 10, wherein the identification of a sub-image in the panorama last captured that matches the live image captured comprises:

estimating a scene depth from the live image captured;

determining a zoom factor from a displacement vector and the relative orientation of the optical sensors at the current location point where the live image is captured, and the scene depth; and

identifying the sub-image using the zoom factor and one or more configurable setting parameters.

13. The apparatus of claim 12, wherein the estimation of a scene depth comprises:

defining a point of interest (POI) of the live image captured;

identifying one or more feature points in the live image captured;

determining a feature point depth of each of one or more feature points;

determining a distance between the POI and each of one or more feature points;

estimating the scene depth by obtaining computation result of;

14. The apparatus of claim 12, wherein the determination of a zoom factor comprises:

determining a displacement vector of movement of the optical sensors from the recorded coordinate of the optical sensors at the location point where the last panorama is captured and the coordinate of the optical sensors at the current location point at where the live image is captured;

determining a depth vector using the scene depth estimated from the live image and the relative orientation by obtaining computation result of;

D cos θ

î

+D sin θ

ĵ

;

wherein;

D is the scene depth estimated from the live image; and

θ

is relative orientation;

î

is the unit vector in the x direction; and

ĵ

is the unit vector in the y direction; and

determining the zoom factor by obtaining computation result of;

15. The apparatus of claim 12, wherein the identification of the sub-image using the zoom factor and one or more configurable setting parameters comprises:

determining a zoom factor threshold based on one or more of the configurable setting parameters, the configurable setting parameters comprise;

one or more intrinsic characteristics of the optical sensors,

image size of the panoramas captured,

the electronic 3D walkthrough environment'"'"'s image display window size, and

the electronic 3D walkthrough environment'"'"'s maximum-zoom-in resolution,

comparing the zoom factor to the zoom factor threshold, and if the zoom factor does not meet the zoom factor threshold, the sub-image is unidentifiable in the panorama last captured that matches the live image captured.

16. The apparatus of claim 10, wherein the identification of a sub-image in the panorama last captured that matches the live image captured comprises:

defining a point of interest (POI) of the live image captured;

obtaining an absolute angular difference between a vertical central line of horizontal view angle of the POI in the last panorama captured and the optical sensors'"'"' direct view of the POI at the current location point; and

if the absolute angular difference is larger than one half of an angular horizontal field of view of the live image captured, then the sub-image is unidentifiable from the last panorama captured.

17. The apparatus of claim 10, wherein the determination of whether a new panorama is required to be captured at the current location point where the optical sensors are with the current relative orientation of the optical sensors further comprise:

determining a matching quality of the sub-image identified; and

if the sub-image identified does not meet a quality threshold in matching the live image captured, then a new panorama is required to be captured at the current location point with the current relative orientation of the optical sensor.

18. The apparatus of claim 10, wherein the one or more optical sensors are configured to form a 360 camera.