Generation of 3D models of an environment

• US 10,109,104 B2
• Filed: 02/21/2014
• Issued: 10/23/2018
• Est. Priority Date: 02/21/2013
• Status: Active Grant

• Current Assignee: Oxford University Innovation Limited (University of Oxford)
• Original Assignee: Oxford University Innovation Limited (University of Oxford)
• Inventors: Newman, Paul Michael, Baldwin, Ian
• Primary Examiner(s): Abraham, Samantha K
• Application Number:
  US14/769,201
• Publication Number:
  US 20150379766A1
• Time in Patent Office: 1,705 Days
• CPC Class Codes: G01S 17/87 Combinations of systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/875 for determining attitude
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/262 using transform domain methods, e.g. Fourier domain methods
• CPC Class Codes: G06T 7/285 using a sequence of stereo image pairs
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 23/54 Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils

Abstract

Generating a 3D reconstruction of an environment around a monitoring-unit as that monitoring-unit is moved through the environment: a) providing at least a camera and a LIDAR sensor, each being controlled by independent clocks; b) using the camera to determine the trajectory of the monitoring-unit and determining a first time series using the clock of the camera, where the first time series details when the monitoring-unit was at predetermined points of the trajectory; c) recording the returns from the LIDAR sensor and determining a second time series using the clock of the LIDAR sensor, where the second time series details when each scan from the LIDAR was taken; d) using a timer to relate the first and second series in order to match the return from the LIDAR sensor to the point on the trajectory at which the return was received; and e) creating the 3D reconstruction based upon the LIDAR returns using information from the two time series.

Claims

1. A method of generating a 3D reconstruction of an environment around a monitoring-unit as the monitoring-unit is moved through the environment, the method comprising:

providing at least a camera and a LIDAR sensor, wherein the camera and the LIDAR sensor are provided at a fixed, predetermined, orientation relative to one another and are controlled by independent clocks;

using the camera to determine a trajectory of the monitoring-unit as the monitoring-unit moves through the environment;

determining a first time series using the clock of the camera, where the first time series details when the monitoring-unit was at predetermined points of the trajectory;

scanning the environment with the LIDAR sensor;

recording returns from the LIDAR sensor;

determining a second time series using the clock of the LIDAR sensor, where the second time series details when each scan from the LIDAR sensor was taken;

processing the first and second time series;

using a timer to relate the first and second time series in order to match the returns from the LIDAR sensor to a point on the trajectory at which a return was received; and

creating, using the fixed, predetermined, orientation between the camera and the LIDAR sensor, the 3D reconstruction based upon the returns.

2. A method according to claim 1 which the timer is provided by a statistical method to relate the first time series and the second time series.

3. A method according to claim 2 which the statistical method uses the TICsync algorithm to provide the timer.

4. A method according to claim 1 which uses a stereoscopic pair of cameras to generate a stream of stereoscopic images of the environment as the monitoring-unit is moved through the environment.

5. A method according to claim 4 which processes the stream of stereoscopic images to generate the trajectory.

6. A method according to claim 1 which determines the fixed, predetermined, orientation of the camera and the LIDAR sensor by processing data received from the camera and the LIDAR sensor.

7. A non-transitory machine readable medium containing instructions which when read by a machine cause that machine to perform the method of claim 1.

8. A monitoring unit comprising:

processing circuitry arranged to receive data from each of a camera and a LIDAR sensor;

wherein the camera is controlled by a first clock, which provides a first time series, and the camera is mounted in a fixed orientation relative to the monitoring unit and wherein the processing circuitry is arranged to generate a trajectory from the data received from the camera;

wherein the LIDAR sensor is controlled by a second clock, which provides a second time series, and the LIDAR sensor is mounted in a fixed orientation relative to the monitoring unit and wherein the LIDAR sensor is arranged to scan the environment of the monitoring unit as the monitoring unit is moved through the environment; and

wherein the processing circuitry is arranged to;

receive the data from the LIDAR sensor in addition to the data from the camera,

process the first and second time series,

use a timer to relate the first and second time series together,

match the data from the LIDAR sensor to the trajectory, and

generate a 3D reconstruction of the environment from the matched data.

9. A unit according to claim 8 comprising at least one two-dimensional LIDAR sensor.

10. A unit according to claim 9 in which the LIDAR sensor is mounted in a push-broom arrangement.

11. A unit according to claim 8 comprising two LIDAR sensors.

12. A unit according to claim 11 wherein the two LIDAR sensors comprise the LIDAR sensor and a two-dimensional LIDAR sensor.

13. A unit according to claim 8 in which the camera and the LIDAR sensor are arranged such that they do not have overlapping fields of view.

14. A unit according to claim 8 in which the monitoring unit is arranged within a single unit capable of being carried by a user.

15. A unit according to claim 14 in which the monitoring unit is arranged as a back-pack capable of being worn by a user.

16. A unit according to claim 8 which comprises a stereoscopic pair of cameras.

17. A unit according to claim 8 comprising a non-transitory machine readable medium containing instructions which when read by a machine causes the processing circuitry to perform at least one step.

18. A unit according to claim 17 wherein the medium comprises a physical medium.

19. A unit according to claim 17 wherein the medium comprises a transmitted signal.