SURVEYING SYSTEM

• US 20170067739A1
• Filed: 05/05/2014
• Published: 03/09/2017
• Est. Priority Date: 05/05/2014
• Status: Active Grant

• Current Assignee: Hexagon Technology Center GmbH (Hexagon AB)
• Original Assignee: Hexagon Technology Center GmbH (Hexagon AB)
• Inventors: SIERCKS, Knut, METZLER, Bernhard, VAN DER ZWAN, Elmar Vincent, FIDLER, Thomas, PARYS, Roman, VELIZHEV, Alexander, SCHEJA, Jochen
• Granted Patent: US 10,234,287 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G01C 11/02 Picture taking arrangements specially adapted for photogrammetry or photographic surveying, e.g. controlling overlapping of pictures
• CPC Class Codes: G01C 11/04 Interpretation of pictures
• CPC Class Codes: G01C 15/002 Active optical surveying means optical plumbing G01C15/105
• CPC Class Codes: G01C 15/006 Detectors therefor
• CPC Class Codes: G01C 25/005 initial alignment, calibration or starting-up of inertial devices
• CPC Class Codes: G01S 19/13 Receivers
• CPC Class Codes: G01S 19/14 specially adapted for specific applications
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection
• CPC Class Codes: G06T 7/0004 Industrial image inspection
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: H04N 23/51 Housings
• CPC Class Codes: H04N 23/55 Optical parts specially adapted for electronic image sensors; Mounting thereof
• CPC Class Codes: H04N 23/57 Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/635 Region indicators; Field of view indicators
• CPC Class Codes: H04N 23/69 Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04N 23/698 : for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

A system is disclosed that comprises a camera module and a control and evaluation unit. The camera module is designed to be attached to the surveying pole and comprises at least one camera for capturing images. The control and evaluation unit has stored a program with program code so as to control and execute a functionality in which a series of images of the surrounding is captured with the at least one camera; a SLAM-evaluation with a defined algorithm using the series of images is performed, wherein a reference point field is built up and poses for the captured images are determined; and, based on the determined poses, a point cloud comprising 3D-positions of points of the surrounding can be computed by forward intersection using the series of images, particularly by using dense matching algorithm.

Claims

1-136. -136. (canceled)

137. Surveying subsystem comprising a camera module and a control and evaluation unit,

the camera module comprising at least one camera for capturing images,

an inertial measuring unit being associated with the surveying subsystem in a fixed spatial relationship relative to the surveying subsystem,

the control and evaluation unit having stored a program with program code so as to control and execute a calibration functionality in which—

when moving along a path through a surrounding

inertial measuring data is gathered while moving along the path,

camera-based localisation data is generated, wherefore

a series of images of the surrounding is captured with the at least one camera, the series comprising an amount of images captured with different poses of the camera, the poses representing respective positions and orientations of the camera,

a SLAM-evaluation with a defined algorithm using the series of images is performed, wherein a plurality of respectively corresponding image points are identified in each of several sub-groups of images of the series of images and, based on resection and forward intersection using the plurality of respectively corresponding image points,

a reference point field is built up comprising a plurality of reference points of the surrounding, wherein coordinates of the reference points are derived, and the poses for the images are determined, and

calibration parameters for the inertial measuring unit are derived based on the gathered inertial measuring data and the camera-based localisation data.

138. The subsystem according to claim 137, wherein

the control and evaluation unit is configured so that the calibration functionality is controlled and executed in such a way, that the calibration parameters are derived using a Kalman filter.

139. The subsystem according to claim 137, wherein

the control and evaluation unit is configured so that the calibration functionality is controlled and executed in such a way, that the inertial measuring unit is calibrated using the derived calibration parameters, wherein a systematic error of the inertial measuring unit is compensated, particularly wherein a bias of the inertial measuring unit is compensated.

140. The subsystem according to claim 137, wherein

the camera module is designed to be attached to a hand-carried surveying pole such that said surveying subsystem is adapted to be used as part of a surveying system that is adapted to determine positions of a position measuring resource also being mounted on the surveying pole.

141. The subsystem according to claim 140, wherein the position measuring resource comprises a GNSS-antenna or a retro-reflector.

142. The subsystem according to claim 140, wherein

the control and evaluation unit is configured so that the calibration functionality is controlled and executed in such a way, that the generation of camera-based localisation data further includes

retrieving—

from the surveying system—

determined positions that are adopted by the position measuring resource when moving along the path; and

deriving external orientations of the at least one camera and/or the surveying pole in the coordinate system of the surveying system, in particular in all three rotational degrees of freedom, more particularly positions and external orientations in six degrees of freedom, based

on the determined poses for the images of the series of images and

on a multitude of determined positions of the position measuring resource, in particular a travelling history for the moved path.

143. The subsystem according to claim 137, wherein

the control and evaluation unit having stored a program with program code so as to control and execute a spatial representation generation functionality in which—

when moving along a path through a surrounding

a series of images of the surrounding is captured with the at least one camera, the series comprising an amount of images captured with different poses of the camera, the poses representing respective positions and orientations of the camera,

inertial measuring data—

considering the derived calibration parameters—

is gathered while moving along the path,

a SLAM-evaluation with a defined algorithm using the series of images as well as the gathered inertial measuring data is performed, wherein a plurality of respectively corresponding image points are identified in each of several sub-groups of images of the series of images and, based on resection and forward intersection using the plurality of respectively corresponding image points,

a reference point field is built up comprising a plurality of reference points of the surrounding, wherein coordinates of the reference points are derived, and

the poses for the images are determined, and

based on the determined poses, a point cloud comprising 3D-positions of points of the surrounding is computed by forward intersection using the images of the series of images.

144. The subsystem according to claim 143, wherein

the control and evaluation unit is configured so that the spatial representation generation functionality is controlled and executed in such a way, that

determined positions of the position measuring resource for points that have been adopted on the path are received by the control and evaluation unit from the surveying system and

the point cloud is scaled, and particularly geo-referenced, with help of the received determined positions.

145. The subsystem according to claim 144, wherein

the control and evaluation unit is configured so that the spatial representation generation functionality is controlled and executed in such a way, that a graphical reproduction is generated for the scaled point cloud, the graphical reproduction being displayable by display means of the surveying system.

146. The subsystem according to claim 137, wherein the camera module comprises at least four cameras arranged relative to each other so that panoramic images with a field of view of 360°

in azimuthal direction are capturable.

147. The subsystem according to claim 146, wherein the at least four cameras are fixedly arranged in a common housing of the camera module.

148. The subsystem according to claim 146, wherein the inertial measuring unit is fixedly integrated into the common housing of the camera module.

149. A surveying system comprising

a hand-carried surveying pole,

a position measuring resource being mounted on the surveying pole, wherein positions of the position measuring resource are determinable by the surveying system, particularly wherein the position measuring resource comprises a GNSS-antenna or a retro-reflector,

a surveying subsystem according claim 137, and

an inertial measuring unit being associated with the surveying subsystem or the position measuring resource in a fixed spatial relationship relative to the surveying subsystem.

150. A surveying subsystem comprising a camera module and a control and evaluation unit, wherein the surveying subsystem is configured to be used as part of a surveying system adapted to determine positions of a position measuring resource, particularly wherein the position measuring resource is a GNSS-antenna or a retro-reflector,

the camera module comprising at least one camera for capturing images and particularly being designed to be attached to the position measuring resource,

the control and evaluation unit having stored a program with program code so as to control and execute a functionality in which—

when moving along a path through a surrounding

a series of images of the surrounding is captured with the at least one camera, the series comprising an amount of images captured with different poses of the camera, the poses representing respective positions and orientations of the camera;

a SLAM-evaluation with a defined algorithm using the series of images is performed, wherein a plurality of respectively corresponding image points are identified in each of several sub-groups of images of the series of images with use of a feature detection process for detecting corresponding features, particularly by means of feature tracking, especially a KLT-algorithm (Kanade-Lucas-Tomasi), and/or by means of feature matching, especially a SIFT-, SURF-, BRISK- or BRIEF-algorithm, and, based on resection and forward intersection using the plurality of respectively corresponding image points,

a reference point field is built up comprising a plurality of reference points of the surrounding, wherein coordinates of the reference points are derived, and

the poses for the images are determined,

wherein the control and evaluation unit comprises several diversified units,

wherein at least a first unit, particularly an FPGA or a GPU (i.e. as abbreviation for graphical processing unit), carries out at least part of the feature detection process for identifying the plurality of respectively corresponding image points, and

wherein at least another, second unit, particularly a CPU, carries out determining of the poses.

151. The subsystem according to claim 150, wherein the first and second units are built together as one SoC (i.e. as abbreviation for system on a chip).

152. The subsystem according to claim 150, wherein the first and the second unit are disposed within the camera module.

153. The subsystem according to claim 150, wherein the first unit is disposed within the camera module and the second unit is a laptop, a tablet-PC, a smartphone, a processing unit of a handheld controller/datalogger of the surveying system, a processing unit of a GNSS-module of the surveying system, a processing unit of a surveying station, particularly total station, of the surveying system or a surveying site server, particularly being installed in a car.

154. The subsystem according to claim 150, wherein

the camera module is designed to be attached to a hand-carried surveying pole as part of a surveying system that is adapted to determine positions of a position measuring resource, and the position measuring resource also being mountable on the surveying pole.

157. The subsystem according to claim 150, wherein

the at least one other unit is a laptop, a tablet-PC, a smartphone, a processing unit of a handheld controller/datalogger of the surveying system, a processing unit of a GNSS-module of the surveying system or a surveying site server, particularly being installed in a car.

158. The subsystem according to claim 150, wherein

several other units, being disposed externally to the camera module, carry out at least parts of computing of the spatial representation in a decentralised manner, the several other units particularly being several smartphones.

159. The subsystem according to claim 150, wherein

at least one unit carrying out a part of identifying the plurality of respectively corresponding image points and/or determining the poses is designed so as to be carriable in a backpack of an operator.

160. The subsystem according to claim 150, wherein

at least one unit carrying out a part of identifying the plurality of respectively corresponding image points and/or determining the poses is a laptop, a tablet-PC, a smartphone, a processing unit of a handheld controller/datalogger of the surveying system, a processing unit of a GNSS-module of the surveying system or a surveying site server, particularly being installed in a car.

161. The subsystem according to claim 150, wherein

the camera module is designed to be attached to a hand-carried surveying pole as part of a surveying system that is adapted to determine positions of a position measuring resource, and the position measuring resource also being mountable on the surveying pole.

155. A surveying subsystem comprising a camera module and a control and evaluation unit, wherein the surveying subsystem is configured to be used as part of a surveying system adapted to determine positions of a position measuring resource, particularly wherein the position measuring resource is a GNSS-antenna or a retro-reflector,

the camera module comprising at least one camera for capturing images, wherein the camera module is built as one single integrated physical unit being designed to be attached to the position measuring resource, and

the control and evaluation unit having stored a program with program code so as to control and execute a functionality in which—

when moving along a path through a surrounding

a series of images of the surrounding is captured with the at least one camera, the series comprising an amount of images captured with different poses of the camera, the poses representing respective positions and orientations of the camera,

a SLAM-evaluation with a defined algorithm using the series of images is performed, wherein a plurality of respectively corresponding image points are identified in each of several sub-groups of images of the series of images and, based on resection and forward intersection using the plurality of respectively corresponding image points,

a reference point field is built up comprising a plurality of reference points of the surrounding, wherein coordinates of the reference points are derived, and

the poses for the images are determined,

a spatial representation, particularly a point cloud, comprising 3d-information about the surrounding is computed by forward intersection using the series of images and the determined poses, particularly by using a dense matching algorithm,

wherein the control and evaluation unit comprises several diversified units, wherein

at least one unit, being integrated in the camera module, carries out at least part of the SLAM-evaluation and

at least one other unit, being disposed externally to the camera module, carries out at least parts of computing of the spatial representation.

156. The subsystem according to claim 155, wherein the at least one other unit is a cloud server.

162. A position measuring resource, particularly being mountable on a surveying pole, to be used as part of a surveying system that is adapted to determine positions of the position measuring resource, comprising

a GNSS-antenna and/or a retro-reflector and

a camera-arrangement with at least one camera for capturing images of a surrounding,

wherein the camera-arrangement and the GNSS-antenna and/or the retro-reflector are integrated into a single common, non-divisible housing of the position measuring resource, and

wherein the common housing comprises a coupling unit designed so that the position measuring resource is modularly attachable to the pole.

163. The position measuring resource according to claim 162, wherein

the camera-arrangement is built and designed in such a way that panoramic images with a field of view of 360°

at least in azimuthal direction are capturable, particularly wherein the camera-arrangement comprises at least two cameras, especially at least four cameras, arranged in the common housing.

164. The position measuring resource according to claim 162, wherein

the position measuring resource comprises a sensor unit being integrated into the common housing, the sensor unit comprising at least one of the following sensors:

inertial measuring unit,

gyroscope,

tilt sensor,

accelerometer and/or

magnetic compass.

165. The position measuring resource according to claim 162, wherein

wherein

a control and evaluation unit having stored a program with program code so as to control and execute a functionality in which—

when moving along a path through a surrounding

a series of images of the surrounding is captured with the at least one camera of the camera-arrangement, the series comprising an amount of images captured with different poses of the camera, the poses representing respective positions and orientations of the camera,

a SLAM-evaluation with a defined algorithm using the series of images is performed, wherein a plurality of respectively corresponding image points are identified in each of several sub-groups of images of the series of images and, based on resection and forward intersection using the plurality of respectively corresponding image points,

a reference point field is built up comprising a plurality of reference points of the surrounding, wherein coordinates of the reference points are derived, and

the poses for the images are determined.