Method and control system for surveying and mapping a terrain while operating a bulldozer

• US 9,481,982 B1
• Filed: 04/19/2016
• Issued: 11/01/2016
• Est. Priority Date: 04/21/2015
• Status: Active Grant

• Current Assignee: Hexagon Technology Center GmbH (Hexagon AB)
• Original Assignee: Hexagon Technology Center GmbH (Hexagon AB)
• Inventors: Pettersson, Bo, Zebhauser, Benedikt
• Primary Examiner(s): CAMBY, RICHARD M
• Application Number:
  US15/133,145
• Publication Number:
  US 20160312446A1
• Time in Patent Office: 196 Days
• Field of Search: 701/50
• US Class Current: 1/1
• CPC Class Codes: E02F 3/76 Graders, bulldozers, or the like with scraper plates or ploughshare-like elements soil-working A01B; Levelling scarifying devices street cleaning E01H; construction of roads E01C19/00, E01C23/00
• CPC Class Codes: E02F 3/842 using electromagnetic, optical or photoelectric beams, e.g. laser beams
• CPC Class Codes: E02F 3/844 for positioning the blade, e.g. hydraulically
• CPC Class Codes: E02F 9/2041 Automatic repositioning of implements, i.e. memorising determined positions of the implement for dipper-arms or bucket-arms E02F3/434, E02F3/437, E02F3/438, E02F3/439 take precedence
• CPC Class Codes: E02F 9/262 with follow-up actions to control the work tool, e.g. controller
• CPC Class Codes: G01C 11/08 the pictures not being supported in the same relative position as when they were taken
• CPC Class Codes: G01S 17/08 for measuring distance only indirect measurement G01S17/46; active triangulation systems G01S17/48
• CPC Class Codes: G05D 1/0251 extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G05D 1/0278 using satellite positioning signals, e.g. GPS
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A Method for generating scaled terrain information while operating a bulldozer. The bulldozer may include a driving unit comprising a set of drive wheels, a motor connected to at least one of the drive wheels, a blade for altering the surface of the terrain, at least one camera for capturing images of the environment, the camera being positioned and aligned in a known manner relative to the bulldozer, and a controlling and processing unit. A method may include moving the bulldozer while concurrently generating a set of image data by capturing an image series of terrain sections with the at least one camera so that at least two images of the image series cover an amount of identical points in the terrain, and either applying a simultaneous localization and mapping (SLAM) algorithm or a stereo photogrammetry algorithm to the set of image data and thereby deriving terrain data.

Claims

1. A method for generating scaled terrain information while operating a bulldozer, the bulldozer comprising

a driving unit comprising:

a set of drive wheels being part of a continuous track, and

a motor connected to at least one of the drive wheels for providing movability of the bulldozer,

a blade for altering the surface of the terrain,

at least one camera for capturing images of the environment, the camera being positioned and aligned in a known manner relative to the bulldozer, and

a controlling and processing unit,

wherein the method comprises;

moving the bulldozer in the terrain whilst concurrently generating a set of image data by capturing an image series of terrain sections with the at least one camera so that at least two successive images of the image series cover an amount of identical points in the terrain, wherein the terrain sections are defined by a viewing field of the at least one camera at respective positions of the camera while moving,

and either applying

a simultaneous localisation and mapping (SLAM) algorithm or

a stereo photogrammetry algorithm

to the set of image data and thereby deriving terrain data, the terrain data comprising

a point cloud representing the captured terrain and

position data relating to a relative position of the bulldozer in the terrain, and

scaling the point cloud by applying an absolute scale information to the terrain data.

2. The method according to claim 1, wherein:

the absolute scale information is derived by receiving movement data from a sensor unit of the bulldozer providing information about a distance covered by moving the bulldozer, the sensor unit being designed as

a sensor for detecting revolutions of at least one of the wheels of the bulldozer, and/or

a distance measuring unit for measuring distances to an object by emitting laser light and receiving the laser light reflected at the object, and/or

a range camera, and/or

an inertial measuring unit (IMU), and/or

a receiving unit for global navigation satellite system (GNSS) signals and/or

differential global navigation satellite system (DGNSS) signals.

3. The method according to claim 1, wherein:

the absolute scale information is derived by capturing a reference image of a reference body of known appearance and/or position in the terrain and deriving the absolute scale information by image processing based on the appearance of the reference body in the captured reference image and of a known magnification ratio of the camera, wherein the dimensions, spatial orientation and/or shape of the reference body is pre-known.

4. The method according to claim 1, wherein:

transmitting the set of image data and/or the terrain data to a database and storing the data in it, wherein

the bulldozer comprises a storing unit comprising the database, and/or

a remote controlling unit for controlling the bulldozer comprises the storing unit comprising the database, and/or

the database is embodied by a data cloud stored on a remote server, wherein

the terrain data is derived from the image data by cloud computing based on the simultaneous localisation and mapping (SLAM) algorithm or the stereo photogrammetry algorithm, and/or

the scaling of the point cloud is performed by cloud computing.

5. The method according to claim 1, wherein:

transmitting the image data and/or the terrain data to other working machines located in the terrain and/or

receiving image data and/or the terrain data from other working machines located in the terrain

via a network provided by the working machines, wherein each working machine is configured to;

to process the image data and/or terrain data for updating the current terrain state, and/or

to transmit the image data and/or terrain data to a database, and/or

to transmit and/or to reference position data relating to a relative position of the other working machines in the terrain.

6. The method according to claim 1, wherein:

extracting at least one state parameter from the set of image data and/or from the terrain data which represents an actual state of at least one designated terrain section,

comparing the state parameter to a predetermined threshold for the respective state, and

deriving constructional information based on the comparison of the predetermined threshold and the state parameter,

wherein the state parameter provides at least one terrain factor of a group of terrain factors, the group of terrain factors comprising at least one or more of the factors selected from the list consisting of;

granularity of the terrain,

substance of the terrain,

brightness or colour of the terrain,

humidity of the terrain,

planarity of the terrain,

volume of the terrain, and

height of the terrain.

7. The method according to claim 6, wherein:

the constructional information is provided to a user of the bulldozer, together with a related recommendation concerning a suggested bulldozing of the respective at least one designated terrain section.

8. The method according to claim 1, wherein:

moving the bulldozer and/or blade, based on the terrain data, wherein

the terrain data represents at least an actual shape of the captured terrain,

constructional data is provided representing a target shape for the terrain and

the blade is guided based on the constructional data so that the actual shape of the terrain is transferred into the target shape,

wherein the bulldozer is positioned at a designated position in the terrain based on the terrain data and the blade is guided according to a designated shape of a terrain based on the terrain data and the constructional data.

9. The method according to claim 1, wherein:

creating a digital terrain map is based on the point cloud.

10. The method according to claim 1, wherein:

creating a digital terrain map includes creating a digital terrain 3D-model.

11. The method according to claim 1, wherein:

creating a digital terrain map is based on the terrain data gathered by other working machines and obtained via the network.

12. The method according to claim 1, wherein recognizing objects constituting an obstacle in the proposed path of the bulldozer, wherein a collision of the bulldozer with a recognized object is automatically avoided.

13. The method according to claim 1, wherein recognizing objects to be excluded or subtracted from the digital terrain map.

14. The method according to claim 1, wherein recognizing the current state of the terrain sections, which remain to receive a treatment to be provided by at least one other working machine, and sending out a signal via the network to the at least one other working machine, directing the at least one other working machine to fulfill the corresponding treatment in the respective terrain section.

20. One or more non-transitory computer-readable media storing one or more programs that are configured, when executed, to cause one or more processors to execute the method as recited in claim 1.

15. A control system for use with a bulldozer, the bulldozer comprising:

a driving unit comprising

a set of drive wheels, and

a motor connected to at least one of the drive wheels for providing movability of the bulldozer and

a blade for altering the surface of the terrain, and

the control system comprising at least

one camera for capturing images of a terrain, the camera to be positioned and aligned in a known manner relative to the bulldozer, and

a controlling and processing unit, wherein;

the control system provides a functionality adapted to generate scaled terrain information by executing, respectively controlling, following steps;

concurrently generating a set of image data by capturing an image series of terrain sections with the camera whilst moving the bulldozer in the terrain so that at least two images of the image series cover an amount of identical points in the terrain, wherein the terrain sections are defined by a viewing area of the camera at respective positions of the camera while moving,

either applying;

a simultaneous localisation and mapping (SLAM) algorithm or

a stereo photogrammetry algorithm

to the set of image data and thereby deriving terrain data, the terrain data comprising

a point cloud representing the captured terrain and

position data relating to a relative position of the bulldozer in the terrain, and

scaling the point cloud by applying an absolute scale information to the terrain data.

16. The control system according to claim 15, further comprising:

a counting sensor counting the number of revolutions of a wheel of the bulldozer, the counting sensor providing the absolute scale information; and

/or

a distance measuring unit for measuring distances to an object by emitting laser light and receiving the laser light reflected at the object; and

/or

a range camera; and

/or

swivelling means for swivelling the camera so that the camera is alignable in a designated direction for capturing the terrain sections or a reference body which provides the absolute scale information.

17. The control system according to claim 15, further comprising:

at least one tilt sensor; and

/or

a compass; and

/or

at least one profiler; and

/or

at least one scanner; and

/or

at least one inertial sensor to be positioned on the blade.

18. The control system according to claim 15, further comprising a receiving unit for

global navigation satellite system (GNSS) signals, and/or

differential global navigation satellite system (DGNSS) signals.

19. The control system according to claim 15, further comprising:

a display for showing the operator of the bulldozer recommendations for altering the terrain; and

/or

controlling means for controlling the path of the bulldozer; and

/or

the position of the blade.