AUTONOMOUS GARDENING VEHICLE WITH CAMERA

• US 20150163993A1
• Filed: 12/08/2014
• Published: 06/18/2015
• Est. Priority Date: 12/12/2013
• Status: Active Grant

• Current Assignee: Hexagon Technology Center GmbH (Hexagon AB)
• Original Assignee: Hexagon Technology Center GmbH (Hexagon AB)
• Inventors: PETTERSSON, Bo
• Granted Patent: US 9,603,300 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: A01D 34/008 for automated or remotely controlled operation
• CPC Class Codes: A01G 25/09 Watering arrangements making use of movable installations on wheels or the like
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0251 extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593
• CPC Class Codes: G05D 1/0253 extracting relative motion information from a plurality of images taken successively, e.g. visual odometry, optical flow determining position or orientation from images G06T7/70
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30

Abstract

Some embodiments described herein include a method for generating scaled terrain information with an unmanned autonomous gardening vehicle. In some embodiments the gardening vehicle includes a driving unit comprising a set of at least one drive wheel and a motor connected to the at least one drive wheel for providing movability of the gardening vehicle, a gardening-tool and a camera for capturing images of a terrain, the camera being positioned and aligned in known manner relative to the gardening vehicle. In context of the method the gardening vehicle is moved in the terrain while concurrently generating a set of image data by capturing an image series of terrain sections so that at least two (successive) images of the image series cover an amount of identical points in the terrain, wherein the terrain sections are defined by a viewing area of the camera at respective positions of the camera while moving.

Claims

1. A method for generating scaled terrain information with an unmanned autonomous gardening vehicle, the gardening vehicle comprising

a driving unit comprising

a set of at least one drive wheel, and

a motor connected to the at least one drive wheel for providing movability of the gardening vehicle;

a gardening-tool; and

a camera for capturing images of a terrain, the camera being positioned and aligned in known manner relative to the gardening vehicle;

wherein the method comprises;

moving the gardening vehicle in the terrain whilst concurrently generating a set of image data by capturing an image series of terrain sections so that at least two images of the image series cover an amount of identical points in the terrain, wherein the terrain sections are defined by a viewing area of the camera at respective positions of the camera while moving;

applying a simultaneous localisation and mapping (SLAM) algorithm to the set of image data and thereby deriving terrain data, the terrain data comprising;

a point cloud representing the captured terrain, and

position data relating to a relative position of the gardening vehicle in the terrain; and

scaling the point cloud by applying an absolute scale information to the terrain data.

2. The method according to claim 1, wherein scaling of the point cloud comprises scaling the position data.

3. The method according to claim 1, wherein the at least two images comprise at least two successive images.

4. The method according to claim 1, wherein the unmanned autonomous gardening vehicle is implemented as an unmanned autonomous lawnmower, wherein the gardening-tool is implemented as a cutting-tool.

5. The method according to claim 1, wherein the absolute scale information is derived by receiving movement data from a sensor unit of the gardening vehicle providing information about a distance covered by moving the gardening vehicle, the sensor unit being designed as at least one of

a sensor for detecting revolutions of a wheel of the gardening vehicle;

a distance measuring unit for measuring distances to an object by emitting laser light and receiving the laser light reflected at the object;

a range camera; and

an inertial measuring unit (IMU).

6. The method according to claim 5, wherein wheel comprises a drive wheel.

7. The method according to claim 1, wherein the absolute scale information is derived by capturing a reference image of a reference body of known appearance and/or position in the terrain and deriving the absolute scale information by image processing based on the appearance of the reference body in the captured reference image and of a known magnification ratio of the camera, wherein at least one of the dimensions, spatial orientation, and shape of the reference body is pre-known.

8. The method according to claim 1, wherein:

transmitting the set of image data and/or the terrain data to a database and storing the data in it, wherein at least one of;

the gardening vehicle comprises a storing unit comprising the database,

a remote controlling unit for controlling the gardening vehicle comprises the storing unit comprising the database, and

the database is embodied by a data cloud stored on a remote server, wherein the terrain data is derived from the image data by cloud computing based on the simultaneous localisation and mapping (SLAM) algorithm and/or the scaling of the point cloud is performed by cloud computing.

9. The method according to claim 1, wherein a borderline of a working area for the gardening vehicle, within which the gardening vehicle is controllable to autonomously move and work, is defined by at least one of:

teaching the borderline by at least one of;

moving the gardening vehicle along a desired path,

capturing a series of border-images of terrain corresponding to the path and of a defined vicinity relative to the path, the series of border-images represents the image series, and

providing the series of border-images for controlling the gardening vehicle, and

setting the borderline on basis of a terrain map, wherein border-position data is provided for controlling the gardening vehicle.

10. The method according to claim 9, wherein in case the borderline is defined by teaching,

continuously comparing the border-images of the series of border-images with actual images by image processing, the actual images being captured while moving the gardening vehicle inside the working area,

deriving a rate of matching for every actual image based on the comparison and

controlling the movement of the gardening vehicle based on the rate of matching so that the gardening vehicle automatically moves only inside the working area, providing a movement-controlling command for adapting the movement direction of the gardening vehicle if the rate of matching exceeds a predetermined matching-threshold.

11. The method according to claim 1, further comprising:

extracting at least one state parameter from the set of image data and/or from the terrain data which represents an actual state of at least one designated terrain section, the state relating to a state of at least one plant and ground,

comparing the state parameter to a predetermined threshold for the respective state and

deriving gardener information based on the comparison of the predetermined threshold and the state parameter.

12. The method according to claim 11, wherein the state parameter provides at least one terrain factor of a group of terrain factors, the group of terrain factors comprising at least one of:

plant height,

plant growth,

humidity of the terrain,

density of plants,

planarity of the terrain and

brightness or colour of the terrain.

13. The method according to claim 11, wherein at least one of:

the gardener information is provided to a user of the gardening vehicle together with a related recommendation concerning a suggested treatment of the respective at least one designated terrain section, and

the gardening-tool is applied based on the gardener information.

14. The method according to claim 13, wherein the gardening tool comprises at least one of a hedge-cutter, a tree-branch cutter, a grass-cutter, scissors, a fertilising unit, a pesticide unit, a watering unit, and a lawn thatcher.

15. The method according to claim 1, further comprising:

applying the gardening-tool based on the terrain data, in particular based on the point cloud, wherein

the terrain data represents at least an actual shape of an object in the captured terrain,

gardening data is provided representing a target shape for the object and

the gardening-tool is guided based on the gardening data so that the actual shape of the object is transferred into the target shape,

wherein the gardening vehicle is positioned at a designated position in the terrain based on the terrain data and the gardening-tool is guided according to a designated shape of a plant based on the terrain data and the gardening data.

16. The method according to claim 1, further comprising at least one of:

providing controlling information for controlling a further gardening unit in the terrain, wherein the controlling information is derived by the terrain data and an actual position and orientation of the further gardening unit in the terrain, wherein the position and orientation of the gardening unit is derived from the terrain data by data processing or by image processing of an image covering at least a part of the gardening unit and a part of the terrain, in particular wherein the gardening unit comprises a 6DoF-target providing the determination of the orientation of the gardening unit with six degrees of freedom by image processing, and

creating a digital terrain map based on the point cloud.

17. An unmanned autonomous gardening vehicle comprising

a driving unit comprising:

a set of at least one drive wheel, and

a motor connected to the at least one drive wheel for providing movability of the gardening vehicle;

a gardening-tool;

a camera for capturing images of a terrain, the camera being positioned and aligned in known manner relative to the gardening vehicle; and

a controlling unit for controlling the gardening vehicle, wherein the gardening vehicle providing a functionality adapted to generate scaled terrain information by performing the following;

moving the gardening vehicle in the terrain whilst concurrently generating a set of image data by capturing an image series of terrain sections so that at least two images of the image series cover an amount of identical points in the terrain, wherein the terrain sections are defined by a viewing area of the camera at respective positions of the camera while moving,

applying a simultaneous localisation and mapping (SLAM) algorithm to the set of image data and thereby deriving terrain data, the terrain data comprising;

a point cloud representing the captured terrain and

position data relating to a relative position of the gardening vehicle in the terrain, and

scaling the point cloud by applying an absolute scale information to the terrain data.

18. The unmanned autonomous gardening vehicle according to claim 17, wherein the unmanned autonomous gardening vehicle is implemented as an unmanned autonomous lawnmower, wherein the gardening-tool is implemented as a cutting-tool.

19. The unmanned autonomous gardening vehicle according to claim 17, wherein the gardening vehicle comprises at least one of:

a counting sensor counting the number of revolutions of a wheel of the gardening vehicle, the counting sensor providing the absolute scale information;

a distance measuring unit for measuring distances to an object by emitting laser light and receiving the laser light reflected at the object;

a range camera; and

swivelling means for swivelling the camera so that the camera is alignable in a desired direction for capturing the terrain sections or a reference body which provides the absolute scale information.

20. A non-transitory computer program product having computer-executable instructions for controlling and executing the method comprising:

moving the gardening vehicle in the terrain whilst concurrently generating a set of image data by capturing an image series of terrain sections so that at least two images of the image series cover an amount of identical points in the terrain, wherein the terrain sections are defined by a viewing area of the camera at respective positions of the camera while moving;

applying a simultaneous localisation and mapping (SLAM) algorithm to the set of image data and thereby deriving terrain data, the terrain data comprising;

a point cloud representing the captured terrain, and

position data relating to a relative position of the gardening vehicle in the terrain; and

scaling the point cloud by applying an absolute scale information to the terrain data.