Autonomous gardening vehicle with camera

• US 9,603,300 B2
• Filed: 12/08/2014
• Issued: 03/28/2017
• Est. Priority Date: 12/12/2013
• Status: Active Grant

• Current Assignee: Hexagon Technology Center GmbH (Hexagon AB)
• Original Assignee: Hexagon Technology Center GmbH (Hexagon AB)
• Inventors: Pettersson, Bo
• Primary Examiner(s): Jabr, Fadey
• Assistant Examiner(s): SHUDY, ANGELINA M
• Application Number:
  US14/563,815
• Publication Number:
  US 20150163993A1
• Time in Patent Office: 841 Days
• Field of Search: 701/28, 701/23
• US Class Current: 1/1
• CPC Class Codes: A01D 34/008 for automated or remotely controlled operation
• CPC Class Codes: A01G 25/09 Watering arrangements making use of movable installations on wheels or the like
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0251 extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593
• CPC Class Codes: G05D 1/0253 extracting relative motion information from a plurality of images taken successively, e.g. visual odometry, optical flow determining position or orientation from images G06T7/70
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30

Abstract

Some embodiments described herein include a method for generating scaled terrain information with an unmanned autonomous gardening vehicle. In some embodiments the gardening vehicle includes a driving unit comprising a set of at least one drive wheel and a motor connected to the at least one drive wheel for providing movability of the gardening vehicle, a gardening-tool and a camera for capturing images of a terrain, the camera being positioned and aligned in known manner relative to the gardening vehicle. In context of the method the gardening vehicle is moved in the terrain while concurrently generating a set of image data by capturing an image series of terrain sections so that at least two (successive) images of the image series cover an amount of identical points in the terrain, wherein the terrain sections are defined by a viewing area of the camera at respective positions of the camera while moving.

Claims

1. A method for generating scaled terrain information with an unmanned autonomous gardening vehicle, the gardening vehicle comprising

a driving unit comprising

a set of at least one drive wheel, and

a motor connected to the at least one drive wheel for providing movability of the gardening vehicle;

a gardening-tool; and

a camera for capturing images of a terrain, the camera being positioned and aligned in known manner relative to the gardening vehicle;

wherein the method comprises;

moving the gardening vehicle in the terrain whilst concurrently generating a set of image data by capturing an image series of terrain sections so that at least two images of the image series cover an amount of identical points in the terrain, wherein the terrain sections are defined by a viewing area of the camera at respective positions of the camera while moving;

applying a simultaneous localization and mapping (SLAM) algorithm to the set of image data and thereby deriving terrain data, the terrain data comprising;

a three-dimensional point cloud representing the captured terrain, and

position data relating to a relative position of the gardening vehicle in the terrain; and

scaling the three-dimensional point cloud by applying an absolute scale information to the terrain data,

wherein the absolute scale information is derived by

capturing a reference image of a reference body, of which at least one of the appearance and the position in the terrain is known, and

deriving the absolute scale information by image processing based on the appearance of the reference body in the captured reference image and of a known magnification ratio of the camera,

wherein at least one of the dimensions, spatial orientation, and shape of the reference body is pre-known.

2. The method according to claim 1, wherein the scaling of the point cloud comprises scaling the position data.

3. The method according to claim 1, wherein the at least two images comprise at least two successive images.

4. The method according to claim 1, wherein the unmanned autonomous gardening vehicle is implemented as an unmanned autonomous lawnmower, wherein the gardening-tool is implemented as a cutting-tool.

5. The method according to claim 1, wherein the absolute scale information is derived by receiving movement data from a sensor unit of the gardening vehicle providing information about a distance covered by moving the gardening vehicle, the sensor unit being designed as at least one of

a sensor for detecting revolutions of a wheel of the gardening vehicle;

a distance measuring unit for measuring distances to an object by emitting laser light and receiving the laser light reflected at the object;

a range camera; and

an inertial measuring unit (IMU).

6. The method according to claim 5, wherein the wheel is one of the at least one drive wheel.

7. The method according to claim 1, wherein:

transmitting at least one of the set of image data and the terrain data to a database and

storing the data in the database, wherein at least one of;

the gardening vehicle comprises a storing unit comprising the database,

a remote controlling unit for controlling the gardening vehicle comprises the storing unit comprising the database, and

the database is embodied by a data cloud stored on a remote server, wherein the terrain data is derived from the image data by cloud computing based on the simultaneous localization and mapping (SLAM) algorithm and/or the scaling of the point cloud is performed by cloud computing.

8. The method according to claim 1, wherein a borderline of a working area for the gardening vehicle, within which the gardening vehicle is controllable to autonomously move and work, is defined by at least one of:

teaching the borderline by at least one of;

moving the gardening vehicle along a desired path,

capturing a series of border-images of terrain corresponding to the path and of a defined vicinity relative to the path, the series of border-images represents the image series, and

providing the series of border-images for controlling the gardening vehicle, and

setting the borderline on basis of a terrain map, wherein border-position data is provided for controlling the gardening vehicle.

9. The method according to claim 8, wherein in case the borderline is defined by teaching,

continuously comparing the border-images of the series of border-images with actual images by image processing, the actual images being captured while moving the gardening vehicle inside the working area,

deriving a rate of matching for every actual image based on the comparison and

controlling the movement of the gardening vehicle based on the rate of matching so that the gardening vehicle automatically moves only inside the working area, providing a movement-controlling command for adapting the movement direction of the gardening vehicle if the rate of matching exceeds a predetermined matching-threshold.

10. The method according to claim 1, further comprising:

extracting at least one state parameter from at least one of the set of image data and the terrain data which represents an actual state of at least one designated terrain section, the actual state relating to a state of at least one plant and ground,

comparing the at least one state parameter to a predetermined threshold for a respective state and

deriving gardener information based on the comparison of the predetermined threshold and the at least one state parameter.

11. The method according to claim 10, wherein the at least one state parameter provides at least one terrain factor of a group of terrain factors, the group of terrain factors comprising at least one of:

plant height,

plant growth,

humidity of the terrain,

density of plants,

planarity of the terrain and

brightness or colour of the terrain.

12. The method according to claim 10, wherein at least one of:

the gardener information is provided to a user of the gardening vehicle together with a related recommendation concerning a suggested treatment of the respective at least one designated terrain section, and

the gardening-tool is applied based on the gardener information.

13. The method according to claim 12, wherein the gardening tool comprises at least one of a hedge-cutter, a tree-branch cutter, a grass-cutter, scissors, a fertilising unit, a pesticide unit, a watering unit, and a lawn thatcher.

14. The method according to claim 1, further comprising:

applying the gardening-tool based on the terrain data, in particular based on the point cloud, wherein

the terrain data represents at least an actual shape of an object in the captured terrain,

gardening data is provided representing a target shape for the object and

the gardening-tool is guided based on the gardening data so that the actual shape of the object is transferred into the target shape,

wherein the gardening vehicle is positioned at a designated position in the terrain based on the terrain data and the gardening-tool is guided according to a designated shape of a plant based on the terrain data and the gardening data.

15. The method according to claim 1, further comprising at least one of:

providing controlling information for controlling a further gardening unit in the terrain, wherein the controlling information is derived by the terrain data and an actual position and orientation of the further gardening unit in the terrain, wherein the position and orientation of the gardening unit is derived from the terrain data by data processing or by image processing of an image covering at least a part of the gardening unit and a part of the terrain, in particular wherein the gardening unit comprises a 6DoF-target providing the determination of the orientation of the gardening unit with six degrees of freedom by image processing, and

creating a digital terrain map based on the point cloud.

16. An unmanned autonomous gardening vehicle comprising

a driving unit comprising:

a set of at least one drive wheel, and

a motor connected to the at least one drive wheel for providing movability of the gardening vehicle;

a gardening-tool;

a camera for capturing images of a terrain, the camera being positioned and aligned in known manner relative to the gardening vehicle; and

a controlling unit for controlling the gardening vehicle, wherein the gardening vehicle providing a functionality adapted to generate scaled terrain information by performing the following;

moving the gardening vehicle in the terrain whilst concurrently generating a set of image data by capturing an image series of terrain sections so that at least two images of the image series cover an amount of identical points in the terrain, wherein the terrain sections are defined by a viewing area of the camera at respective positions of the camera while moving,

applying a simultaneous localization and mapping (SLAM) algorithm to the set of image data and thereby deriving terrain data, the terrain data comprising;

a point cloud representing the captured terrain and

position data relating to a relative position of the gardening vehicle in the terrain, and

scaling the point cloud by applying an absolute scale information to the terrain data,

wherein the absolute scale information is derived by

capturing a reference image of a reference body, of which at least one of the appearance and the position in the terrain is known, and

deriving the absolute scale information by image processing based on the appearance of the reference body in the captured reference image and of a known magnification ratio of the camera,

wherein at least one of the dimensions, spatial orientation, and shape of the reference body is pre-known.

17. The unmanned autonomous gardening vehicle according to claim 16, wherein the unmanned autonomous gardening vehicle is implemented as an unmanned autonomous lawnmower, wherein the gardening-tool is implemented as a cutting-tool.

18. The unmanned autonomous gardening vehicle according to claim 16, wherein the gardening vehicle comprises at least one of:

a counting sensor counting the number of revolutions of a wheel of the gardening vehicle, the counting sensor providing the absolute scale information;

a distance measuring unit for measuring distances to an object by emitting laser light and receiving the laser light reflected at the object;

a range camera; and

at least one swivel for swivelling the camera so that the camera is alignable in a desired direction for capturing the terrain sections or a reference body which provides the absolute scale information.

19. A non-transitory computer program product having computer-executable instructions for controlling and executing the method comprising:

moving the gardening vehicle in the terrain whilst concurrently generating a set of image data by capturing an image series of terrain sections so that at least two images of the image series cover an amount of identical points in the terrain, wherein the terrain sections are defined by a viewing area of the camera at respective positions of the camera while moving;

applying a simultaneous localization and mapping (SLAM) algorithm to the set of image data and thereby deriving terrain data, the terrain data comprising;

a point cloud representing the captured terrain, and

position data relating to a relative position of the gardening vehicle in the terrain; and

scaling the point cloud by applying an absolute scale information to the terrain data,

wherein the absolute scale information is derived by

capturing a reference image of a reference body, of which at least one of the appearance and the position in the terrain is known, and

deriving the absolute scale information by image processing based on the appearance of the reference body in the captured reference image and of a known magnification ratio of the camera,

wherein at least one of the dimensions, spatial orientation, and shape of the reference body is pre-known.