Method and apparatus for detecting objects from terrestrial based mobile mapping data

• US 8,325,979 B2
• Filed: 10/30/2007
• Issued: 12/04/2012
• Est. Priority Date: 10/30/2006
• Status: Active Grant

• Current Assignee: Tomtom Global Content B.V (TomTom NV)
• Original Assignee: Tomtom Global Content B.V (TomTom NV)
• Inventors: Taborowski, Lukasz Piotr, Gliszczynski, Rafal Jan, Wysocki, Arkadiusz, Kmiecik, Marcin Michal
• Primary Examiner(s): Fitzpatrick, Atiba O
• Application Number:
  US12/311,465
• Publication Number:
  US 20100020074A1
• Time in Patent Office: 1,862 Days
• Field of Search: 382100-107, 348/113, 348169-173, 340907-910, 340933-940, 356 3- 30, 73488-492
• US Class Current: 382/103
• CPC Class Codes: G01C 11/025 by scanning the object
• CPC Class Codes: G01C 21/3811 Point data, e.g. Point of Interest [POI]
• CPC Class Codes: G01C 21/3837 Data obtained from a single source
• CPC Class Codes: G01C 21/3848 Data obtained from both position sensors and additional sensors
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30184 Infrastructure
• CPC Class Codes: G06T 2207/30236 Traffic on road, railway or crossing
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80

Abstract

A method of detecting objects from terrestrial based mobile mapping data is disclosed, wherein the terrestrial based mobile mapping data has been captured by way of a terrestrial based mobile mapping vehicle driving on a road having a driving direction, the mobile mapping data including laser scanner data, source images obtained by at least one camera and position and orientation data of the vehicle, wherein the laser scanner data includes laser points, each laser point having associated position and orientation data, and each source image comprises associated position and orientation data. In at least one embodiment, the method includes: retrieving a position and orientation of the vehicle; filtering the laser scanner data in dependence of the position and orientation of the vehicle to obtain laser points corresponding to regions of interest; retrieving a source image associated with the position and orientation of the vehicle; mapping the laser points corresponding to regions of interest to image coordinates of the source image to generate a recognition mask; combining the recognition mask and the source image to obtain candidate 3D images representative of possible objects within the regions of interest; and, detecting a group of objects from the candidate 3D images. By combining image recognition and laser scanner recognition the detection rate can be increased to a very high percentage, thereby substantially reducing human effort. Furthermore, the generating of regions of interest in the laser data, enables a significant reduction of the processing power and/or the processing time needed to detect the objects in the images.

Claims

1. Method of detecting objects from terrestrial based mobile mapping data, wherein the terrestrial based mobile mapping data has been captured by way of a terrestrial based mobile mapping vehicle driving on a road having a driving direction, the mobile mapping data including laser scanner data, source images obtained by at least one camera and position and orientation data of the terrestrial based mobile mapping vehicle, wherein the laser scanner data includes laser points, each laser point having associated position and orientation data, and each source image comprises associated position and orientation data, the method comprising:

retrieving a position and orientation of the terrestrial based mobile mapping vehicle;

filtering the laser scanner data in dependence of the retrieved position and orientation of the terrestrial based mobile mapping vehicle to obtain laser points corresponding to regions of interest;

retrieving a source image associated with the retrieved position and orientation of the terrestrial based mobile mapping vehicle;

mapping the obtained laser points corresponding to regions of interest to image coordinates of the retrieved source image to generate a recognition mask;

combining the generated recognition mask and the retrieved source image to obtain candidate 3D Images representative of possible objects within the regions of interest; and

,

detecting a group of objects from the obtained candidate 3D Images.

2. Method according to claim 1, wherein the combining comprises:

segmenting the retrieved source image in dependence of the generated recognition mask to obtain intermediate candidate images; and

adding depth information corresponding to the obtained laser points of regions of interest to corresponding pixels of said intermediate candidate images to obtain the candidate 3D images.

3. Method according to claim 2, wherein the adding depth information comprises:

using the position and orientation data captured by position and orientation sensors of the terrestrial based mobile mapping vehicle, to accurately transform laser point coordinates into image coordinates of a source image;

determining depth information for said image coordinates in dependence of the said laser point coordinates and the position and orientation data; and

adding the depth information to a pixel having said image coordinates.

4. Method according to claim 2, wherein the detecting comprises for each candidate 3D image:

determining size and shape of candidate 3D image in dependence of said depth information;

determining, if size and shape are according to defined values, the depth consistency of the candidate 3D image surface in dependence of the depth information;

determining, if candidate 3D image surface is depth consistent, the orientation of the candidate 3D image to the driving direction in dependence upon the depth information;

determining, if candidate 3D image surface is not depth consistent, the orientation of the candidate 3D image to the driving direction in dependence of a previous source image; and

adding candidate 3D image in dependence of the determined orientation to the group of objects.

5. Method according to claim 1, wherein the filtering comprises:

selecting laser points corresponding to a surface in the regions of interest in dependence of the driving direction and the local depth consistency of the laser scanner data; and

, wherein the combining comprises;

segmenting the source image in dependence of the recognition mask to obtain intermediate candidate images, and

adding depth information corresponding to the laser points corresponding to the surface to corresponding pixels of said intermediate candidate images to obtain the candidate 3D images.

6. Method according to claim 5, wherein the detecting comprises for each candidate 3D image:

determining size and shape of the candidate 3D image independence of said depth information; and

adding candidate 3D image in dependence of size and shape to the group of objects.

7. Method according to claim 4, wherein depth consistency of a surface is a measure of the planarity of the surface.

9. Method according to claim 1, wherein the position and orientation data of the vehicle has been determined by a positioning platform.

11. A non-transitory processor readable medium carrying a computer program, when loaded on a computer arrangement, is arranged to perform the method according to claim 1.

12. Method according to claim 3, wherein the detecting comprises for each candidate 3D image:

determining size and shape of candidate 3D image in dependence of said depth information;

determining, if size and shape are according to defined values, the depth consistency of the candidate 3D image surface in dependence of the depth information;

determining, if candidate 3D image surface is depth consistent, the orientation of the candidate 3D image to the driving direction in dependence upon the depth information;

determining, if candidate 3D image surface is not depth consistent, the orientation of the candidate 3D image to the driving direction in dependence of a previous source image; and

adding candidate 3D image in dependence of the determined orientation to the group of objects.

13. Method according to claim 5, wherein depth consistency of a surface is a measure of the planarity of the surface.

14. Method according to claim 6, wherein depth consistency of a surface is a measure of the planarity of the surface.

17. A non-transitory computer readable medium including program segments for, when executed on a computer device, causing the computer device to implement the method of claim 1.

8. Method of detecting objects from terrestrial based mobile mapping data, wherein the terrestrial based mobile mapping data has been captured by way of a terrestrial based mobile mapping vehicle driving on a road having a driving direction, the mobile mapping data including laser scanner data, source images obtained by at least one camera and position and orientation data of the terrestrial based mobile mapping vehicle, wherein the laser scanner data includes laser points, each laser point having associated position and orientation data, and each source image comprises associated position and orientation data, the method comprising:

retrieving a position and orientation of the terrestrial based mobile mapping vehicle;

filtering the laser scanner data in dependence of the position and orientation of the terrestrial based mobile mapping vehicle to obtain laser points corresponding to obtain first regions of interest;

retrieving a source image associated with the position and orientation of the terrestrial based mobile mapping vehicle;

mapping the obtained laser points corresponding to the first regions of interest to image coordinates of the retrieved source image to generate a first recognition mask;

combining the generated first recognition mask and the retrieved source image to obtain first candidate 3D images representative of the first regions of interest;

detecting a first group of objects in dependence of the driving direction from the obtained first candidate 3D images;

filtering the laser scanner data in dependence of the position and orientation of the terrestrial based mobile mapping vehicle and the local depth consistency of the laser scanner data to obtain laser points corresponding to second regions of interest;

mapping the obtained laser points corresponding to the second regions of interest to image coordinates of the retrieved source image to generate a second recognition mask;

combining the generated second recognition mask and the retrieved source image to obtain second candidate 3D images representative of the second regions of interest;

detecting a second group of objects from the obtained second candidate 3D images; and

combining the detected first and second group of objects.

15. Method according to claim 8, wherein the position and orientation data of the vehicle has been determined by a positioning platform.

16. A non-transitory processor readable medium carrying a computer program, when loaded on a computer arrangement, is arranged to perform the method according to claim 8.

18. A non-transitory computer readable medium including program segments for, when executed on a computer device, causing the computer device to implement the method of claim 8.

10. An apparatus for capturing objects for use in a map database, the apparatus comprising:

input device;

a processor readable storage medium; and

a processor in communication with said input device and said processor readable storage medium; and

an output device to enable the connection with a display unit, said processor readable storage medium storing code to program said processor to perform detection of objects from terrestrial based mobile mapping data, wherein the terrestrial based mobile mapping data has been captured by way of a terrestrial based mobile mapping vehicle driving on a road having a driving direction, the mobile mapping data including laser scanner data, source images obtained by at least one camera and position and orientation data of the terrestrial based mobile mapping vehicle, wherein the laser scanner data includes laser points, each laser point having associated position and orientation data, and each source image includes associated position and orientation data, the processor being useable to;

retrieve a position and orientation of the terrestrial based mobile mapping vehicle;

filter the laser scanner data in dependence of the retrieved position and orientation of the terrestrial based mobile mapping vehicle to obtain laser points corresponding to regions of interest;

retrieve a source image associated with the retrieved position and orientation of the terrestrial based mobile mapping vehicle;

map the obtained laser points corresponding to regions of interest to image coordinates of the source image to generate a recognition mask;

combine the generated recognition mask and the retrieved source image to obtain candidate 3D images representative of possible objects within the regions of interest;

detect a group of objects from the obtained candidate 3D images; and

store the detected group of objects on the storage medium for characterizing the object.