AUTOMATIC SCENE PARSING

• US 20150138310A1
• Filed: 11/05/2014
• Published: 05/21/2015
• Est. Priority Date: 11/19/2013
• Status: Abandoned Application

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Corporation
• Inventors: FAN, Lixin, BABAHAJIANI, Pouria
• Application Number:
  US14/534,124
• Publication Number:
  US 20150138310A1
• Time in Patent Office: Days
• US Class Current: 348/36
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20112 Image segmentation details
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/64 Three-dimensional objects

Abstract

A method comprising: obtaining an image about an at least one object of interest and a three-dimensional (3D) point cloud about said object of interest; aligning the 3D point cloud with the image; segmenting the image into a plurality of superpixels preserving a graph structure and spatial neighbourhood of pixel data of the image; associating the superpixels in the image with a subset of said 3D points, said subset of 3D points representing a planar patch in said object of interest; extracting a plurality of 3D features for each patch; and assigning at least one vector representing at least one 3D feature with a semantic label on the basis of at least one extracted 3D feature of the patch.

Claims

1. A method comprising:

obtaining an image about at least one object of interest and a three-dimensional (3D) point cloud about said object of interest;

aligning the 3D point cloud with the image;

segmenting the image into a plurality of superpixels preserving a graph structure and spatial neighbourhood of pixel data of the image;

associating the superpixels in the image with a subset of said 3D points, said subset of 3D points representing a planar patch in said object of interest;

extracting a plurality of 3D features for each patch; and

assigning at least one vector representing at least one 3D feature with a semantic label on the basis of at least one extracted 3D feature of the patch.

2. The method according to claim 1, wherein the 3D point cloud is derived using Light Detection And Ranging (LiDAR) method.

3. The method according to claim 1, the method further comprising:

establishing correspondences between at least one subset of 3D points and at least one superpixel of the image.

4. The method according to claim 1, the method further comprising:

segmenting the image into superpixels of substantially the same size.

5. The method according to claim 1, wherein extracting a plurality of 3D features for each patch involves extracting camera pose independent features and camera location dependent features.

6. The method according to claim 1, wherein the camera pose independent features include one or more of the following:

height of the patch above ground;

surface normal of the patch;

patch planarity;

density of 3D points in the patch; and

intensity of the patch defined as a function of reflectance of the light beams.

7. The method according to claim 5, wherein the camera location dependent features include one or more of the following:

horizontal distance of the patch to camera; and

depth information of the patch to camera.

8. The method according to claim 1, the method further comprising:

using a trained classifier algorithm for assigning said at least one vector representing the 3D feature with the semantic label.

9. The method according to claim 8, wherein the trained classifier algorithm is based on boosted decision trees, where a set of 3D features have been associated with manually labeled superpixels in training images during offline training.

10. An apparatus comprising at least one processor, memory including computer program code, the memory and the computer program code configured to, with the at least one processor, cause the apparatus to at least:

obtain an image about at least one object of interest and a three-dimensional (3D) point cloud about said object of interest;

align the 3D point cloud with the image;

segment the image into a plurality of superpixels preserving a graph structure and spatial neighbourhood of pixel data of the image;

associate the superpixels in the image with a subset of said 3D points, said subset of 3D points representing a planar patch in said object of interest;

extract a plurality of 3D features for each patch; and

assign at least one vector representing at least one 3D feature with a semantic label on the basis of at least one extracted 3D feature of the patch.

11. The apparatus according to claim 10, comprising computer program code configured to, with the at least one processor, cause the apparatus further to:

derive the 3D point cloud using Light Detection And Ranging (LiDAR) method.

12. The apparatus according to claim 10, comprising computer program code configured to, with the at least one processor, cause the apparatus further to:

establish correspondences between at least one subset of 3D points and at least one superpixel of the image.

13. The apparatus according to claim 10, comprising computer program code configured to, with the at least one processor, cause the apparatus further to:

segment the image into superpixels of substantially the same size.

14. The apparatus according to claim 10, wherein the plurality of 3D features for each patch comprises camera pose independent features and camera location dependent features.

15. The apparatus according to claim 14, wherein the camera pose independent features include one or more of the following:

height of the patch above ground;

surface normal of the patch;

patch planarity;

density of 3D points in the patch; and

intensity of the patch defined as a function of reflectance of the light beams.

16. The apparatus according to claim 14, wherein the camera location dependent features include one or more of the following:

horizontal distance of the patch to camera; and

depth information of the patch to camera.

17. The apparatus according to claim 10, comprising computer program code configured to, with the at least one processor, cause the apparatus further to:

use a trained classifier algorithm for assigning said at least one vector representing the 3D feature with the semantic label.

18. The apparatus according to claim 17, wherein the trained classifier algorithm is based on boosted decision trees, where a set of 3D features have been associated with manually labeled superpixels in training images during offline training.

19. The apparatus according to claim 10, the apparatus being functionally connected to a vehicle and further comprising one or more of the following:

a panoramic camera capable of capturing a panoramic view around the vehicle

a plurality of hi-resolution cameras, each arranged to capture a segment of the panoramic view around the vehicle;

a laser scanning unit for scanning around the vehicle with a laser beam, analysing reflected light and storing results as the point clouds; and

a satellite positioning unit for determining a location of the vehicle.

20. A computer readable storage medium stored with code thereon for use by an apparatus, which when executed by a processor, causes the apparatus to perform:

obtaining an image about at least one object of interest and a three-dimensional (3D) point cloud about said object of interest;

aligning the 3D point cloud with the image;

segmenting the image into a plurality of superpixels preserving a graph structure and spatial neighbourhood of pixel data of the image;

associating the superpixels in the image with a subset of said 3D points, said subset of 3D points representing a planar patch in said object of interest;

extracting a plurality of 3D features for each patch; and

assigning at least one vector representing at least one 3D feature with a semantic label on the basis of at least one extracted 3D feature of the patch.