Unsupervised online learning of overhanging structure detector for map generation

• US 10,152,635 B2
• Filed: 01/24/2017
• Issued: 12/11/2018
• Est. Priority Date: 01/24/2017
• Status: Active Grant

• Current Assignee: HERE Global B.V. (Nokia Corporation)
• Original Assignee: HERE Global B.V. (Nokia Corporation)
• Inventors: Viswanathan, Anirudh
• Primary Examiner(s): Osifade, Idowu O
• Application Number:
  US15/413,996
• Publication Number:
  US 20180211111A1
• Time in Patent Office: 686 Days
• Field of Search: None
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/231 Hierarchical techniques, i.e. dividing or merging pattern sets so as to obtain a dendrogram
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06N 3/088 Non-supervised learning, e.g. competitive learning
• CPC Class Codes: G06T 2207/30184 Infrastructure
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/774 Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 20/176 Urban or other man-made structures
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road

Abstract

Methods, apparatuses, computer program products, and systems are provided for training a network to act as an overhanging structure detector using an unsupervised machine learning technique. An image is received that was captured by an image capturing device in communication with a probe apparatus on board a vehicle, wherein the image comprises at least a portion of a roadway. A sky projection is generated based on at least a portion of the image. It is determined whether the sky projection comprises a feature that defines a feature direction that is substantially non-vertical. Responsive to determining that the sky projection does comprise a feature that defines a feature direction that is substantially non-vertical, it is determined that the image comprises an overhanging structure.

Claims

1. A method comprising:

receiving an image captured by an image capturing device in communication with a probe apparatus on board a vehicle, wherein the image comprises at least a portion of a roadway;

generating a sky projection based on at least a portion of the image;

determining whether the sky projection comprises a feature that defines a feature direction that is substantially non-vertical; and

responsive to determining that the sky projection does comprise a feature that defines a feature direction that is substantially non-vertical, determining that the image comprises an overhanging structure.

2. A method according to claim 1, further comprising:

responsive to determining that the sky projection does comprise a feature that defines a feature direction that is substantially non-vertical, adding the image to a first cluster, the first cluster consisting of images comprising overhanging structures;

or

responsive to determining that the sky projection does not comprise a feature that defines a feature direction that is substantially non-vertical, adding the image to a second cluster, the second cluster consisting of images not comprising overhanging structures,

wherein the first cluster and second cluster represent an unsupervised overhanging structure detector network.

3. A method according to claim 2, further comprising training a network to act as an overhanging structure detector using the first and second clusters.

4. A method according to claim 1, further comprising:

using the overhanging structure detector to identify an overhanging structure in one or more images captured by one or more probe apparatuses each onboard a vehicle, each of the one or more images tagged with a geolocation determined by a location sensor in communication with the corresponding probe apparatus;

analyzing the image to determine overhanging structure information corresponding to the identified overhanging structure; and

updating map information based on at least one of (a) the geolocation, (b) at least a portion of the overhanging structure information, or (c) both.

5. A method according to claim 1, wherein the feature direction is substantially non-vertical when the feature direction satisfies a configurable angular threshold requirement.

6. A method according to claim 1, wherein the feature direction is substantially non-vertical when the feature direction substantially differs from one or more roadside object directions defined by a roadside object pattern within the sky projection.

7. A method according to claim 1, further comprising analyzing the image to determine overhanging structure information.

8. A method according to claim 7, wherein the overhanging structure information comprises at least one of the height of the overhanging structure, the length of the overhanging structure, or both.

9. A method according to claim 1, wherein generating the sky projection from the image comprises using a homography to map pixels of the image to a top-down view, the top-down view comprising the sky projection, wherein the pixels of at least a bottom portion of the image are not mapped to the top-down view, the bottom portion comprising at least one of a portion of a roadway, a portion of a ground, a portion of a vehicle, or a combination thereof.

10. An apparatus comprising at least one processor and at least one memory storing computer program code, with the at least one memory and the computer program code configured to, with the processor, cause the apparatus to at least:

receive an image captured by an image capturing device in communication with a probe apparatus on board a vehicle, wherein the image comprises at least a portion of a roadway;

generate a sky projection based on at least a portion of the image;

determine whether the sky projection comprises a feature that defines a feature direction that is substantially non-vertical; and

responsive to determining that the sky projection does comprise a feature that defines a feature direction that is substantially non-vertical, determine that the image comprises an overhanging structure.

11. An apparatus according to claim 10, wherein the at least one memory and the computer program code are configured to, with the processor, to cause the apparatus to at least:

responsive to determining that the sky projection does comprise a feature that defines a feature direction that is substantially non-vertical, add the image to a first cluster, the first cluster consisting of images comprising overhanging structures;

or

responsive to determining that the sky projection does not comprise a feature that defines a feature direction that is substantially non-vertical, add the image to a second cluster, the second cluster consisting of images not comprising overhanging structures,

wherein the first cluster and second cluster represent an unsupervised overhanging structure detector network.

12. An apparatus according to claim 11, wherein the at least one memory and the computer program code are configured to, with the processor, to cause the apparatus to at least train a network to act as an overhanging structure detector using the first and second clusters.

13. An apparatus according to claim 10, wherein the at least one memory and the computer program code are configured to, with the processor, to cause the apparatus to at least:

use the overhanging structure detector to identify an overhanging structure in one or more images captured by one or more probe apparatuses each onboard a vehicle, each of the one or more images tagged with a geolocation determined by a location sensor in communication with the corresponding probe apparatus;

analyze the image to determine overhanging structure information corresponding to the identified overhanging structure; and

update map information based on at least one of (a) the geolocation, (b) at least a portion of the overhanging structure information, or (c) both.

14. An apparatus according to claim 10, wherein the feature direction is substantially non-vertical when the feature direction satisfies a configurable angular threshold requirement.

15. An apparatus according to claim 10, wherein the feature direction is substantially non-vertical when the feature direction substantially differs from one or more roadside object directions defined by a roadside object pattern within the sky projection.

16. An apparatus according to claim 10, wherein the at least one memory and the computer program code are configured to, with the processor, to cause the apparatus to at least analyze the image to determine overhanging structure information.

17. An apparatus according to claim 16, wherein the overhanging structure information comprises at least one of the height of the overhanging structure, the length of the overhanging structure, or both.

18. An apparatus according to claim 10, wherein to generate the sky projection from the image, the at least one memory and the computer program code are configured, with the processor, to cause the apparatus to at least use a homography to map pixels of the image to a top-down view, the top-down view comprising the sky projection, wherein the pixels of at least a bottom portion of the image are not mapped to the top-down view, the bottom portion comprising at least one of a portion of a roadway, a portion of a ground, a portion of a vehicle, or a combination thereof.

19. A computer program product comprising at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein with the computer-executable program code instructions comprising program code instructions configured to:

receive an image captured by an image capturing device in communication with a probe apparatus on board a vehicle, wherein the image comprises at least a portion of a roadway;

generate a sky projection based on at least a portion of the image;

determine whether the sky projection comprises a feature that defines a feature direction that is substantially non-vertical; and

responsive to determining that the sky projection does comprise a feature that defines a feature direction that is substantially non-vertical, determine that the image comprises an overhanging structure.

20. A computer program product according to claim 19, wherein the computer-executable program code instructions further comprise program code instructions configured to:

responsive to determining that the sky projection does comprise a feature that defines a feature direction that is substantially non-vertical, add the image to a first cluster, the first cluster consisting of images comprising overhanging structures;

or

responsive to determining that the sky projection does not comprise a feature that defines a feature direction that is substantially non-vertical, add the image to a second cluster, the second cluster consisting of images not comprising overhanging structures,

wherein the first cluster and second cluster represent an unsupervised overhanging structure detector network.