Automated image labeling for vehicles based on maps

• US 10,209,089 B2
• Filed: 04/03/2017
• Issued: 02/19/2019
• Est. Priority Date: 04/03/2017
• Status: Active Grant

• Current Assignee: Robert Bosch GmbH
• Original Assignee: Robert Bosch GmbH
• Inventors: Behrendt, Karsten, Witt, Jonas
• Primary Examiner(s): Rahmjoo, Manuchehr
• Application Number:
  US15/477,993
• Publication Number:
  US 20180283892A1
• Time in Patent Office: 687 Days
• CPC Class Codes: G01C 21/3647 Guidance involving output of stored or live camera images or video streams
• CPC Class Codes: G01C 21/3673 Labelling using text of road map data items, e.g. road names, POI names
• CPC Class Codes: G06F 16/29 Geographical information databases
• CPC Class Codes: G06F 16/54 Browsing; Visualisation therefor
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 18/24133 Distances to prototypes
• CPC Class Codes: G06F 18/253 of extracted features
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 7/344 involving models
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/806 of extracted features
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: G06V 30/19173 Classification techniques
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

A method and system for navigating a vehicle using automatically labeled images. The system includes a camera configured to capture an image of a roadway and an electronic processor communicatively connected to the camera. The electronic processor is configured to load a map that includes a first plurality of features and receive the image. The image includes a second plurality of features. The electronic processor is further configured to project the map onto the image; detect the second plurality of features within the image; and align the map with the image by aligning the first plurality of features with the second plurality of features. The electronic processor is further configured to copy a label describing one of the first plurality of features onto a corresponding one of the second plurality of features to create a labelled image and to use the labelled image to assist in navigation of the vehicle.

Claims

1. A method of navigation for a vehicle using automatically labeled images, the method comprising:

loading a map that includes a first plurality of features into an electronic processor of the vehicle;

capturing an image that includes a second plurality of features with a camera of the vehicle;

projecting the map onto the image;

detecting, with the electronic processor, the second plurality of features within the image;

aligning the map with the image by aligning the first plurality of features with the second plurality of features;

copying a label describing one of the first plurality of features onto a corresponding one of the second plurality of features to create a labelled image; and

using the labelled image to assist in navigation of the vehicle;

training an object detector within the electronic processor with the labelled image;

after training the object detector, detecting the second plurality of features within a second image with the object detector to create an updated labelled image;

correcting an alignment of the map with the second image based on the updated labelled image.

2. The method according to claim 1, further comprising training the object detector within the electronic processor with the second image.

3. The method according to claim 1, wherein the first plurality of features and the second plurality of features are lane markers.

4. The method according to claim 1, wherein loading the map that includes the first plurality of features includes loading a navigation-grade map with roadway infrastructure identified within the navigation-grade map.

5. The method according to claim 1, wherein loading the map that includes the first plurality of features includes receiving the map from a central database.

6. The method according to claim 1, wherein loading the map into the electronic processor includes downloading the map from a central server.

7. The method according to claim 1, wherein loading the map into the electronic processor includes generating the map based on input from a sensor of the vehicle.

8. A system for navigating a vehicle using automatically labeled images, the system comprising:

a camera configured to capture an image of a roadway;

an electronic processor communicatively connected to the camera, the electronic processor configured to

load a map that includes a first plurality of features,

receive the image, the image including a second plurality of features,

project the map onto the image,

detect the second plurality of features within the image,

align the map with the image by aligning the first plurality of features with the second plurality of features,

copy a label describing one of the first plurality of features onto a corresponding one of the second plurality of features to create a labelled image, and

use the labelled image to assist in navigation of the vehicle;

train an object detector within the electronic processor with the labelled image;

after training the object detector, detecting the second plurality of features within a second image with the object detector to create an updated labelled image;

correct an alignment of the map with the second image based on the updated labelled image.

9. The system according to claim 8, wherein the electronic processor is further configured to train the object detector within the electronic processor with the updated labelled image.

10. The system according to claim 8, wherein the first plurality of features and the second plurality of features are lane markers.

11. The system according to claim 8, wherein the electronic processor is further configured to load a navigation-grade map with roadway infrastructure identified within the navigation-grade map.

12. The system according to claim 8, wherein the electronic processor is further configured to receive the map from a central database.

13. The system according to claim 8, wherein the electronic processor is further configured to download the map from a central server.

14. The system according to claim 8, wherein the electronic processor is further configured to generate the map based on input from a sensor of the vehicle.