AUGMENTED ROAD LINE DETECTION AND DISPLAY SYSTEM

• US 20180031384A1
• Filed: 07/28/2016
• Published: 02/01/2018
• Est. Priority Date: 07/28/2016
• Status: Active Grant

• Current Assignee: Michigan Technological University, Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Michigan Technological University, Toyota Motor Engineering & Manufacturing North America Incorporated (Toyota Motor Corporation)
• Inventors: LEE, Jae Seung, SCHMALENBERG, Paul Donald, SUBOTIC, Nikola Stevan
• Granted Patent: US 11,248,925 B2
• Time in Patent Office: Days
• CPC Class Codes: B60R 2300/205 using a head-up display
• CPC Class Codes: B60R 2300/305 merging camera image with lines or icons
• CPC Class Codes: B60R 2300/308 by overlaying the real scene, e.g. through a head-up display on the windscreen
• CPC Class Codes: B60R 2300/8086 for vehicle path indication
• CPC Class Codes: G01C 21/3602 Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera
• CPC Class Codes: G01C 21/365 Guidance using head up displays or projectors, e.g. virtual vehicles or arrows projected on the windscreen or on the road itself
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads

Abstract

An augmented road line display system that includes one or more sensors installed on a vehicle, one or more external databases, and processing circuitry. The processing circuitry is configured to receive inputs from the one or more databases, sensors of the vehicle, and a sub-system of the vehicle, build and validate a road line model to detect or predict a road line based on the inputs received, determine environmental conditions based on the inputs from one or more of the databases, and a sub-system of the vehicle, assign weights to the inputs received based on the environmental conditions to generate weighted inputs, and execute the road line model to determine the road line based on the weighted inputs.

Claims

1. An augmented road line display system, comprising:

one or more sensors installed on a vehicle;

one or more databases; and

processing circuitry configured to

receive inputs from the one or more databases, one or more sensors of the vehicle, and a sub-system of the vehicle,

build and validate a road line model to detect or predict a road line based on the inputs received,

determine environmental conditions based on the inputs from the one or more databases and the sub-system of the vehicle,

assign a first set of confidence levels to a first set of inputs which cannot be trusted due to the determined environmental conditions, the first set of confidence levels chosen from a first range of available confidence levels,

assign a second set of confidence levels to a second set of inputs which can be trusted due to the determined environmental conditions, the second set of confidence levels chosen from a second range of the available confidence levels different from the first range,

assign weights to the inputs based on the first and second set of confidence levels to generate weighted inputs, and

execute the road line model to determine the road line based on the weighted inputs.

2. The system according to claim 1, wherein the processing circuitry is further configured to:

detect one or more objects on or along a road in a driving direction,

construct an augmented road line by augmenting the road line with the one or more objects detected on the road, and

display the augmented road line on a heads-up display or on a windshield located in front of a driver of the vehicle.

3. The system according to claim 1, wherein the one or more databases is external to the vehicle and includes at least one of:

an infrastructure database,

a traffic update server,

a connected vehicle database, and

a public real time kinetic database.

4. The system according to claim 1, wherein the sub-system of the vehicle includes at least one of:

a camera vision sub-system,

a vehicle-to-vehicle communication subsystem,

a navigation sub-system, and

an inertial location sub-system.

5. The system according to claim 1, wherein the one or more sensors of the vehicle includes at least one of a lidar and radar.

6. The system according to claim 1, wherein the road line model is a statistical model based on parameters computed using inputs received from the at least one database, the one or more sensors of the vehicle, and the sub-system of the vehicle.

9. The system according to claim 1, wherein a higher weight is assigned to the second set of inputs received from the one or more sensors of the vehicle compared to the first set of inputs received from the one or more databases when access to the one or more databases is unavailable.

19. The system according to claim 1, wherein the sub-system of the vehicle includes a vehicle-to-vehicle communication subsystem.

20. The system according to claim 1, wherein the one or more sensors of the vehicle includes a chip scale lidar.

7. (canceled)

8. (canceled)

10. A method for an augmented road line display, the method comprising:

receiving, using processing circuitry, inputs from one or more databases, one or more sensors of the vehicle, and a sub-system of the vehicle;

building and validating, using the processing circuitry, a road line model to detect or predict a road line based on the inputs received;

determining, using the processing circuitry, environmental conditions based on the inputs from the one or more databases, and the sub-system of the vehicle;

assigning, using the processing circuitry, a first set of confidence levels to a first set of inputs which cannot be trusted due to the determined environmental conditions, the first set of confidence levels chosen from a first range of available confidence levels,

assigning, using the processing circuitry, a second set of confidence levels to a second set of inputs which can be trusted due to the determined environmental conditions, the second set of confidence levels chosen from a second range of the available confidence levels different from the first range,

assigning, using the processing circuitry, weights to the inputs received based on the first and second set of confidence levels to generate weighted inputs; and

executing, using the processing circuitry, the road line model to determine the road line based on the weighted inputs.

11. The method according to claim 10, further comprising:

detecting, using the processing circuitry, one or more objects along a road in a driving direction;

constructing, using the processing circuitry, an augmented road line by augmenting the road line with the one or more objects detected on the road; and

displaying, using the processing circuitry, the augmented road line on a heads-up display or on a windshield located in front of a driver of the vehicle.

12. The method according to claim 10, wherein the one or more databases is external to the vehicle and includes at least one of:

an infrastructure database,

a traffic update server,

a connected vehicle database, and

a public real time kinetic database.

13. The method according to claim 10, wherein the sub-system of the vehicle includes at least one of:

a camera vision sub-system,

a vehicle-to-vehicle communication subsystem,

a navigation sub-system, and

an inertial location sub-system.

14. The method according to claim 10, wherein the one or more sensors of the vehicle includes at least one of a lidar and radar.

15. The method according to claim 10, wherein a road line model is a statistical model based on parameters computed using inputs received from the one or more databases, one or more sensors of a vehicle, and the sub-system of the vehicle.

18. The method according to claim 10, wherein a higher weight is assigned to the second set of inputs received from the one or more sensors of the vehicle compared to the first set of inputs received from the one or more databases when access to the one or more databases is unavailable.

16. (canceled)

17. (canceled)