Systems and methods for projecting a location of a nearby object into a map according to a camera image

• US 10,409,288 B2
• Filed: 05/03/2017
• Issued: 09/10/2019
• Est. Priority Date: 05/03/2017
• Status: Active Grant

• Current Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Toyota Research Institute, Inc. (Toyota Motor Corporation)
• Inventors: Kanzawa, Yusuke, Lee, Kuan-Hui
• Primary Examiner(s): Nguyen, Nga X
• Application Number:
  US15/585,878
• Publication Number:
  US 20180321686A1
• Time in Patent Office: 860 Days
• CPC Class Codes: B60W 30/00 Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/027 comprising intertial navigation means, e.g. azimuth detector inertial navigation G01C21/16; inertial navigation combined with non-inertial navigation instruments G01C21/165
• CPC Class Codes: G05D 1/0272 comprising means for registering the travel distance, e.g. revolutions of wheels measuring distance traversed on the ground by vehicles, e.g. using odometers G01C22/00
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G05D 1/0278 using satellite positioning signals, e.g. GPS
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G08G 1/166 for active traffic, e.g. moving vehicles, pedestrians, bikes

Abstract

System, methods, and other embodiments described herein relate to locating an object within an image and projecting the object into a map. In one embodiment, a method includes, responsive to identifying a nearby vehicle in an image of an external environment surrounding a scanning vehicle, determining a relative location of the nearby vehicle in relation to the scanning vehicle using the image. The method includes projecting the nearby vehicle into a map according to at least the relative location determined from the image. The method includes controlling the scanning vehicle according to the map.

Claims

1. An object projection system associated with a scanning vehicle, comprising:

one or more processors;

a memory including instructions and that is communicably coupled to the one or more processors, the memory storing;

a projection module for determining a relative location of a nearby vehicle in relation to the scanning vehicle from an image without using sensor data from additional sensors about an external environment when the sensor data about the nearby vehicle from the additional sensors is unavailable and responsive to identifying the nearby vehicle in the image of the external environment surrounding the scanning vehicle, the image being generated by a camera,

wherein the projection module determines the relative location of the nearby vehicle by (i) projecting a current centerline of a lane of the scanning vehicle along a roadway in the image, and (ii) determining a closest point of the current centerline to a bounding box that outlines the nearby vehicle, and

wherein the projection module projects the nearby vehicle into a map according to at least the relative location determined from the image; and

a control module for controlling the scanning vehicle according to the map.

2. The object projection system of claim 1, wherein the projection module identifies the nearby vehicle by analyzing the image to distinguish the nearby vehicle from the external environment and outlining the nearby vehicle using the bounding box.

3. The object projection system of claim 1, wherein the projection module further determines the relative location by generating the relative location according to a calculated distance of the nearby vehicle from the closest point to the current centerline, and wherein the relative location is a position that is relative to the scanning vehicle.

4. The object projection system of claim 1, wherein the projection module further determines the closest point by (i) extending an edge of the bounding box to intersect the current centerline at an intersection point and (ii) searching for the closest point to the bounding box among proximate points around the intersection point by calculating distances of the proximate points to the bounding box.

5. The object projection system of claim 1, wherein the projection module further determines operating characteristics of the scanning vehicle by retrieving associated data from one or more sensors of the scanning vehicle, wherein the one or more sensors include a global position system (GPS) sensor, a tachometer, and an inertial monitoring unit (IMU) sensor, and wherein the operating characteristics include at least a current location of the scanning vehicle.

6. The object projection system of claim 1, wherein the projection module further projects the nearby vehicle into the map by localizing the nearby vehicle using the relative location and one or more operating characteristics of the scanning vehicle, wherein the projection module further projects the current centerline by filtering secondary centerlines of additional travel lanes from the image that interfere with determining the closest point.

7. The object projection system of claim 1, wherein the projection module further acquires the image of the external environment, wherein the image is from the camera and includes a field-of-view along a roadway around the scanning vehicle, and wherein the map is an obstacle map that identifies locations of obstacles in the external environment.

8. A non-transitory computer-readable medium storing instructions that when executed by one or more processors cause the one or more processors to:

responsive to identifying a nearby vehicle in an image of an external environment surrounding a scanning vehicle, the image being generated by a camera, determining a relative location of the nearby vehicle in relation to the scanning vehicle from the image without using sensor data from additional sensors about an external environment when the sensor data about the nearby vehicle from the additional sensors is unavailable,

wherein the instructions to determine the relative location of the nearby vehicle include instructions to (i) project a current centerline of a lane of the scanning vehicle along a roadway in the image, and (ii) determine a closest point of the current centerline to a bounding box that outlines the nearby vehicle;

project the nearby vehicle into a map according to at least the relative location determined from the image; and

control the scanning vehicle according to the map.

9. The non-transitory computer-readable medium of claim 8, wherein the instructions to determine the relative location include instructions to generate the relative location according to a calculated distance of the nearby vehicle from the closest point to the current centerline, and

wherein the instructions to determine the closest point include instructions to (i) extend an edge of the bounding box to intersect the current centerline at an intersection point and (ii) search for the closest point to the bounding box among proximate points around the intersection point by calculating distances of the proximate points to the bounding box.

10. The non-transitory computer-readable medium of claim 8, wherein the instructions to determine operating characteristics of the scanning vehicle include instructions to retrieve associated data from one or more sensors of the scanning vehicle, wherein the one or more sensors include a global position system (GPS) sensor, a tachometer, and an inertial monitoring unit (IMU) sensor, and wherein the operating characteristics include at least a current location of the scanning vehicle.

11. The non-transitory computer-readable medium of claim 8, wherein the instructions to project the nearby vehicle into the map include instructions to localize the nearby vehicle using the relative location and one or more operating characteristics of the scanning vehicle, and wherein the instructions to project the current centerline include instructions to filter secondary centerlines of additional travel lanes from the image that interfere with determining the closest point.

12. A method of localizing objects around a scanning vehicle using an image generated by a camera, comprising:

responsive to identifying a nearby vehicle in the image of an external environment surrounding the scanning vehicle, determining a relative location of the nearby vehicle in relation to the vehicle from the image without using sensor data from additional sensors about the external environment when the sensor data about the nearby vehicle from the additional sensors is unavailable,

wherein determining the relative location of the nearby vehicle includes (i) projecting a current centerline of a lane of the scanning vehicle along a roadway in the image, and (ii) determining a closest point of the current centerline to a bounding box that outlines the nearby vehicle;

projecting the nearby vehicle into a map according to at least the relative location determined from the image; and

controlling the scanning vehicle according to the map.

13. The method of claim 12, wherein identifying the nearby vehicle includes analyzing the image to distinguish the nearby vehicle from the external environment and outlining the nearby vehicle using the bounding box.

14. The method of claim 12, wherein determining the relative location includes generating the relative location according to a calculated distance of the nearby vehicle from the closest point to the current centerline,

wherein projecting the nearby vehicle into the map includes localizing the nearby vehicle using the relative location and one or more operating characteristics of the scanning vehicle, wherein the one or more operating characteristics include at least a current location of the scanning vehicle, and wherein the relative location is a position that is relative to the scanning vehicle.

15. The method of claim 12, further comprising:

determining operating characteristics of the scanning vehicle by retrieving associated data from one or more sensors of the scanning vehicle, wherein the one or more sensors include a global position system (GPS) sensor, a tachometer, and an inertial monitoring unit (IMU) sensor, and

wherein projecting the current centerline includes filtering secondary centerlines of additional travel lanes from the image that interfere with determining the closest point.

16. The method of claim 12, wherein determining the closest point includes (i) extending an edge of the bounding box to intersect the current centerline at an intersection point and (ii) searching for the closest point to the current centerline among proximate points around the intersection point by calculating distances of the proximate points to the bounding box.

17. The method of claim 12, further comprising:

acquiring, by the scanning vehicle, the image of the external environment, wherein the image is from the camera and includes a field-of-view along a roadway around the scanning vehicle.