Advanced path prediction

• US 10,871,783 B2
• Filed: 09/21/2018
• Issued: 12/22/2020
• Est. Priority Date: 03/23/2016
• Status: Active Grant

• Current Assignee: Netradyne Inc.
• Original Assignee: Netradyne Inc.
• Inventors: Julian, David Jonathan, Agrawal, Avneesh, Annapureddy, Venkata Sreekanta Reddy
• Primary Examiner(s): Odeh, Nadeem
• Application Number:
  US16/138,059
• Publication Number:
  US 20190025853A1
• Time in Patent Office: 823 Days
• Field of Search: 701 25
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G05D 1/0278 using satellite positioning signals, e.g. GPS
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle

Abstract

The present disclosure provides systems and methods for mapping a determined path of travel. The path of travel may be mapped to a camera view of a camera affixed to a vehicle. In some embodiments, the path of travel may be mapped to another view that is based on a camera, such as a bird'"'"'s eye view anchored to the camera'"'"'s position at a given time. These systems and methods may determine the path of travel by incorporating data from later points in time.

Claims

1. A method comprising:

determining an actual path of travel of a vehicle from a first time to a second time;

receiving a captured image, wherein the captured image is captured by a camera at the first time, and wherein the camera is mounted to the vehicle;

generating a first reference frame associated with the first time from the captured image using an inverse perspective model, wherein the first reference frame is an image derived from the captured image, and wherein the first reference frame is a bird'"'"'s eye view reference frame anchored to a position of the camera; and

mapping, at or after the second time, the actual path of travel to the first reference frame.

2. The method of claim 1, wherein mapping the actual path comprises:

orienting the actual path to align with the bird'"'"'s eye view reference frame; and

scaling the actual path based on a scale of the bird'"'"'s eye view reference frame.

3. The method of claim 2, further comprising:

determining a perspective mapping from the bird'"'"'s eye view reference frame to a camera reference frame; and

mapping the actual path from the first reference frame to the camera reference frame based on the perspective mapping.

4. The method of claim 1, further comprising:

receiving visual data from the camera at the first time;

determining a location of an object based on the visual data; and

determining a distance between the object and the actual path of travel in the first reference frame.

5. The method of claim 4, wherein the object is at least one of a second vehicle, a lane boundary, a pedestrian, a traffic sign, an animal, or a road debris.

6. The method of claim 4, further comprising:

determining a driver behavior metric based at least in part on the determined distance.

7. The method of claim 6, wherein the object is a second vehicle, and wherein the driver behavior metric is a tailgating score based on the determined distance.

8. The method of claim 1, further comprising:

predicting a second path of travel of the vehicle;

determining a difference between the predicted second path and the actual path.

9. The method of claim 8, further comprising:

determining a driver behavior metric based on the determined difference.

10. The method of claim 8, further comprising:

training a path prediction model based on the determined difference.

11. The method of claim 1, further comprising:

receiving visual data from the camera at the first time;

estimating a lane location at the first time based on the visual data; and

refining the estimated lane location based on the actual path of travel to produce a refined lane location.

12. The method of claim 11, further comprising:

training a lane detection model based on the estimated lane location and the refined lane location.

13. The method of claim 1, further comprising:

receiving visual data from the camera at the first time;

determining a plurality of regions of interest in the visual data based on the actual path of travel;

processing a first region of interest of the plurality of regions of interest at a first resolution;

processing a second region of interest of the plurality of regions of interest at a second resolution; and

detecting at least one object in at least one the first region of interest or the second region of interest.

14. The method of claim 13, wherein the first resolution is based on an estimated distance of objects within the first region of interest.

15. The method of claim 14, wherein the estimated distance of the objects is based on at least one of a typical lane width in the first region of interest, or the actual path of travel to the first region of interest, wherein the typical lane width is 3.7 meters.

16. An apparatus comprising:

at least one memory unit; and

at least one processor coupled to the at least one memory unit, in which the at least one processor is configured to;

determine an actual path of travel of a vehicle from a first time to a second time;

receive a captured image, wherein the captured image is captured by a camera at the first time, and wherein the camera is mounted to the vehicle;

generate a first reference frame associated with the first time from the captured image using an inverse perspective model, wherein the first reference frame is an image derived from the captured image, and wherein the first reference frame is a bird'"'"'s eye view reference frame anchored to a position of the camera; and

map, at or after the second time, the actual path of travel to the first reference frame.

17. A computer program product, the computer program product comprising:

a non-transitory computer-readable medium having program code recorded thereon, the program code, when executed by a processor, causes the processor to;

determine an actual path of travel of a vehicle from a first time to a second time;

receive a captured image, wherein the captured image is captured by a camera at the first time, and wherein the camera is mounted to the vehicle;

generate a first reference frame associated with the first time from the captured image using an inverse perspective model, wherein the first reference frame is an image derived from the captured image, and wherein the first reference frame is a bird'"'"'s eye view reference frame anchored to a position of the camera; and

map, at or after the second time, the actual path of travel to the first reference frame.