Real-time automatic vehicle camera calibration

• US 10,694,175 B2
• Filed: 11/11/2016
• Issued: 06/23/2020
• Est. Priority Date: 12/28/2015
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Bovyrin, Alexander, Kozlov, Alexander
• Primary Examiner(s): Nawaz, Talha M
• Application Number:
  US15/769,124
• Publication Number:
  US 20180324415A1
• Time in Patent Office: 1,320 Days
• Field of Search: None
• CPC Class Codes: B60R 2300/402 Image calibration
• CPC Class Codes: B60W 30/02 Control of vehicle driving stability
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 20/582 of traffic signs
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 23/57 Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices

Abstract

A camera facing the front of a vehicle while the vehicle is moving on the road may be calibrated by receiving sequential images from the camera. Image key points in the area limited by the road location are selected. The key points are tracked using an optical flow method. A filtering procedure is applied to the key points to identify the straight-line motion of the vehicle. At least two straight lines corresponding to opposite sides of the road. A calibration algorithm is applied to the at least two lines to determine a vanishing point. The pitch and/or yaw angles of the camera are then calculated.

Claims

1. A method of calibrating a camera facing the front of a vehicle while the vehicle is moving on the road, comprising:

receiving sequential images from the camera;

finding a plurality of key points in an image area limited by a location of the road;

tracking the plurality of key points using an optical flow method;

determining a plurality of lines based on trajectories of the plurality of key points;

finding a plurality of intersections for pairs of lines of the plurality of lines;

determining a plurality of map values based on lengths of the lines associated with the plurality of intersections;

determining a vanishing point using the plurality of map values; and

calculating pitch or yaw angles of the camera based on the vanishing point determined using the plurality of map values.

2. The method of claim 1 including determining whether the pitch or yaw angles of the camera are within a predetermined acceptable threshold.

3. The method of claim 1 including fitting the plurality of lines to the trajectories of the plurality of key points.

4. The method of claim 1, wherein determining the vanishing point comprises determining that a maximum map value exceeds a first threshold for calibration.

5. The method of claim 1, wherein determining each map value comprises averaging line lengths for a pair of lines.

6. The method of claim 1 including determining whether tangents of the plurality of lines are below a second threshold to determine vehicle motion stability.

7. The method of claim 6 including comparing a tangent sign to a majority of tangent signs.

8. The method of claim 7 including calculating a ratio of lines not found to lines found.

9. The method of claim 8 including determining if the ratio exceeds the second threshold.

10. The method of claim 9 including, in response to a determination that the second threshold is exceeded, incrementing an error count.

11. The method of claim 10 including, in response to a determination that the error count has reached the second threshold, determining that proposed camera parameters are incorrect.

12. One or more non-transitory computer readable media storing instructions to perform a sequence comprising:

receiving sequential images from the camera;

finding a plurality of key points in an image area limited by a location of the road;

tracking the plurality of key points using an optical flow method;

determining a plurality of lines based on trajectories of the plurality of key points;

finding a plurality of intersections for pairs of lines of the plurality of lines;

determining a plurality of map values based on lengths of the lines associated with the plurality of intersections;

determining a vanishing point using the plurality of map values; and

calculating pitch or yaw angles of the camera based on the vanishing point determined using the plurality of map values.

13. The medium of claim 12, further storing instructions to perform a sequence including determining whether the pitch or yaw angles of the camera are within a predetermined acceptable threshold.

14. The medium of claim 12, further storing instructions to perform a sequence including the plurality of lines to the trajectories of the plurality of key points.

15. The medium of claim 12, wherein determining the vanishing point comprises determining that a maximum map value exceeds a first threshold for calibration.

16. The medium of claim 12, wherein determining each map value comprises averaging line lengths for a pair of lines.

17. The medium of claim 12, further storing instructions to perform a sequence including determining whether tangents of the plurality of lines are below a second threshold to determine vehicle motion stability.

18. The medium of claim 17, further storing instructions to perform a sequence including comparing a tangent sign to a majority of tangent signs.

19. The medium of claim 18, further storing instructions to perform a sequence including calculating a ratio of lines not found to lines found.

20. The medium of claim 19, further storing instructions to perform a sequence including determining if the ratio exceeds the second threshold.

21. The medium of claim 20, further storing instructions to perform a sequence including, in response to a determination that the second threshold is exceeded, incrementing an error count.

22. The medium of claim 21, further storing instructions to perform a sequence including, in response to a determination that the error count has reached the second threshold determining that proposed camera parameters are incorrect.

23. An apparatus comprising:

a processor to;

receive sequential images from a camera of a vehicle,

find a plurality of key points in an image area limited by a location of a road,

track the plurality of key points using an optical flow method,

determine a plurality of lines based on trajectories of the plurality of key points,

find a plurality of intersections for pairs of lines of the plurality of lines,

determine a plurality of map values based on lengths of the lines associated with the plurality of intersections,

determine a vanishing point using the plurality of map values, and

calculate pitch or yaw angles of the camera based on the vanishing point determined using the plurality of map values; and

a memory coupled to said processor.

24. The apparatus of claim 23, said processor to determine whether the pitch or yaw angles of the camera are within a predetermined acceptable threshold.

25. The apparatus of claim 23, said processor to fit the plurality of lines to the trajectories of the plurality of key points.

26. The apparatus of claim 23, said processor to determine the vanishing point based on a determination that a maximum map value exceeds a first threshold for calibration.

27. The apparatus of claim 23, said processor to determine each map value by averaging line lengths for a pair of lines.

28. The apparatus of claim 27, said processor to determine whether tangents of the plurality of lines are below a second threshold to determine vehicle motion stability.

29. The apparatus of claim 28, said processor to compare a tangent sign to a majority of tangent signs.

30. The apparatus of claim 28, said processor to calculate a ratio of lines not found to lines found.