Automatic camera calibration

• US 10,580,164 B2
• Filed: 04/05/2018
• Issued: 03/03/2020
• Est. Priority Date: 04/05/2018
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Ramalingam, Ganesan, Ramjee, Ramachandran, Bhardwaj, Romil, Tummala, Gopi Krishna
• Primary Examiner(s): Liu, Li
• Application Number:
  US15/946,580
• Publication Number:
  US 20190311494A1
• Time in Patent Office: 698 Days
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/241 relating to the classification model, e.g. parametric or non-parametric approaches
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 2207/30236 Traffic on road, railway or crossing
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/54 of traffic, e.g. cars on the road, trains or boats

Abstract

This document relates to camera calibration. One example uses real-world distances and image coordinates of object features in images to determine multiple candidate camera calibrations for a camera. This example filters out at least some of the multiple candidate camera calibrations to obtain remaining calibrations, and obtains a final calibration for the camera from the remaining calibrations

Claims

1. A method performed on a computing device, the method comprising:

obtaining images captured by a camera;

identifying image coordinates of object features of objects shown in the images;

accessing a database of real-world distances between the object features for different object types;

obtaining, from the database, at least a first real-world distance between two object features for a first object type and a second real-world distance between the two object features for a second object type;

using the real-world distances and the image coordinates of the object features, determining multiple candidate camera calibrations for the camera, the multiple candidate camera calibrations comprising at least two candidate calibrations determined for a specific object in a specific image, including at least a first candidate calibration determined using the first real-world distance between the two object features for the first object type and a second candidate calibration determined using the second real-world distance between the two object features for the second object type;

filtering out at least some of the multiple candidate camera calibrations to obtain remaining calibrations; and

obtaining a final calibration for the camera from the remaining calibrations.

2. The method of claim 1, the objects comprising automobiles and the different object types comprising different automobile models.

3. The method of claim 2, wherein the specific object comprises a specific automobile in the specific image, the first candidate calibration is determined for the specific automobile using known dimensions of a first automobile model, and the second candidate calibration is determined for the specific automobile using known dimensions of a second automobile model other than the first automobile model.

4. The method of claim 3, wherein the determining the multiple candidate camera calibrations comprises:

for the specific automobile shown in the specific image;

providing, to a perspective-n-point solver, respective image coordinates of multiple automobile features in the specific image, first real-world coordinates of the multiple automobile features for the first automobile model, and second real-world coordinates of the multiple automobile features for the second automobile model; and

obtaining, from the perspective-n-point solver, a first candidate camera calibration that is specific to the first automobile model and a second candidate camera calibration that is specific to the second automobile model.

5. The method of claim 4, wherein the multiple automobile features include at least a left tail lamp and a right side mirror.

6. The method of claim 5, wherein the first real-world coordinates comprise first three-dimensional coordinates of the left tail lamp and the right side mirror for the first automobile model, and the second real-world coordinates comprise second three-dimensional coordinates of the left tail lamp and the right side mirror for the second automobile model.

13. The method of claim 1, wherein the multiple candidate camera calibrations are determined without using a classifier to determine object types of the objects shown in the images.

14. The method of claim 13, further comprising:

using a neural network to detect the object features in the images.

7. A system comprising:

a camera configured to capture images of objects;

a hardware processing unit; and

a storage resource storing computer-readable instructions which, when executed by the hardware processing unit, cause the hardware processing unit to;

obtain the images captured by the camera;

determine image coordinates of object features of objects shown in the images;

provide the image coordinates and real-world distances between the object features for different object types to a perspective-n-point solver, the perspective-n-point solver outputting multiple candidate camera calibrations for the different object types, the multiple candidate camera calibrations including at least two different candidate calibrations derived from a specific object in a specific image;

filter out at least some of the multiple candidate camera calibrations to obtain remaining calibrations; and

use the remaining calibrations to obtain a final calibration of the camera.

8. The system of claim 7, wherein the instructions, when executed by the hardware processing unit, cause the hardware processing unit to:

calculate respective Z axis unit vectors for the multiple candidate camera calibrations;

determine a mean Z axis unit vector across the multiple candidate camera calibrations; and

remove, from the multiple candidate camera calibrations, certain candidate camera calibrations having respective Z axis unit vectors with relatively large angular deviations from the mean Z axis unit vector.

9. The system of claim 7, wherein the instructions, when executed by the hardware processing unit, cause the hardware processing unit to:

calculate respective camera displacements for the multiple candidate camera calibrations;

determine a median camera displacement across the multiple candidate camera calibrations; and

remove, from the multiple candidate camera calibrations, at least some candidate camera calibrations having larger camera displacements than the median camera displacement and at least some other candidate camera calibrations having smaller camera displacements than the median camera displacement.

10. The system of claim 7, wherein the instructions, when executed by the hardware processing unit, cause the hardware processing unit to:

perform at least one averaging operation on the remaining calibrations to obtain the final calibration.

11. The system of claim 7, wherein the instructions, when executed by the hardware processing unit, cause the hardware processing unit to:

obtain further images separated in time;

using the final calibration of the camera, determine real-world locations of a particular object in the further images; and

calculate a speed of the particular object.

12. The system of claim 7, wherein the instructions, when executed by the hardware processing unit, cause the hardware processing unit to:

process further images using the final calibration to determine real-world locations of various objects shown in the further images; and

use the real-world locations of the various object to detect near-miss instances where two different objects come within a specified distance of one another.

15. A system comprising:

a camera configured to capture images of objects;

a hardware processing unit; and

a storage resource storing computer-readable instructions which, when executed by the hardware processing unit, cause the hardware processing unit to;

obtain first images captured by the camera;

provide the first images to a camera calibration module, the camera calibration module configured to determine a final camera calibration for the camera by filtering multiple different candidate calibrations derived from a specific object in a specific first image;

obtain the final camera calibration from the camera calibration module;

obtain second images captured by the camera; and

using the final camera calibration, perform at least one application function on the second images.

16. The system of claim 15, further comprising the camera calibration module.

17. The system of claim 15, wherein the computer-readable instructions, when executed by the hardware processing unit, cause the hardware processing unit to:

send the first images to a server that provides the camera calibration module; and

receive the final camera calibration from the server.

18. The system of claim 15, the at least one application function comprising detecting near-misses in the second images.

19. The system of claim 15, the at least one application function comprising detecting traffic congestion in the second images.

20. The system of claim 15, the at least one application function comprising detecting speeds of objects present in the second images.