Method and apparatus for automatic intrinsic camera calibration using images of a planar calibration pattern

• US 10,380,766 B2
• Filed: 03/06/2019
• Issued: 08/13/2019
• Est. Priority Date: 05/04/2017
• Status: Active Grant

• Current Assignee: Genius Sports SS LLC
• Original Assignee: Second Spectrum, Inc. (Genius Sports Ltd.)
• Inventors: Dubout, Charles Xavier Quentin, Ben Shitrit, Horesh Beny
• Primary Examiner(s): Bost, Dwayne D
• Assistant Examiner(s): Youssef, Menatoallah
• Application Number:
  US16/294,484
• Publication Number:
  US 20190197732A1
• Time in Patent Office: 160 Days
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30208 Marker matrix
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 10/757 Matching configurations of points or features

Abstract

The present disclosure relates to a high precision method, model, and apparatus for calibrating the intrinsic camera parameter values of one or more intrinsic parameters of a camera. The intrinsic parameters include focal lengths, skew, optical center, and radial distortion coefficients. The practical implementation of the solution consists of a camera capturing several images of a planar calibration pattern from different viewpoints, and a computer implemented processing pipeline. The solution does not require additional human input and is robust to misdetection of the calibration pattern in some of the images.

Claims

1. A method for automatic camera calibration of an image capture device, the method comprising:

obtaining, by a computing unit, a plurality of images of a planar calibration pattern captured by the image capture device from different respective angles;

detecting, by the computing unit, a plurality of distinguishable features depicted in the planar calibration pattern for each obtained image;

for each image, generating, by the computing unit, a respective lattice graph corresponding to the image based on the distinguishable features detected from the image, wherein each respective lattice graph has a plurality of nodes corresponding to distinguishable features and is defined with reference to a global planar coordinate system having an arbitrary origin;

repeatedly calibrating, by the computing unit, a plurality of intrinsic camera parameter values corresponding to one or more intrinsic camera parameters of the image capture device based on two or more distinguishable features detected in two or more of the images and the lattice graphs corresponding to the two or more images, wherein the calibrating is performed for two or more different combinations of the images, and wherein the repeatedly calibrating comprises, for each image, a plurality of recursions each starting with a single node detected in the image, one or more n-edge vectors and one or more visited directions and continuing via a repeated selection of one or more n-closest neighbors to the one or more n-edge vectors;

selecting, by the computing unit, one or more intrinsic camera parameter values from the plurality of intrinsic camera parameter values, wherein the one or more intrinsic camera parameter values are selected to minimize a robust estimator of a reprojection error over the obtained plurality of images; and

refining, by the computing unit, calibration of the plurality of intrinsic camera parameter values based on the one or more selected intrinsic camera parameters.

2. The method of claim 1, further comprising, repeated calibrating a plurality of extrinsic camera parameter values using two and three-dimensional correspondences for the calibration images and the plurality of intrinsic camera parameter values, wherein the repeated calibrating is performed for two or more different combinations of obtained images.

3. The method of claim 1, wherein obtaining the plurality of images includes obtaining a video containing the plurality of images.

4. The method of claim 1, wherein the plurality of images is a series of standstill images.

5. The method of claim 1, wherein the plurality of images is captured by the image capture device, wherein the computing unit is contained in a form factor of the image capture device.

6. The method of claim 1, wherein the plurality of images is captured by the image capture device and received by the computing unit via a network.

7. The method of claim 1, wherein detecting the plurality of distinguishable features includes, for each image, performing a blob detection process on the image.

8. The method of claim 7, wherein the blob detection process is Laplacian of Gaussian.

9. The method of claim 7, wherein the blob detection process is Maximally Stable Extremum Regions.

10. The method of claim 1, wherein each node of the lattice graph is based on a respective center of a respective distinguishable feature of the plurality of distinguishable features.

11. The method of claim 1, wherein refining the plurality of intrinsic camera parameter values is further based on all images of the plurality of obtained images that correspond to the one or more selected intrinsic camera parameter values.

12. The method of claim 1, further comprising configuring the one or more intrinsic camera parameters of the image capture device in accordance with the refined plurality of intrinsic camera parameter values.

13. The method of claim 1, wherein the one or more intrinsic camera parameters include focal lengths, skew, optical center, and/or radial distortion coefficients.

14. A computing unit configured for automatic camera calibration of an image capture device, the computing unit comprising:

memory storing a plurality of computer-executable instructions; and

one or more processors that execute the computer-executable instructions, the computer-executable instructions causing the one or more processors to;

obtain a plurality of images of a planar calibration pattern captured by the image capture device from different respective angles;

detect a plurality of distinguishable features depicted in the planar calibration pattern for each obtained image;

generate a lattice graph having a plurality of nodes based on the distinguishable features detected from each image, the lattice graph being generated with reference to a global planar coordinate system having an arbitrary origin;

repeatedly calibrate a plurality of intrinsic camera parameter values corresponding to one or more intrinsic camera parameters of the image capture device based on two or more distinguishable features detected in two or more of the images and the lattice graphs corresponding to the two or more images, wherein the calibrating is performed for two or more different combinations of the images, and wherein to repeatedly calibrate comprises, for each image, a plurality of recursions each starting with a single node detected in the image, one or more n-edge vectors and one or more visited directions and continuing via a repeated selection of one or more n-closest neighbors to the one or more n-edge vectors;

select, one or more intrinsic camera parameter values from the plurality of intrinsic camera parameter values, wherein the one or more intrinsic camera parameter values are selected to minimize a robust estimator of a reprojection error over the obtained plurality of images; and

refine calibration of the plurality of intrinsic camera parameters based on the one or more selected intrinsic camera parameters.

15. The computing unit of claim 14, wherein obtaining the plurality of images includes obtaining a video containing the plurality of images.

16. The computing unit of claim 14, wherein the plurality of images is a series of standstill images.

17. The computing unit of claim 14, wherein the plurality of images is captured by the image capture device, and wherein the computing unit is integrated in a form factor of the image capture device.

18. The computing unit of claim 14, wherein the plurality of images is captured by the image capture device and received by the computing unit via a network.

19. The computing unit of claim 14, wherein detecting the plurality of distinguishable features includes, for each image, performing a blob detection process on the image.

20. The computing unit of claim 19, wherein the blob detection process is Laplacian of Gaussian.

21. The computing unit of claim 19, wherein the blob detection process is Maximally Stable Extremum Regions.

22. The computing unit of claim 21, wherein each node of the lattice graph is based on a respective center of a respective distinguishable feature of the plurality of distinguishable features.

23. The computing unit of claim 21, wherein refining the plurality of intrinsic camera parameter values is further based on all images of the plurality of obtained images that correspond to the one or more selected intrinsic camera parameter values.

24. The computing unit of claim 14, wherein the computer-executable instructions further cause the one or more processors to configure the one or more intrinsic camera parameters of the image capture device in accordance with the refined plurality of intrinsic camera parameter values.

25. The computing unit of claim 14, wherein the computer-executable instructions further cause the one or more processors to repeatedly calibrate a plurality of extrinsic camera parameter values of one or more extrinsic camera parameters of the image capture device using two and three-dimensional correspondences for the calibration images and the plurality of intrinsic camera parameter values, wherein the repeated calibrating is performed for two or more different combinations of obtained images.

26. The computing unit of claim 14, wherein the one or more intrinsic camera parameters include focal lengths, skew, optical center, and/or radial distortion coefficients.

27. A method for automatic calibration of an image capture device, the method comprising:

obtaining, by a computing unit, a plurality of images of a planar calibration pattern captured by the image capture device, at least a portion of the plurality of images being captured from different respective angles;

detecting, by the computing unit, a plurality of distinguishable features depicted in the planar calibration pattern for each obtained image;

for each image, generating, by the computing unit, a respective lattice graph corresponding to the image based on the distinguishable features detected from the image, wherein each respective lattice graph is generated with reference to a global planar coordinate system having an arbitrary origin;

repeatedly calibrating, by the computing unit, a plurality of camera parameter values corresponding to one or more camera parameters of the image capture device based on two or more distinguishable features detected in two or more of the images and the lattice graphs corresponding to the two or more images;

selecting, by the computing unit, one or more camera parameter values from the plurality of camera parameter values, wherein the one or more camera parameter values are selected based on a robust estimator of a reprojection error over the obtained plurality of images; and

refining, by the computing unit, calibration of the plurality of camera parameter values based on the one or more selected camera parameters.

28. The method of claim 27, further comprising, repeatedly calibrating a second plurality of camera parameter values using at least one of two-dimensional and three-dimensional correspondences for the calibration images and the plurality of camera parameter values.

29. The method of claim 27, wherein each respective lattice graph has a plurality of nodes corresponding to distinguishable features and wherein each node of the plurality of nodes is based on a respective center of a corresponding distinguishable feature.

30. The method of claim 27, wherein the repeatedly calibrating comprises, for each image, a plurality of recursions each starting with a single node detected in the image, one or more n-edge vectors and one or more visited directions and continuing via a repeated selection of one or more n-closest neighbors to the one or more n-edge vectors.