Method and system for calibrating surveillance cameras

• US 9,684,962 B2
• Filed: 07/30/2014
• Issued: 06/20/2017
• Est. Priority Date: 12/10/2013
• Status: Active Grant

• Current Assignee: Tsinghua University
• Original Assignee: Tsinghua University
• Inventors: Zhou, Jie, Deng, Lei, Yu, Chuan, Zhang, Haoyang
• Primary Examiner(s): Akhavannik, Hadi
• Application Number:
  US14/765,748
• Publication Number:
  US 20150371385A1
• Time in Patent Office: 1,056 Days
• Field of Search: None
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A method and a system for calibrating a plurality of surveillance cameras are provided. The method includes: sampling a plurality of first images in a monitoring scene of the plurality of surveillance cameras by a sampling device and sampling a second image by each of the plurality of surveillance cameras; performing a feature matching on the plurality of first images to obtain a matching result between the plurality of first images; reconstructing a three-dimensional point set of the monitoring scene according to the matching result between the plurality first of images; and calculating parameters of each of the plurality of surveillance cameras according to the three-dimensional point set and the second image sampled by each of the plurality of surveillance cameras.

Claims

1. A method for calibrating a plurality of surveillance cameras, comprising:

sampling a plurality of first images in a monitoring scene of the plurality of surveillance cameras by a sampling device and sampling a second image in the monitoring scene by each of the plurality of surveillance cameras;

performing a feature matching on the plurality of first images to obtain a matching result between the plurality of first images;

reconstructing a three-dimensional point set of the monitoring scene according to the matching result between the plurality of first images; and

calculating parameters of each of the plurality of surveillance cameras according to the three-dimensional point set and the second image sampled by each of the plurality of surveillance cameras,

wherein reconstructing a three-dimensional point set of the monitoring scene according to the matching result between the plurality of first images comprises;

selecting two first images from the plurality of first images as initial images, in which a number of matching point pairs on the initial images is larger than a predetermined threshold and a degradation degree between the initial images is smaller than a predetermined degree;

calculating camera matrices corresponding to the initial images according to the matching point pairs on the initial images;

reconstructing an initial three-dimensional point set of the monitoring scene according to the matching relationship between the initial images;

expanding the three-dimensional point set of the monitoring scene according to the initial three-dimensional point set and other first images except for the initial images,

or, wherein reconstructing a three-dimensional point set of the monitoring scene according to the matching result between the plurality of first images comprises;

obtaining GPS location information and gesture information corresponding to each of the plurality of first images and calculating the camera matrix corresponding to each of the plurality of first images according to the GPS location information, the gesture information and the intrinsic parameters of the sampling device;

for each two adjacent first images, calculating three-dimensional coordinates of matching points on the two adjacent first images according to the camera matrices corresponding to the two adjacent first images by means of a triangulation algorithm;

optimizing the camera matrices and the three-dimensional coordinates of the matching points by means of a bundle adjustment method.

2. The method according to claim 1, wherein sampling a plurality of first images in a monitoring scene of the plurality of surveillance cameras by a sampling device comprises:

calibrating intrinsic parameters of the sampling device;

sampling the plurality of first images in the monitoring scene of the plurality of surveillance cameras by the sampling device with calibrated intrinsic parameters, in which at least some of the plurality of first images have an area overlapped with a total viewing field of the plurality of surveillance cameras and larger than a predetermined area threshold.

3. The method according to claim 1, wherein performing a feature matching on the plurality of first images to obtain a matching result between the plurality of first images comprises:

when the sampling device has a GPS sensor, obtaining first images adjacent to each other according to GPS location information obtained during sampling the plurality of first images, and matching the adjacent first images with each other; and

when the sampling device has no GPS sensor, matching the plurality of first images with each other.

4. The method according to claim 1, wherein expanding the three-dimensional point set of the monitoring scene according to the initial three-dimensional point set and other first images except for the initial images comprises:

determining whether each of the plurality of first images has been used to reconstruct the three-dimensional point set of the monitoring scene;

if no, finding out a first image at most matched with the reconstructed three-dimensional point set from images having not been used to reconstruct the three-dimensional point set of the monitoring scene, and calculating a camera matrix corresponding to the first image at most matched with the reconstructed three-dimensional point set;

expanding the three-dimensional point set of the monitoring scene according to matching point pairs between the first image at most matched with the reconstructed three-dimensional point set and other first images with calculated camera matrices;

optimizing the calculated camera matrices and three-dimensional coordinates of points in the three-dimensional point set.

5. The method according to claim 1, wherein calculating parameters of each of the plurality of surveillance cameras according to the three-dimensional point set and the second image sampled by each of the plurality of surveillance cameras comprises:

for each surveillance camera to be calculated, matching the sampled second image with the first images used to reconstruct the three-dimensional point set of the monitoring scene respectively to obtain a first feature point set, and calculating an intersection set between the first feature point set and a feature point set corresponding to the three-dimensional point set, in which the feature point set corresponding to the three-dimensional point set comprises points located on the first images used to reconstruct the three-dimensional point set and corresponding to points in the three-dimensional point set;

determining whether a number of feature points in the intersection set is larger than a predetermined number;

if no, sampling at least one third image in the monitoring scene of the plurality of surveillance cameras by the sampling device and expanding the three-dimensional point set according to the at least one third image, so as to obtain an updated intersection set in which the number of feature points is larger than the predetermined number;

calculating the parameters of the surveillance camera to be calibrated according to the updated intersection set and a three-dimensional point set corresponding to the updated intersection set.

6. A system for calibrating a plurality of surveillance cameras, comprising:

at least one sampling device, configured to sample a plurality of first images in a monitoring scene of the plurality of surveillance cameras; and

a computing device, configured to perform a feature matching on the plurality of first images so as to obtain a matching result between the plurality of first images, to reconstruct a three-dimensional point set of the monitoring scene of the plurality of surveillance cameras according to the matching result between the plurality of first images, and to calculate parameters of each of the plurality of surveillance cameras according to the three-dimensional point set and a second image sampled by each of the plurality of surveillance cameras,

wherein the computing device is configured to;

select two images from the plurality of first images as initial images, in which a number of matching point pairs on the initial images is larger than a predetermined threshold and a degradation degree between the initial images is smaller than a predetermined degree;

calculate camera matrices corresponding to the initial images according to the matching point pairs on the initial images;

reconstruct an initial three-dimensional point set of the monitoring scene of the plurality of surveillance cameras according to the matching relationship between the initial images and the camera matrices corresponding to the initial images; and

expand the three-dimensional point set of the monitoring scene according to the initial three-dimensional point set and other first images except for the initial images,

or, wherein the computing device is configured to;

obtain GPS location information and gesture information corresponding to each of the plurality of first images and calculate the camera matrix corresponding to each of the plurality of first images according to the GPS location information, the gesture information and the intrinsic parameters of the at least one sampling device;

calculate three-dimensional coordinates of matching points on each two adjacent first images according to the camera matrices corresponding to the two adjacent first images by means of a triangulation algorithm; and

optimize the camera matrices and the three-dimensional coordinates of the matching points by means of a bundle adjustment method.

7. The system according to claim 6, wherein intrinsic parameters of the at least one sampling device are calibrated, and the at least one sampling device with calibrated intrinsic parameters actively samples the plurality of first images in the monitoring scene of the plurality of surveillance cameras, in which at least some of the plurality of first images have an area overlapped with a total viewing field of the plurality of surveillance cameras and larger than a predetermined area threshold.

8. The system according to claim 6, wherein the computing device is further configured to:

determine whether each of the plurality of first images has been used to reconstruct the three-dimensional point set of the monitoring scene;

if no, find out a first image at most matched with the reconstructed three-dimensional point set from images having not been used to reconstruct the three-dimensional point set of the monitoring scene, and calculate a camera matrix corresponding to the first image at most matched with the reconstructed three-dimensional point set;

expand the three-dimensional point set of the monitoring scene according to matching point pairs between the first image at most matched with the reconstructed three-dimensional point set and other first images with calculated camera matrices; and

optimize the calculated camera matrices and three-dimensional coordinates of points in the three-dimensional point set.

9. The system according claim 6, wherein the computing device is further configured to:

match the second image sampled by each of the plurality of surveillance cameras with the first images used to reconstruct the three-dimensional point set of the monitoring scene respectively to obtain a first feature point set, and calculate an intersection set between the first feature point set and a feature point set corresponding to the three-dimensional point set, in which the feature point set corresponding to the three-dimensional point set comprises points located on the first images used to reconstruct the three-dimensional point set and corresponding to points in the three-dimensional point set;

determine whether a number of feature points in the intersection set is bigger than a predetermined number;

if no, receive at least one third image sampled by the at least one sampling device and expand the three-dimensional point set according to the at least one third image so as to obtain an updated intersection set in which the number of feature points is bigger than the predetermined number; and

calculate the parameters of the surveillance camera according to the updated intersection set and a three-dimensional point set corresponding to the updated intersection set.

10. A non-transitory computer-readable storage medium having stored therein instructions that, when executed by a processor of an apparatus, causes the apparatus to perform a method for calibrating a plurality of surveillance cameras, the method comprising:

sampling a plurality of first images in a monitoring scene of the plurality of surveillance cameras by a sampling device and sampling a second image in the monitoring scene by each of the plurality of surveillance cameras;

performing a feature matching on the plurality of first images to obtain a matching result between the plurality of first images;

reconstructing a three-dimensional point set of the monitoring scene according to the matching result between the plurality of first images; and

calculating parameters of each of the plurality of surveillance cameras according to the three-dimensional point set and the second image sampled by each of the plurality of surveillance cameras,

wherein reconstructing a three-dimensional point set of the monitoring scene according to the matching result between the plurality of first images comprises;

selecting two first images from the plurality of first images as initial images, in which a number of matching point pairs on the initial images is larger than a predetermined threshold and a degradation degree between the initial images is smaller than a predetermined degree;

calculating camera matrices corresponding to the initial images according to the matching point pairs on the initial images;

reconstructing an initial three-dimensional point set of the monitoring scene according to the matching relationship between the initial images;

expanding the three-dimensional point set of the monitoring scene according to the initial three-dimensional point set and other first images except for the initial images,

or, wherein reconstructing a three-dimensional point set of the monitoring scene according to the matching result between the plurality of first images comprises;

obtaining GPS location information and gesture information corresponding to each of the plurality of first images and calculating the camera matrix corresponding to each of the plurality of first images according to the GPS location information, the gesture information and the intrinsic parameters of the sampling device;

for each two adjacent first images, calculating three-dimensional coordinates of matching points on the two adjacent first images according to the camera matrices corresponding to the two adjacent first images by means of a triangulation algorithm;

optimizing the camera matrices and the three-dimensional coordinates of the matching points by means of a bundle adjustment method.