Method of calibrating a direction of a pan, tilt, zoom, camera with respect to a fixed camera, and a system in which such a calibration is carried out

• US 10,430,972 B2
• Filed: 08/23/2018
• Issued: 10/01/2019
• Est. Priority Date: 08/29/2017
• Status: Active Grant

• Current Assignee: Axis AB (Canon Inc.)
• Original Assignee: Axis AB (Canon Inc.)
• Inventors: Dahlstrom, Robin, Gylin, Mattias
• Primary Examiner(s): Hannett, James M
• Application Number:
  US16/110,535
• Publication Number:
  US 20190066335A1
• Time in Patent Office: 404 Days
• Field of Search: None
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 2207/20164 Salient point detection; Corner detection
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G08B 13/196 using television cameras
• CPC Class Codes: G08B 13/1963 Arrangements allowing camera rotation to change view, e.g. pivoting camera, pan-tilt and zoom [PTZ]
• CPC Class Codes: G08B 29/24 Self-calibration, e.g. compensating for environmental drift or ageing of components
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 23/69 Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04N 23/695 Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

A method of calibrating a pan, tilt, zoom (PTZ) camera with a fixed camera utilizing an overview image of a scene captured by the fixed camera, and an image of the scene captured by the PTZ camera when directed in a first direction. By matching features in the overview image and the PTZ camera image, a first calibration is carried out by correlating the first direction to matching features in the overview image. A mapping between the PTZ camera image and the overview image is defined based on the matching features. The mapping is used to map an object from the PTZ camera image to the overview image. Based on an appearance of the mapped object, a quality of the mapping is calculated. If the quality is not good enough, the PTZ camera is redirected to a second direction, and a further calibration is carried out by again.

Claims

1. A method of calibrating a direction of a pan, tilt, zoom, PTZ, camera with respect to a first, fixed, camera, comprising:

receiving an overview image of a scene captured by a first, fixed, camera,

directing a PTZ camera in a first direction,

when the PTZ camera is in the first direction, performing the steps of;

a) receiving an image of the scene captured by the PTZ camera, wherein a field of view of the image captured by the PTZ camera partly overlaps a field of view the overview image,

b) identifying a first set of features in the image of the scene captured by the PTZ camera,

c) localizing at least a portion of the first set of features in the overview image so as to associate the at least a portion of first set of features in the image captured by the PTZ camera with a second set of features in the overview image,

d) logging positional data of the second set of features in the overview image,

e) defining a mapping between the image captured by the PTZ camera and the overview image based on the at least a portion of first set of features and the second set of features, and calculating a quality of the mapping,

performing a first calibration of the PTZ camera by correlating the first direction of the PTZ camera with the positional data of the second set of features being logged when the PTZ camera is directed in the first direction,

in case the quality of the mapping is below a first threshold;

redirecting the PTZ camera to a second direction,

performing steps a)-d) when the PTZ camera is in the second direction, and

performing a second calibration of the PTZ camera by correlating the second direction of the PTZ camera with positional data of the second set of features being logged when the PTZ camera is directed in the second direction,

wherein step e) further comprises mapping an object in the image captured by the PTZ camera to the overview image by using the defined mapping, wherein calculating the quality of the mapping includes calculating a similarity between an appearance of the object in the image captured by the PTZ camera and an appearance of the object after mapping to the overview image.

2. The method of claim 1, further comprising performing step e) when the PTZ camera is in the second direction, wherein the step of performing a second calibration of the PTZ camera is made on a condition that the quality of the mapping calculated in step e) when the PTZ camera is in the second direction is greater than or equal to the first threshold.

3. The method of claim 1, further comprising:

keep redirecting the PTZ camera to further directions, and repeating steps a)-e) until the quality of the mapping calculated in step e) is greater than or equal to the first threshold.

4. The method of claim 1, further comprising:

identifying an area in the overview image where a density of features in the overview image exceeds a second threshold, and

selecting the second direction on basis of the first calibration of the PTZ camera such that an image captured by the PTZ camera when directed in the second direction covers the identified area in the overview image.

5. The method of claim 1, further comprising:

selecting the second direction on basis of the first calibration of the PTZ camera such that an image captured by the PTZ camera when directed in the second direction covers a center of the overview image.

6. The method of claim 1, wherein, prior to calculating the similarity, the appearance of the object in the image captured by the PTZ camera is adjusted on basis of properties of a lens system of the first camera.

7. The method of claim 1, wherein the first threshold depends on the positional data of the second set of features, such that the first threshold decreases with a distance from a center of the overview image.

8. The method of claim 1, wherein the appearance of an object is at least one of a size of the object and a geometrical shape of the object.

9. The method of claim 1, wherein the object in the image captured by the PTZ camera corresponds to a periphery of the image captured by the PTZ camera.

10. The method of claim 1, wherein a feature includes at least one of an edge or a corner in a captured image of the scene.

11. The method of claim 1, further comprising

capturing a further overview image of the scene using a second, fixed, camera having a known position and direction in relation to the first camera, and

calibrating the PTZ camera with respect to the further overview image of the scene captured by the second camera on basis of the first or second calibration of the first camera and the known position and direction of the second camera in relation to the first camera.

14. A non-transitory computer-readable medium having computer code instructions stored thereon for carrying out the method according to claim 1.

12. A system, comprising:

a first, fixed, camera arranged to capture an overview image of a scene,

a pan, tilt, zoom, PTZ, camera which is separate from the first, fixed camera, and

a controller operatively connected to the first camera and the PTZ camera, the controller being configured to direct PTZ camera in a first direction, and, when the PTZ camera is in the first direction, to perform the steps of;

a) controlling the PTZ camera to capture an image of the scene, wherein a field of view of the image captured by the PTZ camera partly overlaps a field of view the overview image,

b) identifying a first set of features in the image of the scene captured by the PTZ camera,

c) localizing at least a subset of the first set of features in the overview image of the scene so as to associate the at least a subset of the first set of features in the image captured by the PTZ camera with a second set of features in the overview image,

d) logging positional data of the second set of features in the overview image,

e) defining a mapping between the image captured by the PTZ camera and the overview image based on the at least a subset of the first set of features and the second set of features, and calculating a quality of the mapping,

the controller further being configured to;

perform a first calibration of the PTZ camera by correlating the first direction of the PTZ camera with the positional data of the second set of features being logged when the PTZ camera is directed in the first direction,

in case the quality of the mapping is below a first threshold;

redirect the PTZ camera to a second direction,

perform steps a)-d) when the PTZ camera is in the second direction, and

perform a second calibration of the PTZ camera by correlating the second direction of the PTZ camera with positional data of the second set of features being logged when the PTZ camera is directed in the second direction,

wherein step e) further comprises mapping an object in the image captured by the PTZ camera to the overview image by using the defined mapping, wherein calculating the quality of the mapping includes calculating a similarity between an appearance of the object in the image captured by the PTZ camera and an appearance of the object after mapping to the overview image.

13. The system of claim 12, further comprising:

at least one further fixed camera, wherein the first camera and the at least one further fixed camera are directed in different directions so as to capture overview images covering different portions of the scene,

wherein the PTZ camera is mounted in relation to the first camera and the at least one further fixed camera such that the PTZ camera is directable to capture images which overlap an overview image captured by the first camera, and to capture images which overlap an overview image captured by the at least one further fixed camera.