Camera calibration using feature identification

• US 9,286,678 B2
• Filed: 12/28/2011
• Issued: 03/15/2016
• Est. Priority Date: 12/28/2011
• Status: Active Grant

• Current Assignee: Pelco (Schneider Electric SE)
• Original Assignee: Pelco (Schneider Electric SE)
• Inventors: Zhu, Hongwei, Aghdasi, Farzin, Millar, Greg
• Primary Examiner(s): Ortiz Criado, Jorge L
• Assistant Examiner(s): Kir, Albert
• Application Number:
  US13/338,555
• Publication Number:
  US 20130169822A1
• Time in Patent Office: 1,539 Days
• Field of Search: 348/61, 348/169, 348/180, 348/184, 382/103, 382/100, 340/937, 381/103
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 2207/30236 Traffic on road, railway or crossing
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration

Abstract

Disclosed are methods, systems, computer readable media and other implementations, including a method to calibrate a camera that includes capturing by the camera a frame of a scene, identifying features appearing in the captured frame, the features associated with pre-determined values representative of physical attributes of one or more objects, and determining parameters of the camera based on the identified features appearing in the captured frame and the pre-determined values associated with the identified features.

Claims

1. A method to calibrate a camera, the method comprising:

capturing by the camera a frame of a scene;

performing, by a programmable processor, object classification for multiple objects of multiple object types appearing in the captured frame to determine a respective object type for each of the multiple objects;

determining, by the programmable processor, the respective object type for the each of the multiple objects based on geometrical information, including line orientation information;

identifying, by the programmable processor, salient line features, including at least one of;

one or more line pairs that are parallel or at an angle to each other, one or more lines that are on a plane, and one or more lines that are perpendicular to the plane, for the multiple objects of the multiple object types appearing in the captured frame;

assigning, by the programmable processor, each of the identified salient line features a respective pre-determined value representative of an average physical dimension based on the respectively determined object type for the each of the multiple objects appearing in the captured frame;

deriving, by the programmable processor, multiple constraint terms based on the identified salient line features for the multiple objects of the multiple object types appearing in the captured frame and the pre-determined values assigned to the identified salient line features; and

determining, by the programmable processor, parameters of the camera that minimize an error function computed using the multiple constraint terms derived based on the identified salient line features for the multiple objects of the multiple object types appearing in the captured frame.

2. The method of claim 1, wherein identifying the salient line features appearing in the captured frame comprises:

automatically detecting at least one object appearing in one or more frames;

automatically classifying the detected at least one object appearing in the one or more frames; and

automatically identifying features of the classified at least one object, the automatically identified features assigned pre-determined average values representative of physical dimensions for the at least one classified object.

3. The method of claim 2, wherein automatically classifying the detected at least one object comprises:

determining aspect ratios based on dimensions of the at least one detected object;

determining a corresponding object type for the at least one object based on the determined aspect ratios; and

comparing the dimensions of the at least one object to statistical measures stored in a cluster corresponding to the determined object type.

4. The method of claim 3, wherein comparing the dimensions of the at least one object comprises:

updating, based on the dimensions of the at least one object, local models of object size for locations visited by persistently tracked objects; and

updating global clusters by associating local models with the global clusters, the local models having an object size matching that associated with the global cluster.

5. The method of claim 2, wherein the classified at least one object includes one or more of:

a person associated with a pre-determined average height, and a car associated with a pre-determined average length.

6. The method of claim 1, wherein determining the parameters of the camera comprises:

determining one or more of;

a focal length f of the camera, a tilt angle φ

of the camera, a yaw angle β

of the camera, and a height h of the camera above a plane of the scene corresponding to the captured frame.

7. The method of claim 1, further comprising:

removing noisy constraints (outliers) using RANSAC (RANdom SAmple Consensus) procedure.

8. The method of claim 1, wherein the identified salient line features comprise one or more of:

distance between two points in the captured frame of the scene, a vertical line extending from a plane of the captured frame, and two lines extending between two pairs of points in the captured frame with one of the two lines being at an angle relative to the other of the two lines.

9. The method of claim 8, wherein the distance between the two points in the captured frame comprises length of a car in the captured frame of the scene, the car associated with an approximate pre-determined length.

10. The method of claim 8, wherein the vertical line extending from the plane of the scene comprises height of a person in the scene, the person associated with an approximate pre-determined height.

11. The method of claim 1, further comprising:

determining attributes of objects located in the scene based on camera coordinates of images of the objects appearing in one or more captured frames, and based on the determined parameters of the camera.

12. The method of claim 11, wherein the determined attributes of the objects comprise one or more of:

spatial position in the scene of at least one of the objects, and length of the at least one of the objects in a particular direction.

21. The method of claim 1, further comprising:

determining context associated with the captured frame;

selecting, based on the determined context, the respective pre-determined value, for the each of the identified salient line features, representative of the average physical dimension for the object type respectively determined for the each of the multiple objects appearing in the captured frame; and

assigning to the each of the identified salient line features the respective pre-determined value representative of the average physical dimension for the object type respectively determined for the each of the multiple objects of the multiple object types appearing in the captured frame.

13. A system comprising:

at least one camera to capture frame data; and

one or more processors configured to perform operations to calibrate the at least one camera, the operations comprising;

capturing by the at least one camera a frame of a scene;

performing object classification for multiple objects of multiple object types appearing in

the captured frame to determine a respective object type for each of the multiple objects;

determining the respective object type for the each of the multiple objects based on geometrical information, including line orientation information;

identifying salient line features, including at least one of;

one or more line pairs that are parallel or at an angle to each other, one or more lines that are on a plane, and one or more lines that are perpendicular to the plane, for the multiple objects of the multiple object types appearing in the captured frame;

assigning each of the identified salient line features a respective pre-determined value representative of an average physical dimension based on the respectively determined object type for the each of the multiple objects appearing in the captured frame;

deriving multiple constraint terms based on the identified salient line features for the multiple objects of the multiple object types appearing in the captured frame and the pre-determined values assigned to the identified salient line features; and

determining parameters of the camera that minimize an error function computed using the multiple constraint terms derived based on the identified salient line features for the multiple objects of the multiple object types appearing in the captured frame.

14. The system of claim 13, wherein the one or more processor configured to perform the operations of identifying the salient line features appearing in the captured frame are configured to perform operations comprising:

automatically detecting at least one object appearing in one or more frames;

automatically classifying the detected at least one object appearing in the one or more frames; and

automatically identifying features of the classified at least one object, the automatically identified features assigned pre-determined average values representative of physical dimensions for the at least one classified object.

15. The system of claim 14, wherein the one or more processors configured to perform the operations of automatically classifying the detected at least one object are configured to perform operations comprising:

determining aspect ratios based on dimensions of the at least one detected object;

determining a corresponding object type for the at least one object based on the determined aspect ratios; and

comparing the dimensions of the at least one object to statistical measures stored in a cluster corresponding to the determined object type.

16. The system of claim 14, wherein the classified at least one object includes one or more of:

a person associated with a pre-determined average height, and a car associated with a pre-determined average length.

17. The system of claim 13, wherein the one or more processors configured to perform the operations of determining the parameters of the camera are configured to perform operations comprising:

determining one or more of;

a focal length f of the camera, a tilt angle φ

of the camera, a yaw angle β

of the camera, and a height h of the camera above a plane of the scene corresponding to the captured frame.

18. A non-transitory computer readable media programmed with a set of computer instructions executable on a processor that, when executed, cause operations comprising:

capturing by a camera a frame of a scene;

performing object classification for multiple objects of multiple object types appearing in the captured frame to determine a respective object type for each of the multiple objects;

determining the respective object type for the each of the multiple objects based on geometrical information, including line orientation information;

identifying salient line features, including at least one of;

one or more line pairs that are parallel or at an angle to each other, one or more lines that are on a plane, and one or more lines that are perpendicular to the plane, for the multiple objects of the multiple object types appearing in the captured frame;

assigning each of the identified salient line features a respective pre-determined value representative of an average physical dimension based on the respectively determined object type for the each of the multiple objects appearing in the captured frame;

deriving multiple constraint terms based on the identified salient line features for the multiple objects of the multiple object types appearing in the captured frame and the pre-determined values assigned to the identified salient line features; and

determining parameters of the camera that minimize an error function computed using the multiple constraint terms derived based on the identified salient line features for the multiple objects of the multiple object types appearing in the captured frame.

19. The computer readable media of claim 18, wherein the computer instructions that cause the operations of identifying the salient line features appearing in the captured frame comprise instructions to cause operations comprising:

automatically detecting at least one object appearing in one or more frames;

automatically classifying the detected at least one object appearing in the one or more frames; and

automatically identifying features of the classified at least one object, the automatically identified features assigned pre-determined average values representative of physical dimensions for the at least one classified object.

20. The computer readable media of claim 18, wherein the computer instructions that cause the operations of determining the parameters of the camera comprise instructions that cause operations comprising:

determining one or more of;

a focal length f of the camera, a tilt angle φ

of the camera, a yaw angle β

of the camera, and a height h of the camera above a plane of the scene corresponding to the captured frame.