Automatic camera ground plane calibration method and system

• US 10,643,078 B2
• Filed: 11/06/2017
• Issued: 05/05/2020
• Est. Priority Date: 11/06/2017
• Status: Active Grant

• Current Assignee: Johnson Controls Tyco IP Holdings LLP (Johnson Controls International plc)
• Original Assignee: Sensormatic Electronics LLC (Johnson Controls International plc)
• Inventors: Gottumukkal, Rajkiran Kumar, Westmacott, Ian C.
• Primary Examiner(s): Azima, Shaghayegh
• Application Number:
  US15/803,988
• Publication Number:
  US 20190138818A1
• Time in Patent Office: 911 Days
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/62 of area, perimeter, diameter or volume
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 10/457 by analysing connectivity, e.g. edge linking, connected component analysis or slices
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

A surveillance camera system and method is disclosed. The system includes one or more surveillance cameras that capture images of scenes, and one or more calibration systems that automatically generate ground planes from the captured images from the surveillance cameras. Foreground objects in the scenes are then analyzed against the ground planes to determine whether the ground planes may require updating and/or recalculation.

Claims

1. A surveillance camera system, comprising:

one or more surveillance cameras that capture images of scenes; and

one or more calibration systems that automatically generate ground planes from the captured images from the surveillance camera;

a video analytics system having an object detection and tracking system that tracks foreground objects within the scenes, wherein the object detection and tracking system generates object tracking information, object classification information, and/or object size information for each of the foreground objects;

wherein the calibration systems determine whether ambiguities in the foreground objects exist relative to the ground planes, the ambiguities in the foreground objects suggesting errors in the ground planes, and wherein the calibration systems modify the ground planes to address the errors in the ground planes with reference to the object tracking information, object classification information, and/or object size information for the foreground objects found to have ambiguities.

2. The system of claim 1, wherein each of the calibration systems includes a calibrator module that automatically generates the ground planes from the captured images by:

executing trained weights of a machine learning algorithm against one or more of the captured images to obtain depth images for the captured images, wherein the depth images estimate depth for pixels within the captured images; and

creating the ground planes from the captured images and the depth images for the captured images.

3. The system of claim 1, wherein the one or more surveillance cameras include the one or more calibration systems.

4. The system of claim 1, wherein each of the calibration systems includes an ambiguity detector module that compares foreground objects against the ground planes to determine ranges and estimated heights or sizes of the foreground objects, and that detects ambiguities in the foreground objects that suggest errors in the ground planes based upon the ranges and the estimated heights or sizes of the foreground objects.

5. The system of claim 4, wherein the ambiguity detector module concludes ambiguities in the foreground objects when the estimated heights or sizes of the foreground objects change as the foreground objects move across the scene.

6. The system of claim 4, wherein the ambiguity detector module concludes ambiguities in the foreground objects when estimated heights of the foreground objects are determined to be larger or smaller than expected.

7. The system of claim 1, wherein the calibration systems modify the ground planes with reference to bounding boxes and trajectory information of the object tracking information for the foreground objects found to have ambiguities.

8. The system of claim 1, wherein the calibration systems modify the ground planes by moving the ground planes up and/or down relative to a Y axis of the ground planes.

9. The system of claim 1, wherein the calibration systems modify the ground planes by rotating the ground planes around an X and/or a Z axis of the ground planes.

10. A method for configuring a surveillance camera system, the method comprising:

one or more surveillance cameras capturing images of scenes;

automatically generating ground planes from the captured images from the surveillance cameras;

tracking foreground objects within the scenes;

generating object tracking information, object classification information, and/or object size information for each of the foreground objects;

determining whether ambiguities in the foreground objects exist relative to the ground planes, the ambiguities in the foreground objects suggesting errors in the ground planes; and

modifying the ground planes to address the errors in the ground planes with reference to the object tracking information, object classification information, and/or object size information for the foreground objects found to have ambiguities.

11. The method of claim 10, wherein automatically generating the ground planes from the captured images comprises:

executing trained weights of a machine learning algorithm against one or more of the captured images to obtain depth images for the captured images, wherein the depth images estimate depth of pixels within the captured images; and

creating the ground planes from the captured images and the depth images for the captured images.

12. The method of claim 11, further comprising training the machine learning algorithm with reference 2D image and corresponding ground truth depth image information, prior to executing the trained weights of the machine learning algorithm against the one or more of the captured images.

13. The method of claim 10, further comprising determining ambiguities in foreground objects relative to the ground planes.

14. The method of claim 13, wherein determining the ambiguities in the foreground objects relative to the ground planes comprises determining that estimated heights or sizes of the foreground objects have changed with movement of the foreground objects.

15. The method of claim 13, wherein determining the ambiguities in the foreground objects relative to the ground planes comprises determining that the estimated heights of the foreground objects are larger or smaller than expected.

16. The method of claim 10, further comprising the object tracking information including bounding boxes and trajectory information.

17. The method of claim 10, wherein modifying the ground planes comprises moving the ground planes up and/or down relative to a Y axis of the ground planes.