Monitoring method and camera

• US 9,648,285 B2
• Filed: 06/13/2014
• Issued: 05/09/2017
• Est. Priority Date: 06/14/2013
• Status: Active Grant

• Current Assignee: Axis AB (Canon Inc.)
• Original Assignee: Axis AB (Canon Inc.)
• Inventors: Traff, Gustav, Hamida, Anis Ben, Pendse, Mikael
• Primary Examiner(s): Luo, Kate
• Application Number:
  US14/304,284
• Publication Number:
  US 20140368646A1
• Time in Patent Office: 1,061 Days
• CPC Class Codes: G06V 10/143 Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G08B 13/19686 Interfaces masking personal details for privacy, e.g. blurring faces, vehicle license plates
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A scene is monitored by receiving image data representing a thermal image captured by a thermal image sensor (13) and a visual image captured by a visual image sensor (15), the thermal image and the visual image depicting a same view of the scene. Objects (9) are detected in the visual image and in the thermal image. A mask (17a) is defined in the view of the scene, and it is determined if an object (9) or an object part, inside the mask (17a), is present in both the thermal image and the visual image. If so, a modified mask (17b) is provided by excluding an area occupied by the object (9) or the object part from the mask (17a).

Claims

1. A method of monitoring a scene, the method comprising:

receiving image data representing a thermal image captured by a thermal image sensor and a visual image captured by a visual image sensor, the thermal image and the visual image depicting a same view of the scene;

detecting objects which have a different temperature than their environment in the visual image and in the thermal image;

defining a mask covering a window in the view of the scene; and

matching the detected objects between the visual image and the thermal image in order to determine if an object or an object part, inside the mask, is present in both the thermal image and the visual image, and, if so, providing a modified mask by excluding an area occupied by the object or the object part from the mask, wherein

matching the detected objects between the thermal and the visual image in order to determine if an object or object part, inside the mask, is present in both the thermal image and the visual image comprises;

for pixels in the thermal image where an object is detected, checking if an object is detected in corresponding pixels in the visual image;

counting the number of pixels where there is a match for the object;

determining a ratio between the number of matching pixels and a total number of pixels belonging to the object detected in the thermal image; and

comparing the ratio to a threshold value, and if the ratio is below the threshold value, determining that the object only is present in the thermal image.

2. The method of claim 1, wherein

matching the detected objects between the visual image and the thermal image comprises comparing object tracking histories.

3. The method of claim 1, wherein

the mask is a privacy mask, and

the method further comprises displaying the visual image, wherein image data inside the modified mask is hidden from being displayed.

4. The method of claim 1, further comprising

setting imaging parameters for the visual image based on image data of the visual image, wherein image data inside the modified mask is excluded.

5. The method of claim 1, further comprising:

adjusting camera settings for the visual image sensor based on image data of the visual image, wherein image data inside the modified mask is excluded.

6. The method of claim 1, further comprising:

compressing the image data of the visual image inside the modified mask at a first compression ratio; and

compressing the remaining image data of the visual image at a second compression ratio which is lower than the first compression ratio.

7. The method of claim 1, wherein

determining if an object or object part, inside the mask, is present in both the thermal image and the visual image comprises comparing object feature descriptors or specific feature points from a scale-invariant feature transform (SIFT) or speeded-up robust features (SURF) algorithm.

8. A method of monitoring a scene, the method comprising:

receiving image data representing a thermal image captured by a thermal image sensor and a visual image captured by a visual image sensor, the thermal image and the visual image depicting a same view of the scene;

detecting objects which have a different temperature than their environment in the visual image and in the thermal image;

defining a mask covering a window in the view of the scene; and

matching the detected objects between the visual image and the thermal image in order to determine if an object or an object part, inside the mask, is present in both the thermal image and the visual image, and, if so, providing a modified mask by excluding an area occupied by the object or the object part from the mask, wherein

matching the detected objects between the visual image and the thermal image in order to determine if an object or part of an object, inside the mask, is present in both the thermal image and the visual image comprises;

computing gradients of the thermal image and the visual image;

for the pixels in the thermal image that have a gradient magnitude above a magnitude threshold value, comparing the gradient in each pixel to the gradient of corresponding pixels in the visual image, and assigning a match score to each pixel depending on degree of correspondence between the gradients; and

for each object, determining a sum of the match scores in the thermal image, normalizing the sum by dividing by the total number of pixels belonging to the object, and if the sum for an object is below a match score threshold value, determining that the object only is present in the thermal image.

9. An image processing system comprising:

a camera configured to acquire image data representing a thermal image captured by a thermal image sensor and a visual image captured by a visual image sensor, the thermal image and the visual image depicting a same view of the scene; and

a video processor including hardware configured to execute software stored by a non-transitory computer-readable medium to

detect objects which have a different temperature than their environment in the visual image and in the thermal image;

define a mask covering a window in the view of the scene; and

match the detected objects between the visual image and the thermal image in order to determine if an object or an object part, inside the mask, is present in both the thermal image and the visual image, and, if so, providing a modified mask by excluding an area occupied by the object or the object part from the mask, wherein

matching the detected objects between the thermal and the visual image in order to determine if an object or object part, inside the mask, is present in both the thermal image and the visual image comprises;

for pixels in the thermal image where an object is detected, checking if an object is detected in corresponding pixels in the visual image;

counting the number of pixels where there is a match for the object;

determining a ratio between the number of matching pixels and a total number of pixels belonging to the object detected in the thermal image; and

comparing the ratio to a threshold value, and if the ratio is below the threshold value, determining that the object only is present in the thermal image.

10. The image system of claim 9, wherein the image processing system is the camera including the thermal image sensor and the visual image sensor.

11. The image processing system of claim 9, wherein

the image processing system is a video encoder including a communication interface configured to receive the image data from the camera including the thermal image sensor and the visual image sensor.

12. The image processing system of claim 9, wherein

the video processor is configured to match the detected objects between the visual image and the thermal image by comparing object tracking histories.

13. The image processing system of claim 9, wherein

the mask is a privacy mask, and

the video processor is configured to control displaying the visual image, wherein image data inside the modified mask is hidden from being displayed.

14. The image processing system of claim 9, wherein

the video processor is configured to set imaging parameters for the visual image based on image data of the visual image, wherein image data inside the modified mask is excluded.

15. The image processing system of claim 9, wherein

the video processor is configured to adjust camera settings for the visual image sensor based on image data of the visual image, wherein image data inside the modified mask is excluded.

16. The image processing system of claim 9, wherein the video processor is configured to:

compress the image data of the visual image inside the modified mask at a first compression ratio; and

compress the remaining image data of the visual image at a second compression ratio which is lower than the first compression ratio.

17. The image processing system of claim 9, wherein

determining if an object or object part, inside the mask, is present in both the thermal image and the visual image includes comparing object feature descriptors or specific feature points from a scale-invariant feature transform (SIFT) or speeded-up robust features (SURF) algorithm.