Image stabilization techniques for video surveillance systems

• US 9,674,442 B2
• Filed: 01/05/2016
• Issued: 06/06/2017
• Est. Priority Date: 11/12/2012
• Status: Active Grant

• Current Assignee: Intellective Ai, Inc.
• Original Assignee: Omni AI, Inc.
• Inventors: Cobb, Wesley Kenneth, Yang, Tao, Saitwal, Kishor Adinath
• Primary Examiner(s): Henn, Timothy J
• Application Number:
  US14/988,475
• Publication Number:
  US 20160134812A1
• Time in Patent Office: 518 Days
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera

Abstract

A behavioral recognition system may include both a computer vision engine and a machine learning engine configured to observe and learn patterns of behavior in video data. Certain embodiments may provide image stabilization of a video stream obtained from a camera. An image stabilization module in the behavioral recognition system obtains a reference image from the video stream. The image stabilization module identifies alignment regions within the reference image based on the regions of the image that are dense with features. Upon determining that the tracked features of a current image is out of alignment with the reference image, the image stabilization module uses the most feature dense alignment region to estimate an affine transformation matrix to apply to the entire current image to warp the image into proper alignment.

Claims

1. A method for stabilizing images captured by a recording device, comprising:

ranking a plurality of alignment regions in a reference image based on a density of tracked features in each of the alignment regions;

receiving a current image captured by the recording device;

determining an alignment of the current image relative to the reference image based on a plurality of tracked features of each of the current image and the reference image, wherein the reference image is captured by the recording device previous to the capturing of the current image; and

upon determining, based on the alignment, that the current image is unstable relative to the reference image, stabilizing the current image relative to the reference image by aligning at least one or more of the highest ranking alignment regions of the reference image with corresponding regions of the current image.

2. The method of claim 1, further comprising, prior to receiving the current image captured by the recording device:

detecting one or more rectangular alignment regions in the reference image.

3. The method of claim 2, wherein the alignment identifies affine transformations in the at least one of the tracked features of the current image.

4. The method of claim 3, wherein stabilizing the current image relative to the reference image further comprises:

designating one of the detected rectangular alignment regions for alignment;

estimating an affine transformation matrix for the current image based on the designated rectangular alignment region; and

transforming each of the at least one of the tracked features of the current image from a tracked position in the current image to an aligned position in the reference image.

5. The method of claim 2, further comprising:

assigning a statistical weight to the designated rectangular alignment region; and

modifying the statistical weight based on whether the stabilized current image and the reference image are within a specified tolerance.

6. The method of claim 1, wherein the stabilized current image includes a plurality of foreground components and a plurality of background components.

7. The method of claim 6, further comprising:

separating the foreground components from the background components in the stabilized current image.

8. A non-transitory computer-readable storage medium storing instructions, which, when executed on a processor, perform an operation for stabilizing images captured by a recording device, the operation comprising:

ranking a plurality of alignment regions in a reference image based on a density of tracked features in each of the alignment regions;

receiving a current image captured by the recording device;

determining an alignment of the current image relative to the reference image based on a plurality of tracked features of each of the current image and the reference image, wherein the reference image is captured by the recording device previous to the capturing of the current image; and

upon determining, based on the alignment, that the current image is unstable relative to the reference image, stabilizing the current image relative to the reference image by aligning at least one or more of the highest ranking alignment regions of the reference image with corresponding regions of the current image.

9. The non-transitory computer-readable storage medium of claim 8, wherein the operation further comprises, prior to receiving the current image captured by the recording device:

detecting one or more rectangular alignment regions in the reference image.

10. The non-transitory computer-readable storage medium of claim 9, wherein the alignment identifies affine transformations in the at least one of the tracked features of the current image.

11. The non-transitory computer-readable storage medium of claim 10, wherein stabilizing the current image relative to the reference image further comprises:

designating one of the detected rectangular alignment regions for alignment;

estimating an affine transformation matrix for the current image based on the designated rectangular alignment region; and

transforming each of the at least one of the tracked features of the current image from a tracked position in the current image to an aligned position in the reference image.

12. The non-transitory computer-readable storage medium of claim 9, wherein the operation further comprises:

assigning a statistical weight to the designated rectangular alignment region; and

modifying the statistical weight based on whether the stabilized current image and the reference image are within a specified tolerance.

13. The non-transitory computer-readable storage medium of claim 8, wherein the stabilized current image includes a plurality of foreground components and a plurality of background components.

14. The non-transitory computer-readable storage medium of claim 13, wherein the operation further comprises:

separating the foreground components from the background components in the stabilized current image.

15. A system, comprising:

a processor; and

a memory storing program code, which, when executed on the processor, performs an operation for stabilizing images captured by a recording device, the operation comprising;

ranking a plurality of alignment regions in a reference image based on a density of tracked features in each of the alignment regions,

receiving a current image captured by the recording device,

determining an alignment of the current image relative to the reference image based on a plurality of tracked features of each of the current image and the reference image, wherein the reference image is captured by the recording device previous to the capturing of the current image, and

upon determining, based on the alignment, that the current image is unstable relative to the reference image, stabilizing the current image relative to the reference image by aligning at least one or more of the highest ranking alignment regions of the reference image with corresponding regions of the current image.

16. The system of claim 15, wherein the operation further comprises, prior to receiving the current image captured by the recording device:

detecting one or more rectangular alignment regions in the reference image.

17. The system of claim 16, wherein the alignment identifies affine transformations in the at least one of the tracked features of the current image.

18. The system of claim 17, wherein stabilizing the current image relative to the reference image further comprises:

designating one of the detected rectangular alignment regions for alignment;

estimating an affine transformation matrix for the current image based on the designated rectangular alignment region; and

transforming each of the at least one of the tracked features of the current image from a tracked position in the current image to an aligned position in the reference image.

19. The system of claim 16, wherein the operation further comprises:

assigning a statistical weight to the designated rectangular alignment region; and

modifying the statistical weight based on whether the stabilized current image and the reference image are within a specified tolerance.

20. The system of claim 19, wherein the stabilized current image includes a plurality of foreground components and a plurality of background components.