Methods and systems for stabilizing live video in the presence of long-term image drift

• US 8,947,529 B2
• Filed: 05/05/2014
• Issued: 02/03/2015
• Est. Priority Date: 09/30/2011
• Status: Expired due to Fees

• Current Assignee: Siemens Schweiz AG (Siemens AG)
• Original Assignee: Siemens Industry Incorporated (Siemens AG)
• Inventors: Strine, Lloyd, Bragg, Jeffrey, Bobick, Aaron
• Primary Examiner(s): Kim, Hee-Yong
• Application Number:
  US14/269,600
• Publication Number:
  US 20140240496A1
• Time in Patent Office: 274 Days
• Field of Search: 348/143
• US Class Current: 348/143
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

Methods and systems stabilization of a camera image for short term or ‘pole shake’ and longer term ‘pole drift’ are provided. The camera is attached to a fixed structure. The pole drift is over periods of times long enough that the imagery can change significantly between a base reference frame and the current stabilized transient frame. A multitude of mapped frames and associated information is maintained. The slowly varying camera orientation (“pole drift”) is decoupled from the rapid motion jitter (“pole shake”), and separate transformations are calculated for each. Up to three transformations may be combined together for any one input frame. The separate transformations are combined together, so that only one full-blown image transformation computation is performed for each frame. A surveillance system applies a stabilization method.

Claims

1. A method for surveillance with a camera that is attached to a structure in a substantially fixed position relative to a ground of an environment, generating a plurality of frames, comprising:

obtaining with the camera a base frame of a scene containing one or more objects and a processor establishing a reference coordinate system, wherein the base frame includes a plurality of detectable features;

the processor storing in a memory a table that assigns pixel coordinates in the base frame to world coordinates in the scene;

the processor selecting a transient frame from the plurality of frames, establishing a transient coordinate system and determining a transformation that transforms the transient coordinate system to the reference coordinate system;

the processor transforming an incoming frame from the camera to the reference coordinate system via the transient coordinate system; and

the processor determining world coordinates of the one or more objects in the incoming frame.

2. The method of claim 1, wherein the camera is subject to pole shake and pole drift.

3. The method of claim 1, wherein the camera is subject to pole drift.

4. The method of claim 1, wherein the processor determines that pixels representing the one or more objects in the incoming frame that have moved relative to previous frames of the scene represent the one or more objects that have not moved in the scene.

5. The method of claim 2, wherein pole shake includes a movement of the camera with a frequency between 0.1 and 10 Hz.

6. The method of claim 1, wherein the transforming by the processor of the incoming frame to the reference coordinate system via the transient coordinate system is performed by a single transformation that combines a transformation from the incoming frame to the transient coordinate system and a transformation from the transient coordinate system to the reference coordinate system.

7. The method of claim 1, wherein the processor determines the incoming frame to be a new transient frame to determine a new transient coordinate system.

8. The method of claim 1, wherein the structure experiences movement which is not larger than a field-of-vision of the camera.

9. The method of claim 1, wherein each transformation is a parameterized image-based transformation and not a camera motion parameter.

10. The method of claim 2, wherein the transient frame is unbiased with respect to pole shake.

11. A surveillance system, comprising:

a camera that is in a attached to a structure that is in a substantially fixed position relative to a ground of an environment to generate a plurality of frames of video;

a memory to store and retrieve data, including instructions;

a processor in communication with the memory to retrieve instructions to perform the steps;

processing a base frame of a scene containing one or more objects to establish a reference coordinate system, wherein the base frame includes a plurality of detectable features;

storing in the memory a table that assigns pixel coordinates in the base frame to world coordinates in the scene;

selecting a transient frame from the plurality of frames to establish a transient coordinate system and determining a transformation that transforms the transient coordinate system to the reference coordinate system;

transforming an incoming frame from the camera to the reference coordinate system via the transient coordinate system; and

determining world coordinates of the one or more objects in the incoming frame.

12. The surveillance system of claim 11, wherein the camera is subject to pole shake and pole drift.

13. The surveillance system of claim 11, wherein the camera is subject to pole drift.

14. The surveillance system of claim 11, wherein the processor determines that pixels representing the one or more objects in the incoming frame that have moved relative to previous frames of the scene represent the one or more objects that have not moved in the scene.

15. The surveillance system of claim 12, wherein pole shake includes a movement of the camera with a frequency between 0.1 and 10 Hz.

16. The surveillance system of claim 11, wherein the transforming by the processor of the incoming frame to the reference coordinate system via the transient coordinate system is performed by a single transformation that combines a transformation from the incoming frame to the transient coordinate system and a transformation from the transient coordinate system to the reference coordinate system.

17. The surveillance system of claim 11, wherein the processor determines the incoming frame to be a new transient frame to determine a new transient coordinate system.

18. The surveillance system of claim 11, wherein the structure experiences movement which is not larger than a field-of-vision of the camera.

19. The surveillance system of claim 11, wherein each transformation is a parameterized image-based transformation and not a camera motion parameter.

20. The surveillance system of claim 11, wherein the transient frame is unbiased with respect to pole shake.