Method and system for improving video metadata through the use of frame-to-frame correspondences

• US 20060256200A1
• Filed: 03/27/2006
• Published: 11/16/2006
• Est. Priority Date: 03/25/2005
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: Sarnoff Corporation (SRI International, Inc.)
• Inventors: Spence, Clay Douglas, Matei, Bogdan Calin Mihai, Hanna, Barbara Viviane, Pope, Arthur Robert, Hansen, Michael Wade
• Granted Patent: US 7,313,252 B2
• Time in Patent Office: Days
• US Class Current: 348/208.140
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

A method and system for improving the accuracy and timeliness of video metadata by incorporating information related to the motion of the camera as derived from the video imagery itself. Frame-to-frame correspondences are used to accurately estimate changes in camera pose. While the method and system do not require geo-registration, geo-registration results, if available, may be considered in processing the video images and generating improved camera pose estimates.

Claims

1. A video processing system comprising:

a camera pose measurement extractor configured to extract at least one camera pose measurement from video metadata associated with a video input provided by at least one communicatively connected camera;

a frame correspondence generator configured to generate at least one frame-to-frame correspondence based at least in part on video frames associated with the video input provided by the at least one communicatively connected camera;

an augmented Extended Kalman Filter (EKF) configured to generate an estimated camera state based at least in part on the camera pose measurements received from the communicatively connected camera pose measurement extractor, the frame-to-frame correspondences received from the frame correspondence generator, and a range estimation received from a communicatively connected range estimator; and

a camera pose estimator configured to generate a refined camera pose estimate based at least in part on the estimated camera state received from the communicatively connected augmented EKF.

2. The video processing system according to claim 1, wherein the frame correspondence generator generates the at least one frame-to-frame correspondence based at least in part on a comparison of a first video frame and a subsequent video frame received from a communicatively connected frame storage database configured to store the video frames associated with the video input received from the at least one camera.

3. The system of claim 1, wherein the augmented EKF is configured to:

predict an estimated camera state based on the at least one camera pose measurement and the at least one frame-to-frame correspondence;

compute a frame-to-frame homography based on the range estimation and estimated camera state;

correct the predicted estimated camera state based on the at least one camera pose measurement, the at least one frame-to-frame correspondence, and the frame-to-frame homography.

4. The system of claim 1, wherein the augmented EKF is configured to update the estimated camera state according to an update cycle.

5. The system of claim 4, wherein the update cycle is once per video frame.

6. The system of claim 4, wherein the update cycle is an interval such that the at least one camera pose measurement and the at least one frame-to-frame correspondence are received asynchronously by the augmented EKF.

7. The system of claim 1, wherein the estimated camera state comprises an instant pose of the camera, a rate of change of the pose of the camera, and uncertainty in the instant pose of the camera and the rate of change of the pose of the camera.

8. The system of claim 1, wherein the estimated camera state is further based on at least one frame-to-ground correspondence.

9. The system of claim 1, wherein the at least one camera pose measurement comprises at least one of a location of the camera, a velocity of the camera, an acceleration of the camera, an orientation of the camera, a rate of rotation of the camera, a focal length of the camera, and a rate of change of the focal length of the camera.

10. A method of processing a video input, comprising the steps of:

receiving the video input comprising video metadata and video frames from at least one camera;

extracting at least one camera pose measurement from the video metadata;

generating a frame-to-frame correspondence based on a comparison of at least two of the video frames;

estimating a camera state based on the at least one camera pose measurement, frame-to-frame correspondence, and a range estimation; and

estimating a refined camera pose estimate based on the estimated camera state.

11. The method of claim 10, wherein the step of estimating the camera state further comprises:

predicting a predicted estimated camera state,

computing a homography based at least in part on the predicted estimated camera state and a range estimation; and

correcting the predicted estimated camera state based at least on the frame-to-frame correspondence, the at least one camera pose measurement, and the homography.

12. The method of claim 11, wherein the homography is a frame-to-frame homography.

13. The method of claim 11, wherein the step of correcting includes consideration of a frame-to-ground correspondence.

14. The method of claim 10, wherein the range estimation is generated by a digital terrain elevation model.

15. The method of claim 10, wherein the at least one camera pose measurement includes at least one of a location of the camera, a velocity of the camera, an acceleration of the camera, an orientation of the camera, a rate of rotation of the camera, a focal length of the camera, and a rate of change of the focal length of the camera.

16. The method of claim 10, wherein the step of estimating the camera state is performed by an augmented Extended Kalman Filter.

17. The method of claim 10, wherein the step of estimating the camera state is performed by a discrete sequential estimation filter.

18. The method of claim 10, wherein the estimated camera state comprises an instant pose of the at least one camera, a rate of change of the pose of the at least one camera, and an uncertainty in the instant pose and the rate of change of the pose of the at least one camera.

19. The method of claim 10, wherein the step of estimating the camera state is repeated at a set interval.

20. The method of claim 19, wherein the set interval is once per video frame.

21. A computer-readable storage medium storing computer code for monitoring a target scene, wherein the computer code comprises:

code for receiving a video input comprising video metadata and video frames;

code for extracting at least one camera pose measurement from the video metadata;

code for generating a frame-to-frame correspondence based on a comparison of at least two of the video frames;

code for estimating a camera state based on the at least one camera pose measurement, frame-to-frame correspondence, and a range estimation; and

code for estimating a refined camera pose estimate based on the estimated camera state.

22. The computer-readable storage medium of claim 21 further comprising:

code for predicting a predicted estimated camera state,

code for computing a homography based at least in part on the predicted estimated camera state and a range estimation; and

code for correcting the predicted estimated camera state based at least on the frame-to-frame correspondence, the at least one camera pose measurement, and the homography.