Real-time camera tracking system using optical flow feature points

• US 9,696,404 B1
• Filed: 05/06/2015
• Issued: 07/04/2017
• Est. Priority Date: 05/06/2014
• Status: Active Grant

• Current Assignee: US Department of The Air Force (U.S. Department Of Defense)
• Original Assignee: United States Of America As Represented By The Secretary Of The Air Force
• Inventors: Doyle, Daniel D., Jennings, Alan L., Black, Jonathan T.
• Primary Examiner(s): Hsia, Sherrie
• Application Number:
  US14/705,922
• Time in Patent Office: 790 Days
• Field of Search: 348 36, 348 37, 348 39, 348169-172, 34820899, 3482081, 3482083, 34820814, 342 75, 342 76, 34053913
• CPC Class Codes: G01S 3/7864 T.V. type tracking systems
• CPC Class Codes: H04N 23/6815 by distinguishing pan or tilt from motion
• CPC Class Codes: H04N 23/695 Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects

Abstract

A new apparatus and method for tracking a moving object with a moving camera provides a real-time, narrow field-of-view, high resolution and on target image by combining commanded motion with an optical flow algorithm for deriving motion and classifying background. Commanded motion means that movement of the pan, tilt and zoom (PTZ) unit is “commanded” by a computer, instead of being observed by the camera, so that the pan, tilt and zoom parameters are known, as opposed to having to be determined, significantly reducing the computational requirements for tracking a moving object. The present invention provides a single camera pan and tilt system where the known pan and tilt rotations are used to calculate predicted optical flow points in sequential images, so that resulting apparent movement can be subtracted from the movement determined by an optical flow algorithm to determine actual movement, following by use of a Kalman filter algorithm to predict subsequent locations of a determined moving object and command the pan and tilt unit to point the camera in that direction.

Claims

1. A computer vision system for tracking a moving object, comprising:

(a) a pan and tilt unit;

(b) a digital camera mounted on the pan and tilt unit;

(c) a programmable computer processing unit, connected to the pan and tilt unit, for controlling panoramic and tilt rotation of the pan and tilt unit such that a received image from the digital camera follows the moving object, wherein the programmable computer processing unit is programmed to successively process received images from the digital camera to;

(i) detect a first set of optical flow point locations in a first received image;

(ii) match a second set of optical flow point locations in a second received image to corresponding optical flow point locations in the first received image by an optical flow algorithm;

(iii) predict locations of the second set of optical flow points based on panoramic and tilt rotation commanded by the programmable computer processing unit from the first received image to the second received image;

(iv) subtract the predicted locations from the matched second set of optical flow point locations to derive actual background motion of the optical flow points;

wherein optical flow points with a derived motion greater than a threshold value are characterized as moving points;

(v) calculate the location of a moving object as a location among the moving points;

(vi) command the pan and tilt unit to point the camera at the calculated location of the moving object; and

,

(vii) return to step (c)(i).

2. The computer vision system for tracking a moving object according to claim 1, wherein the optical flow points are determined by locating good features in an image.

3. The computer vision system for tracking a moving object according to claim 1, wherein the location of a moving object in step (c)(v) is calculated as the centroid of the moving points.

4. The computer vision system for tracking a moving object according to claim 1, the digital camera including a lens, wherein the predicted locations of the second set of optical flow points in step (c) (iii) are predicted as the transform of optical flow points (x,y) in the first received image to optical flow points (x′

,y′

) in the second received image, based on;

5. A programmable computer processor implemented computer vision method for tracking a moving object using a digital camera, including a lens, mounted on a pan and tilt unit, and a programmable computer processing unit, comprising the steps of:

(a) detecting a first set of optical flow point locations in a first received image;

(b) matching a second set of optical flow point locations in a second received image to corresponding optical flow point locations in the first received image by an optical flow algorithm;

(c) predicting locations of the second set of optical flow points based on panoramic and tilt rotation commanded by the programmable computer processing unit from the first received image to the second received image;

(d) subtracting the predicted locations from the matched second set of optical flow point locations to derive actual background motion of the optical flow points;

wherein optical flow points with a derived motion greater than a threshold value are characterized as moving points;

(e) calculating the location of a moving object as a location among the moving points;

(f) commanding the pan and tilt unit to point the camera at the calculated location of the moving object; and

,

(g) returning to step (a).

6. The computer vision method for tracking a moving object according to claim 5, wherein the optical flow points are determined by locating good features in an image.

7. The computer vision method for tracking a moving object according to claim 5, wherein the location of a moving object in step (e) is calculated as the centroid of the moving points.

8. The computer vision method for tracking a moving object according to claim 5, wherein the predicted locations of the second set of optical flow points in step (c) are predicted as the transform of optical flow points (x,y) in the first received image to optical flow points (x′

,y′

) in the second received image, based on;