Method and system for tracking a fast moving object

• US 20010046309A1
• Filed: 03/29/2001
• Published: 11/29/2001
• Est. Priority Date: 03/30/2000
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Kamei, Toshio
• Granted Patent: US 6,819,778 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06T 7/254 involving subtraction of images

Abstract

A moving object is detected by subtraction between successive frames produced by a video camera and a position and a size of the moving object are determined. Preceding and succeeding frames produced by the video camera are stored into respective memories and the panning, tilting and zooming functions of the video camera are controlled according to the determined position and size of the moving object. A motion compensation is performed on the preceding frame to compensate for a motion of background image caused by a tracking movement of the camera so that coordinates of the motion-compensated frame are transformed to coordinates of the succeeding frame. An image difference between the motion-compensated frame and the succeeding frame is extracted as a moving object and a position and a size of the extracted image are determined, with which the video camera is adaptively controlled.

Claims

1. A method or following a detected moving object using a video camera having panning, tilting and zooming functions, comprising the steps of:

a) storing succeeding and preceding frames produced by said camera into first and second memories respectively and controlling the panning, tilting and zooming functions of said video camera according to a position and a size of said moving object;

b) performing a motion compensation on said preceding frame to compensate for a motion of a background image of said moving object so that coordinates of said preceding frame are transformed to coordinates of the succeeding frame;

c) detecting an image difference between the motion-compensated preceding frame and said succeeding frame and determining from the image difference a position and a size of the moving object; and

d) repeating the steps (a) to (c) by using said position and said size of the step (c) when the step (b) is repeated.

2. The method of

claim 1

, further comprising the steps of detecting a moving object by subtraction between successive frames produced by sad video camera and determining a position and a size of the moving object as the position and the size of the step (a).

3. The method of

claim 1

, wherein the step (b) comprises the steps of (b₁) calculating motion parameters according to an iterative gradient method under constrained optical flow and (b₂) performing said motion compensation according to the motion parameters.

4. The method of

claim 3

, wherein the step (b₁) comprises the steps of:

determining initial values of motion parameters by using an SSD (sum of squared differences) matching method;

performing an affine transformation of the preceding frame to the succeeding frame by using said initial values of motion parameters;

estimating said motion parameters of background image by using the affine-transformed preceding frame under constraint conditions of optical flow; and

repeating the affine transformation and the estimation of the motion parameters until said motion parameters have a sufficient degree of precision.

5. The method of

claim 1

, wherein the time interval between the motion-compensated frame and said succeeding frame of the step (c) is adaptively varied according to the size of said moving object.

6. The method of

claim 1

, wherein the time interval between the motion-compensated frame and said succeeding frame of the step (c) is adaptively varied according to a speed of movement of said moving object.

7. A method of following a detected moving object using a video camera having panning, tilting and zooming functions, comprising the steps of:

a) producing a template representing the detected moving object;

b) storing succeeding and preceding frames produced by said camera into first and second memories respectively and controlling the panning, tilting and zooming functions of said video camera according to a position and a size of the moving object;

c) selecting one of first, second and third tracking modes depending on a movement of the detected moving object;

d) if sand first tracking mode is selected, performing a motion compensation on said preceding frame to compensate for a motion of a background image of said moving object so that coordinates of said preceding frame are transformed to coordinates of the succeeding frame;

e) detecting an image difference between the motion-compensated frame and said succeeding frame and determining from the image difference a position and a size of the moving object, and returning to the step (a);

f) if said second tracking mode is selected, transforming the size of the template as an updated size of the moving object according to a zooming operation of said video camera caused by the step (b), and determining a position of said moving object that matches the transformed template, and returning to the step (b);

g) if said third tracking mode is selected, transforming the size of the template as an updated size of the moving object according to a zooming operation of said video camera caused by the step (a);

h) performing a motion compensation on the preceding frame to compensate for a motion of a background image of said moving object so that coordinates of said preceding frame are transformed to coordinates of the succeeding frame; and

i) detesting an image difference between the motion-compensated frame and said succeeding frame, determining a position of the moving object by using the transformed template and the image difference and returning to the step (a).

8. The method of

claim 7

, further comprising the steps of detecting a moving object by subtraction between successive frames produced by said video camera and determining a position and a size of the moving object as the position and the size of the step (b).

9. The method of

claim 7

, wherein each of the steps (d) and (h) comprises the steps of calculating motion parameters according to an iterative gradient method under constrained optical flow and performing said motion compensation according to the motion parameters.

10. The method of

claim 9

, wherein the step of motion parameter calculation comprises the steps of:

determining initial values of motion parameters by using an SSD (sum of squared differences) matching method;

performing an affine transformation of the preceding frame to the succeeding frame by using said initial values of motion parameters;

estimating said motion parameters of background image by using the affine-transformed preceding frame under constraint conditions of optical flow; and

repeating the affine transformation and the estimation of the motion parameters until said motion parameters have a sufficient degree of precision.

11. The method of

claim 7

, wherein the time interval between the motion-compensated frame and said succeeding frame of the step (e) is adaptively varied according to the size of said moving object.

12. The method of

claim 7

, wherein the time interval between the motion-compensated frame and said succeeding frame of the step (e) is adaptively varied according to a speed of movement of said moving object.

13. The method of

claim 7

, wherein the step (f) comprises the step of determining said position of said moving object by using an SAD (sum of absolute differences) matching method.

14. The method of

claim 7

, wherein the step (i) comprises the step of determining said position of the moving object by using a linear sum of an SAD (sum of absolute differences) matching value and said image difference.

15. An image tracking system for tracking a detected moving object, comprising:

a video camera having panning, tilting and zooming functions;

first and second memories for storing succeeding and preceding frames produced by said camera, respectively;

tracking control circuitry connected to the video camera for performing a motion compensation on the preceding frame to compensate for a motion of a background image of the detected moving object so that coordinates of said preceding frame are transformed to coordinates of the succeeding frame, detecting an image difference between the motion-compensated frame and the succeeding frame, and determining from the image difference a position and a size of the moving object; and

camera control circuitry for controlling the panning, tilting and zooming functions of said video camera according to said position and said size of the moving object.

16. The image tracking system of

claim 15

, wherein said tracking control circuitry is configured to detect a moving object by subtraction between successive frames produced by said video camera and determine initial values of said position and said size of the moving object.

17. The image tracking system of

claim 15

, wherein said tracking control circuitry is configured to use an iterative gradient method under constrained optical flow to determine motion parameters and perform said motion compensation according to the determined motion parameters.

18. The image tracking system of

claim 15

, wherein the function (b) comprises the functions of (b₁) calculating motion parameters according to an iterative gradient method under constrained optical flow and (b₂) performing said motion compensation according to the motion parameters.

19. The image tracking system of

claim 18

, wherein the function (b₁) comprises the functions of:

determining initial values of motion parameters by using an SSD (sum of squared differences) matching method;

performing an affine transformation of the preceding frame to the succeeding frame by using said initial values of motion parameters;

estimating said motion parameters of background image by using the affine-transformed preceding frame under constraint conditions of optical flow; and

repeating the affine transformation and the estimation of the motion parameters until said motion parameters have a sufficient degree of precision.

20. The image tracking system of

claim 15

, wherein the time interval between the motion-compensated frame and said succeeding frame is adaptively varied according to the size of said moving object.

21. The image tracking system of

claim 15

, wherein the time interval between the motion-compensated frame and said succeeding frame is adaptively varied according to a speed of movement of said moving object.

22. An image tracking system for tracking a detected moving object, comprising:

a video camera having panning, tilting and zooming functions;

a template memory;

first and second memories for storing succeeding and preceding frames produced by said camera, respectively; and

tracking control circuitry for performing the functions of;

a) storing the detected moving object as a template in said template memory;

b) controlling the panning, tilting and zooming functions of said video camera according to a position and a size of the moving object;

c) selecting one of first second and third tracking modes depending on a movement of the detected moving object;

d) if said first tracking mode is selected, performing a motion compensation on said preceding frame to compensate for a motion of a background image of said moving object so that coordinates of said preceding frame are transformed to coordinates of the succeeding frame;

e) detecting an image difference between the motion-compensated frame and said succeeding frame and determining from the image difference a position and a size of the moving object, and returning to the function (a);

f) if said second tracking mode is selected, transforming the size of the template as an updated size of the moving object according to the zooming operation of said video camera caused by the function (b), and determining a position of said moving object that matches the transformed template, and returning to the function (b);

g) if said third tracking mode is selected, transforming the size of the template as an updated size of the moving object according to a zooming operation of said video camera caused by the function (b);

h) performing a motion compensation on the preceding frame to compensate for a motion of a background image of said moving object so that coordinates of said preceding frame are transformed to coordinates of the succeeding frame; and

i) detecting an image difference between the motion-compensated frame and said succeeding frame, determining a position of the moving object by using the transformed template and the image difference and returning to the function (a).

23. The image tracking system of

claim 22

, wherein said tracking control circuitry is configured to detect a moving object by subtraction between successive frames produced by said video camera and determine a position and a size of the moving object as the position and the size of the function (b).

24. The image tracking system of

claim 22

, wherein said tracking control circuitry is configured to use an iterative gradient method under constrained optical flow to determine motion parameters and perform said motion compensation according to the determined motion parameters.

25. The image tracking system of

claim 22

, wherein each of the functions (d) and (h) comprises the functions of calculating motion parameters according to an iterative gradient method under constrained optical flow and performing said motion compensation according to the motion parameters.

26. The image tracking system of

claim 25

, wherein the function of motion parameter calculation comprises the steps of:

determining initial values of motion parameters by using an SSD (sum-of-squared differences) matching method;

performing an affine transformation of the preceding frame to the succeeding frame by using said initial values of motion parameters;

estimating said motion parameters of background image by using the affine-transformed preceding frame under constraint conditions of optical flow; and

repeating the affine transformation and the estimation of the motion parameters until said motion parameters have a sufficient degree of precision.

27. The image tracking system of

claim 22

, wherein the time interval between the motion-compensated frame and said succeeding frame is adaptively varied according to the size of said moving object.

28. The image tracking system of

claim 22

, wherein the time interval between the motion-compensated frame and said succeeding frame is adaptively varied according to a speed of movement of said moving object.

29. The image tracking system of

claim 22

, wherein the function (f) comprises determining said position of said moving object by using an SAD (sum of absolute differences) matching method.

30. The image tracking system of

claim 22

, wherein the function (i) comprises determining said position of the moving object by using a linear sum of an SAD (sum of absolute differences) matching value and said image difference.