Detecting multiple moving objects in crowded environments with coherent motion regions

• US 8,462,987 B2
• Filed: 06/23/2009
• Issued: 06/11/2013
• Est. Priority Date: 06/23/2009
• Status: Active Grant

• Current Assignee: UT-Battelle LLC
• Original Assignee: UT-Battelle LLC
• Inventors: Cheriyadat, Anil M., Radke, Richard J.
• Primary Examiner(s): Bella, Matthew
• Assistant Examiner(s): BROOKS, JULIAN D
• Application Number:
  US12/489,589
• Publication Number:
  US 20100322474A1
• Time in Patent Office: 1,449 Days
• Field of Search: 382/103, 382/181, 382/190, 382/225, 382/107, 348169-172
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 20/53 Recognition of crowd images, e.g. recognition of crowd congestion

Abstract

Coherent motion regions extend in time as well as space, enforcing consistency in detected objects over long time periods and making the algorithm robust to noisy or short point tracks. As a result of enforcing the constraint that selected coherent motion regions contain disjoint sets of tracks defined in a three-dimensional space including a time dimension. An algorithm operates directly on raw, unconditioned low-level feature point tracks, and minimizes a global measure of the coherent motion regions. At least one discrete moving object is identified in a time series of video images based on the trajectory similarity factors, which is a measure of a maximum distance between a pair of feature point tracks.

Claims

1. An apparatus for identifying at least one discrete moving object within a series of video images, said apparatus comprising:

an image recording device that stores a time series of video images; and

an image analysis device that is configured to execute a program of machine-executable instructions to identify at least one moving object in said time series of video images, wherein said program includes the steps of;

identifying a plurality of feature point tracks in said time series of video images, wherein for each video image of said time series of video images, all feature points belonging to a same feature point track are located within a polygon of a predefined size;

calculating a trajectory similarity factor for each pair of feature point tracks said plurality of feature point tracks, wherein each trajectory similarity factor is a measure of a maximum distance between a pair of feature point tracks; and

generating a trajectory similarity matrix S, wherein said trajectory similarity matrix S is a Z×

Z matrix and Z is a total number of feature point tracks in said time series of video images, and wherein each element S_ij in an i-th row and a j-th column in said trajectory similarity matrix S is given by S(i,j) that represents a trajectory similarity factor between an i-th feature point track and a j-th feature point track for all combinations of values for i and j.

2. The apparatus of claim 1, wherein said program further includes the step of generating at least one motion region from said plurality of feature point tracks, wherein each of said at least one motion region is a three-dimensional volume in a space formed by a two dimensional coordinate in said time series of video images and an index in a time dimension, and each of said at least one motion region includes at least one of said plurality of feature point tracks.

3. The apparatus of claim 2, wherein said program further includes the step of generating at least one coherent motion region from each of said at least one motion region, wherein each pair of feature point tracks in each of said at least one coherent motion region has a trajectory similarity factor that exceeds a predetermined threshold value.

4. The apparatus of claim 3, wherein each trajectory similarity factor between a pair of a feature point track in a coherent motion region and another feature point track not belonging to said coherent motion region has a value that does not exceed said predetermined threshold value.

5. The apparatus of claim 1, wherein said S(i,j) is given by the formula, S(i,j)=exp[−

α

×

DQ(i,j)−

β

×

DVQ(i,j))], wherein α and

β

are positive constants, DQ(i,j) is a quantity representing a first statistical quantity of a set of non-linear parameters derived from distances between said i-th feature point track and said j-th feature point track in said time series of video images, and DVQ(i,j) is a second statistical quantity of said set of non-linear parameters for all combination of values for i and j.

6. The apparatus of claim 1, wherein said program further includes the step of generating a Z×

M matrix A, wherein Z is a total number of feature point tracks in said time series of video images, M is a total number of motion regions in said time series of video images, and an element of A in a k-th row and an l-th column is set at 1 only if a k-th feature point track is present in an l-th motion region in at least one of said time series of video images, and is set at 0 if not, and , wherein each of said at least one motion region is a three-dimensional volume in a space formed by a two dimensional coordinate in said time series of video images and a time dimension, and each of said at least one motion region includes at least one of said plurality of feature point tracks.

7. The apparatus of claim 6, wherein said program further includes the steps of:

detecting whether a previously undetected motion region is present in each of said time series of video images; and

if a previously undetected motion region is detected, adding to a Z×

m matrix A′

a column vector representing presence or absence of each feature point track in said time series of video images to form a Z×

(m+1) matrix A″

, wherein m is a positive integer less than M and said Z×

m matrix A′

has m columns representing presence or absence of each feature point track in said time series of video images in each of previously detected motion regions.

8. The apparatus of claim 7, wherein said program further include the step of modifying at least one preexisting column of said Z×

m matrix A′

to reflect detection of an additional feature point track in one of previously detected motion regions corresponding to said at least one preexisting column.

9. The apparatus of claim 7, wherein said program further includes the step of repeating said steps of detecting and adding until all video images in said time series of video images are examined, wherein said (m+1) becomes equal to M upon a last execution of said adding step during said examining

10. The apparatus of claim 6, wherein said program further includes the step of calculating a coherency strength coefficient L(p) for all value of p, wherein p is an integer from 1to M and said coherency strength coefficient L(p) is a measure of a likelihood that a p-th motion region represents movement of a single object.

11. The apparatus of claim 10, wherein said coherency strength coefficient L(p) is given by L(p)=A^T (p)SA(p), wherein A(p) is a column vector that is identical to a p-th column of said Z×

M matrix A, S is said trajectory similarity matrix, and _A^T (p) is a p-th row of a transpose of said Z×

M matrix A.

12. The apparatus of claim 10, wherein said program further includes the step of determining whether each of said coherency strength coefficient L(p) exceeds a minimum value.

13. The apparatus of claim 12, wherein said program further includes the step of generating a set of at least one coherent motion region, wherein each of said at least one coherent motion region has a coherency strength coefficient that exceeds said minimum value, and each motion region that does not belong to said set of at least one coherent motion region has a coherency strength coefficient that does not exceed said minimum value.

14. The apparatus of claim 1, further comprising a video camera configured to generate said time series of video images by continuously capturing video images in real time.

15. The apparatus of claim 1, wherein said image analysis device includes a processor in a computer and said image recording device is embedded in said computer or externally connected to said computer.

16. An apparatus for identifying at least one discrete moving object within a series of video images, said apparatus comprising:

an image recording device that stores a time series of video images; and

an image analysis device that is configured to execute a program of machine-executable instructions to identify at least one moving object in said time series of video images, wherein said program includes the steps of;

identifying a plurality of feature point tracks in said time series of video images, wherein for each video image of said time series of video images, all feature points belonging to a same feature point track are located within a polygon of a predefined size;

calculating a trajectory similarity factor for each pair of feature point tracks said plurality of feature point tracks, wherein each trajectory similarity factor is a measure of a maximum distance between a pair of feature point tracks,

wherein a trajectory similarity factor between an i-th feature point track and a j-th feature point track is given by the formula, S(i,j)=exp[−

α

×

DQ(i,j)−

β

×

DVQ(i,j))], wherein S(i,j) is said trajectory similarity factor between said i-th feature point track and said j-th feature point track, α and

β

are positive constants, DQ(i,j) is a quantity representing a first statistical quantity of a set of non-linear parameters derived from distances between said i-th feature point track and said j-th feature point track in said time series of video images, and DVQ(i,j) is a second statistical quantity of said set of non-linear parameters for all combination of values for i and j.

17. The apparatus of claim 16, wherein said distances are measured distances between pixels of said i-th feature point track and said j-th feature point track in said time series of video images, wherein said first statistical quantity is a maximum, and wherein said second statistical quantity is a standard deviation.

18. The apparatus of claim 17, wherein said non-linear parameters consists of a set of D_t(i,j), wherein D_t(i,j)=√

{square root over ([d_t^x(i,j)]²+[d_t^y(i,j)]²)}{square root over ([d_t^x(i,j)]²+[d_t^y(i,j)]²)},

19. The apparatus of claim 18, wherein said polygon is a rectangle, said w is a width of said rectangle, and said h is a height of said rectangle.

20. The apparatus of claim 18, wherein said program further includes the step of providing at least one external input representing a dimension of said polygon.

21. A method for identifying at least one discrete moving object within a series of video images, said method comprising:

storing a time series of video images in an image recording device;

identifying a plurality of feature point tracks in said time series of video images, wherein for each video image of said time series of video images, all feature points belonging to a same feature point track are located within a polygon of a predefined size;

calculating a trajectory similarity factor for each pair of feature point tracks in said plurality of feature point tracks, wherein each trajectory similarity factor is a measure of a maximum distance between a pair of feature point tracks;

generating a trajectory similarity matrix S, wherein said trajectory similarity matrix S is a Z×

Z matrix and Z is a total number of feature point tracks in said time series of video images, and wherein each element S_i,j in an i-th row and a j-th column in said trajectory similarity matrix S is given by S(i,j) that represents a trajectory similarity factor between an i-th feature point track and a j-th feature point track for all combinations of values for i and j; and

identifying at least one discrete moving object in said time series of video images based on said trajectory similarity factors.

22. The method of claim 21, further comprising generating at least one motion region from said plurality of feature point tracks, wherein each of said at least one motion region is a three-dimensional volume in a space formed by a two dimensional coordinate in said time series of video images and an index in a time dimension, and each of said at least one motion region includes at least one of said plurality of feature point tracks.

23. The method of claim 22, further comprising generating at least one coherent motion region from each of said at least one motion region, wherein each pair of feature point tracks in each of said at least one coherent motion region has a trajectory similarity factor that exceeds a predetermined threshold value.

24. The method of claim 23, wherein each trajectory similarity factor between a pair of a feature point track in a coherent motion region and another feature point track not belonging to said coherent motion region has a value that does not exceed said predetermined threshold value.

25. The method of claim 21, wherein said S(i,j) is given by the formula, S(i,j)=exp[−

α

×

DQ(i,j)−

β

×

DVQ(i,j))], wherein α and

β

are quantity representing a first statistical quantity of a set of non-linear parameters derived from distances between said i-th feature point track and said j-th feature point track in said time series of video images, and DVQ(i,j) is a second statistical quantity of said set of non-linear parameters for all combination of values for i and j.

26. The method of claim 21, further comprising generating a Z×

M matrix A, wherein Z is a total number of feature point tracks in said time series of video images, M is a total number of motion regions in said time series of video images, and an element of A in a k-th row and an l-th column is set at 1 only if a k-th feature point track is present in an l-th motion region in at least one of said time series of video images, and is set at 0 if not, and , wherein each of said at least one motion region is a three-dimensional volume in a space formed by a two dimensional coordinate in said time series of video images and a time dimension, and each of said at least one motion region includes at least one of said plurality of feature point tracks.

27. The method of claim 26, further comprising:

detecting whether a previously undetected motion region is present in each of said time series of video images; and

if a previously undetected motion region is detected, adding to a Z×

m matrix A′

a column vector representing presence or absence of each feature point track in said time series of video images to form a Z×

(m+1) matrix A″

, wherein m is a positive integer less than M and said Z×

m matrix A′

has m columns representing presence or absence of each feature point track in said time series of video images in each of previously detected motion regions.

28. The method of claim 27, further comprising modifying at least one preexisting column of said Z×

m matrix A′

to reflect detection of an additional feature point track in one of previously detected motion regions corresponding to said at least one preexisting column.

29. The method of claim 27, further comprising repeating said steps of detecting and adding until all video images in said time series of video images are examined, wherein said (m +1) becomes equal to M upon a last execution of said adding step during said examining.

30. The method of claim 26, further comprising calculating a coherency strength coefficient L(p) for all value of p, wherein p is an integer from 1 to M and said coherency strength coefficient L(p) is a measure of a likelihood that a p-th motion region represents movement of a single object.

31. The method of claim 30, wherein said coherency strength coefficient L(p) is given by L(p)=A^T (p)SA(p), wherein A(p) is a column vector that is identical to a p-th column of said Z ×

M matrix A, S is said trajectory similarity matrix, and A^T (p) is a p-th row of a transpose of said Z×

M matrix A.

32. The method of claim 31, further comprising determining whether each of said coherency strength coefficient L(p) exceeds a minimum value.

33. The method of claim 32, further comprising generating a set of at least one coherent motion region, wherein each of said at least one coherent motion region has a coherency strength coefficient that exceeds said minimum value, and each motion region that does not belong to said set of at least one coherent motion region has a coherency strength coefficient that does not exceed said minimum value.

34. The method of claim 21, further comprising generating said time series of video images by continuously capturing video images in real time.

35. The method of claim 21, wherein said image analysis device includes a processor in a computer and said image recording device is embedded in said computer or externally connected to said computer and said identifying of said at least one discrete moving object is effected by executing a program of machine-executable instructions.

36. A method for identifying at least one discrete moving object within a series of video images, said method comprising:

storing a time series of video images in an image recording device;

identifying a plurality of feature point tracks in said time series of video images, wherein for each video image of said time series of video images, all feature points belonging to a same feature point track are located within a polygon of a predefined size;

calculating a trajectory similarity factor for each pair of feature point tracks in said plurality of feature point tracks, wherein each trajectory similarity factor is a measure of a maximum distance between a pair of feature point tracks;

identifying at least one discrete moving object in said time series of video images based on said trajectory similarity factors,

wherein a trajectory similarity factor between an i-th feature point track and a j-th feature point track is given by the formula, S(i,j)=exp[−

α

×

DQ(i,j)−

β

×

DVQ(i,j))], wherein S(i,j) is said trajectory similarity factor between said i-th feature point track and said j-th feature point track, α and

β

are positive constants, DQ(i,j) is a quantity representing a first statistical quantity of a set of non-linear parameters derived from distances between said i-th feature point track and said j-th feature point track in said time series of video images, and DVQ(i,j) is a second statistical quantity of said set of non-linear parameters for all combination of values for i and j.

37. The method of claim 36, wherein said distances are measured distances between pixels of said i-th feature point track and said j-th feature point track in said time series of video images, wherein said first statistical quantity is a maximum, and wherein said second statistical quantity is a standard deviation.

38. The method of claim 37, wherein said non-linear parameters consists of a set of D_t(i,j), wherein D_t(i,j)=√

{square root over ([d_t^x(i,j)]²+[d_t^y(i,j)]²)}{square root over ([d_t^x(i,j)]²+[d_t^y(i,j)]²)},

39. The method of claim 38, wherein said polygon is a rectangle, said w is a width of said rectangle, and said h is a height of said rectangle.

40. The method of claim 38, further comprising setting a value for each of at least one dimension of said polygon prior to calculating values for said set of D_t (i,j).

41. A machine-readable data storage device embodying a program of machine-executable instructions to identify at least one discrete moving object, wherein said program includes the steps of:

identifying a plurality of feature point tracks in said time series of video images, wherein for each video image of said time series of video images, all feature points belonging to a same feature point track are located within a polygon of a predefined size;

calculating a trajectory similarity factor for each pair of feature point tracks said plurality of feature point tracks, wherein each trajectory similarity factor is a measure of a maximum distance between a pair of feature point tracks;

generating a trajectory similarity matrix S, wherein said trajectory similarity matrix S is a Z×

Z matrix and Z is a total number of feature point tracks in said time series of video images, and wherein each element S_i,j in an i-th row and a j-th column in said trajectory similarity matrix S is given by S(i,j) that represents a trajectory similarity factor between an i-th feature point track and a j-th feature point track for all combinations of values for i and j; and

identifying at least one discrete moving object in said time series of video images based on said trajectory similarity factors.

42. The machine-readable data storage device of claim 41, wherein said program further includes the step of generating at least one motion region from said plurality of feature point tracks, wherein each of said at least one motion region is a three-dimensional volume in a space formed by a two dimensional coordinate in said time series of video images and an index in a time dimension, and each of said at least one motion region includes at least one of said plurality of feature point tracks.

43. The machine-readable data storage device of claim 42, wherein said program further includes the step of generating at least one coherent motion region from each of said at least one motion region, wherein each pair of feature point tracks in each of said at least one coherent motion region has a trajectory similarity factor that exceeds a predetermined threshold value.

44. The machine-readable data storage device of claim 43, wherein each trajectory similarity factor between a pair of a feature point track in a coherent motion region and another feature point track not belonging to said coherent motion region has a value that does not exceed said predetermined threshold value.

45. The machine-readable data storage device of claim 41, wherein said S(ij) is given by the formula, S(ij)=exp [α

×

DQ(i,j)−

β

×

DVQ(i,j))], wherein α and

α

are positive constants, DQ(i,j) is a quantity representing a first statistical quantity of a set of non-linear parameters derived from distances between said i-th feature point track and said j-th feature point track in said time series of video images, and DVQ(i,j) is a second statistical quantity of said set of non-linear parameters for all combination of values for i and j.

46. The machine-readable data storage device of claim 41, wherein said program further includes the step of generating a Z×

M matrix A, wherein Z is a total number of feature point tracks in said time series of video images, M is a total number of motion regions in said time series of video images, and an element of A in a k-th row and an l-th column is set at 1 only if a k-th feature point track is present in an l-th motion region in at least one of said time series of video images, and is set at 0 if not, and , wherein each of said at least one motion region is a three-dimensional volume in a space formed by a two dimensional coordinate in said time series of video images and a time dimension, and each of said at least one motion region includes at least one of said plurality of feature point tracks.

47. The machine-readable data storage device of claim 46, wherein said program further includes the steps of:

detecting whether a previously undetected motion region is present in each of said time series of video images; and

if a previously undetected motion region is detected, adding to a Z×

m matrix A′

a column vector representing presence or absence of each feature point track in said time series of video images to form a Z×

(m +1) matrix A″

, wherein m is a positive integer less than M and said Z×

m matrix A′

has m columns representing presence or absence of each feature point track in said time series of video images in each of previously detected motion regions.

48. The machine-readable data storage device of claim 47, wherein said program further include the step of modifying at least one preexisting column of said Z×

m matrix A′

to reflect detection of an additional feature point track in one of previously detected motion regions corresponding to said at least one preexisting column.

49. The machine-readable data storage device of claim 46, wherein said program further includes the step of calculating a coherency strength coefficient L(p) for all value of p, wherein p is an integer from 1 to M and said coherency strength coefficient L(p) is a measure of a likelihood that a p-th motion region represents movement of a single object.

50. The machine-readable data storage device of claim 49, wherein said coherency strength coefficient L(p) is given by L(p)=_AT (p)SA(p), wherein A(p) is a column vector that is identical to a p-th column of said Z×

M matrix A, S is said trajectory similarity matrix, and A^t (p) is a p-th row of a transpose of said Z×

M matrix A.

51. A machine-readable data storage device embodying a program of machine-executable instructions to identify at least one discrete moving object, wherein said program includes the steps of:

identifying a plurality of feature point tracks in said time series of video images, wherein for each video image of said time series of video images, all feature points belonging to a same feature point track are located within a polygon of a predefined size;

calculating a trajectory similarity factor for each pair of feature point tracks said plurality of feature point tracks, wherein each trajectory similarity factor is a measure of a maximum distance between a pair of feature point tracks;

identifying at least one discrete moving object in said time series of video images based on said trajectory similarity factors, wherein a trajectory similarity factor between an i-th feature point track and a j-th feature point track is given by the formula, S(i,j)=exp [−

α

×

DQ(i,j) −

β

×

DVQ(i,j)) ], wherein S(i,j) is said trajectory similarity factor between said i-th feature point track and said j-th feature point track, α and

β

are positive constants, DQ(i,j) is a quantity representing a first statistical quantity of a set of non-linear parameters derived from distances between said i-th feature point track and said j-th feature point track in said time series of video images, and DVQ(i,j) is a second statistical quantity of said set of non-linear parameters for all combination of values for i and j.

52. The machine-readable data storage device of claim 51, wherein said distances are measured distances between pixels of said i-th feature point track and said j-th feature point track in said time series of video images, wherein said first statistical quantity is a maximum, and wherein said second statistical quantity is a standard deviation.