Linear system based, qualitative independent motion detection from compressed MPEG surveillance video

• US 8,773,536 B1
• Filed: 04/17/2012
• Issued: 07/08/2014
• Est. Priority Date: 02/12/2003
• Status: Active Grant

• Current Assignee: The Research Foundation for The State University of New York (State University of New York)
• Original Assignee: The Research Foundation for The State University of New York (State University of New York)
• Inventors: Zhang, Zhongfei
• Primary Examiner(s): HALLENBECK-HUBER, JEREMIAH CHARLES
• Application Number:
  US13/448,519
• Time in Patent Office: 812 Days
• Field of Search: 348/144, 348/152, 348/155
• US Class Current: 348/155
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: H04N 19/48 using compressed domain processing techniques other than decoding, e.g. modification of transform coefficients, variable length coding [VLC] data or run-length data motion estimation in a transform domain H04N19/547; processing of decoded motion vectors H04N19/513
• CPC Class Codes: H04N 19/543 using regions
• CPC Class Codes: H04N 19/61 in combination with predictive coding

Abstract

The present invention features a qualitative method to detect independent motion revealed in successive frames of a compressed surveillance MPEG video stream using linear system consistency analysis without decompression of the stream, identifying the segments containing independent motion in a real-time or faster manner, for the retrieval of these segments. The linear system is constructed using the macroblocks of MPEG compressed video frames. The normal flow value of the macroblock is obtained by taking the dot product between the macroblock gradient vector, computed by averaging the four block gradient vectors, and the motion vector of this macroblock. The normal flow value is filtered for inclusion in the linear system, and the statistic of the matrices of the resulting linear system is determined, filtered to screen out false negatives and outliers, and used to determine the presence or absence of independent motion.

Claims

1. A method for detection of independent motion of a plurality of objects presented in a plurality of images acquired at different spatial positions, comprising:

a) processing each of the plurality of images with an automated processor to generate a respective linear system representing the plurality of objects in the form of a matrix having at least three dimensions stored in a memory;

b) analyzing a plurality of matrices with an automated processor to qualitatively determine whether there is a unique solution for the plurality of matrices as representing a respective linear system, based on a ratio of a singular value decomposition of a matrix with respect to an associated augmented matrix, wherein;

{dot over (x)}+θ

{dot over (y)}+α

x+β

y+y=0

represents a linear constraint for each point at coordinates (x,y) with derivatives {dot over (x)}, {dot over (y)}, represented in respective images, wherein α

, β

, γ

, and θ

are functions of both the motion parameters and affine transform parameters between respective images; and

c) outputting a signal corresponding to the qualitative determination, representing whether or not there is independent motion between the plurality of objects represented in the time sequence of images.

2. The method according to claim 1, wherein:

the singular value decomposition is applied to both a respective matrix D and a respective corresponding augmented matrix Db, defining

3. The method according to claim 1, wherein an affine transform is employed in which parallel lines of an object represented in a plurality of the images remain parallel in the linear system after transformation.

4. The method according to claim 1, wherein a change in a perspective of camera acquiring the plurality of images with respect to objects represented in the plurality of images does not lead to a determination of independent motion.

5. The method according to claim 1, wherein a change in relative positions of objects in the plurality of images leads to a determination of independent motion.

6. The method according to claim 1, wherein said plurality of images are received as a stream comprising a time sequence of images, and said processing and said analyzing are performed in at least real time speed with respect to the time sequence.

7. The method according to claim 1, wherein the plurality of images are in an Motion Picture Experts Group 2 (ISO 13818-2) digital video format.

8. The method according to claim 1, wherein the plurality of images are compressed in a digital video format which comprises blocks comprising arrays of image pixels, and macroblocks comprising arrays of the blocks, further comprising estimating a motion parameter from a transformation at least one of said macroblocks over time, without rendering the compressed digital video data as a raster image.

9. The method according to claim 8, further comprising estimating said motion parameters using a normal flow computation, and wherein the normal flow computation is not used in said processing or said analyzing.

10. An apparatus for detecting independent motion of an object represented in a plurality of images acquired at different spatial positions, comprising:

a) at least one automated processor configured to process each of the plurality of images with an automated processor to generate a respective linear system representing the plurality of objects in the form of a matrix having at least three dimensions, and to store the generated matrix in a memory;

b) at least one automated processor configured to retrieve a plurality of matrices from the memory, and to analyze a plurality of matrices to qualitatively determine whether there is a unique solution for the plurality of matrices as representing a respective linear system, based on a ratio of a singular value decomposition of a matrix with respect to an associated augmented matrix, wherein;

{dot over (x)}+θ

{dot over (y)}+α

x+β

y+γ

=0

represents a linear constraint for each point at coordinates (x,y) with derivatives {dot over (x)}, {dot over (y)}, represented in respective images, wherein α

, β

, γ

, and θ

are functions of both the motion parameters and affine transform parameters between respective images; and

c) an output port configured to produce a signal corresponding to the qualitative determination, the signal representing whether or not there is independent motion between the plurality of objects represented in the time sequence of images.

11. The apparatus according to claim 10, wherein:

the singular value decomposition is applied to both a respective matrix D and a respective corresponding augmented matrix Db, defining

12. The apparatus according to claim 10, wherein the at least one processor for processing is configured to perform at least an affine transform in which parallel lines of an object represented in a plurality of the images remain parallel in the linear system after transformation.

13. The apparatus according to claim 10, the at least one processor for analyzing is configured to implement an algorithm in wherein a change in a perspective of camera acquiring the plurality of images with respect to objects represented in the plurality of images does not lead to a determination of independent motion.

14. The apparatus according to claim 10, wherein a change in relative positions of objects in the plurality of images leads the at least one processor for analyzing to produce a determination of independent motion.

15. The apparatus according to claim 10, wherein the at least one processor for processing receives said plurality of images a stream comprising a time sequence of images, and said output port is configured to produce the signal at least real time speed with respect to the time sequence.

16. The apparatus according to claim 10, wherein the plurality of images are received by the at least one processor for processing in an Motion Picture Experts Group 2 (ISO 13818-2) digital video format.

17. The apparatus according to claim 10, wherein the plurality of images are received the at least one processor for processing compressed in a digital video format which comprises blocks comprising arrays of image pixels, and macroblocks comprising arrays of the blocks, wherein the at least one processor for processing is further configured to estimate a motion parameter from a transformation at least one of said macroblocks over time, without rendering the compressed digital video data as a raster image.

18. A method of detecting independent motion in a plurality of stream of digitally compressed video data, the steps comprising:

a) selecting a pair of video frames from a stream of digitally compressed video data representing a three dimensional scene;

b) automatically building a linear system representing the three-dimensional scene for said pair of video frames using at least a three-dimensional to two-dimensional affine transform in which parallel lines remain parallel after affine transformation, and a change in camera position with respect to objects represented in the pair of video frames does not render the linear system inconsistent;

c) automatically analyzing said linear system represented in a matrix D to qualitatively determine the consistency thereof, substantially without quantitatively solving said linear system, based on a ratio of a singular value decomposition of a matrix D with respect to an associated augmented matrix Db, to determine a Rank; and

d) comparing the respective Ranks of a plurality of matrices D to determine independent motion therebetween,

wherein;

{dot over (x)}+θ

{dot over (y)}+α

x+β

y+γ

=0

represents a linear constraint for each point at coordinates (x,y) with derivatives {dot over (x)}, {dot over (y)}, represented in the digitally compressed video data, wherein α

, β

, γ

, and θ

are functions of both the motion parameters and affine transform parameters between respective frames of the digitally compressed video data.

19. The method according to claim 18, wherein Rank is defined such that a matrix A is of Rank r if it contains at least one square submatrix of order r with a non-zero determinant, while the determinant of any square submatrix having (r+1) or higher order, possibly contained in A, is zero, the equivalence of the of respective rank Rank(D) of the matrix D and rank Rank(Db) of an augmented matrix Db being determined by applying a Singular Value Decomposition to both matrix D and augmented matrix Db, defining

20. The method according to claim 18, wherein a change in camera position of a camera acquiring the video frames of the digitally compressed video data, with respect to objects represented in the time sequence of images does not render the linear system inconsistent, while a motion of one object with respect to another object renders the linear system inconsistent.