Moving object spotting by forward-backward motion history accumulation

• US 9,911,197 B1
• Filed: 03/11/2014
• Issued: 03/06/2018
• Est. Priority Date: 03/14/2013
• Status: Active Grant

• Current Assignee: HRL Laboratories LLC (The Boeing Co.)
• Original Assignee: HRL Laboratories LLC (The Boeing Co.)
• Inventors: Kim, Kyungnam, Jeong, Changsoo S., Khosla, Deepak, Chen, Yang, Cheng, Shinko Y., Honda, Alexander L., Zhang, Lei
• Primary Examiner(s): Lefkowitz, Sumati
• Assistant Examiner(s): Sun, Jiangeng
• Application Number:
  US14/204,471
• Time in Patent Office: 1,456 Days
• Field of Search: None
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06T 7/32 using correlation-based methods

Abstract

Described is a system for detecting moving objects using multi-frame motion history images. An input video sequence of consecutive registered image frames is received. The sequence of consecutive registered image frames comprises forward and backward registered image frames registered to a coordinate system of a reference image frame. Frame differences are computed between each of the consecutive registered image frames and the reference image frame. The frame differences are accumulated based on characteristics of the input video sequence to compute a corresponding motion response value. A selected threshold value is then applied to the motion response value to produce at least one binary image used for detection of moving objects in the input video sequence. Additionally, the invention includes a system for adaptive parameter optimization by input image characterization, wherein parameters that are based on characteristics of the image influence the motion detection process.

Claims

1. A system for detecting moving objects using multi-frame motion history images, the system comprising:

one or more processors and a non-transitory computer-readable medium having instructions encoded thereon such that when the instructions are executed, the one or more processors perform operations of;

receiving an input video sequence of consecutive registered image frames, wherein the sequence of consecutive registered image frames comprises forward and backward registered image frames registered to a coordinate system of a reference image frame;

computing frame differences between each of the consecutive registered image frames and the reference image frame;

accumulating the frame differences based on characteristics of the input video sequence to compute a corresponding motion response value; and

applying a selected threshold value to the motion response value to produce at least one binary image used for detection of moving objects in the input video sequence, wherein the one or more processors further perform operations of;

accumulating absolute values of the frame differences to generate the motion response value; and

applying the selected threshold to the motion response value according to the following;

D=Th[Σ

_i=0^{2 k}abs(F_k−

F_i)],

where Th denotes threshold, Σ

represents a summation, 2 k is the total number of registered image frames registered to the coordinate system of the reference image frame, abs represents absolute value, F_k represents the reference image frame, and F_i represents a registered image frame.

2. A system for detecting moving objects using multi-frame motion history images, the system comprising:

one or more processors and a non-transitory computer-readable medium having instructions encoded thereon such that when the instructions are executed, the one or more processors perform operations of;

receiving an input video sequence of consecutive registered image frames, wherein the sequence of consecutive registered image frames comprises forward and backward registered image frames registered to a coordinate system of a reference image frame;

computing frame differences between each of the consecutive registered image frames and the reference image frame;

accumulating the frame differences based on characteristics of the input video sequence to compute a corresponding motion response value; and

applying a selected threshold value to the motion response value to produce at least one binary image used for detection of moving objects in the input video sequence, wherein the one or more processors further perform operations of;

calculating, then accumulating absolute values of the frame differences for forward and backward registered image frames separately to generate a motion response value for each of the forward and backward registered image frames;

applying the selected threshold to each of the motion response values; and

computing the intersection of the motion response values according to the following;

D=Th[Σ

_i=0<sup>k−

1</sup>abs(F_k−

F_i)]∩

Th[Σ

_i=k+1^{2 k}abs(F_k−

F_i)],

where Th denotes threshold, Σ

represents a summation, k is an index of the reference image frame, 2 k is the total number of registered image frames registered to the coordinate system of the reference image frame, abs represents absolute value, F_k represents the reference image frame, F_i represents a registered image frame, and ∩

represents an intersection.

3. A system for detecting moving objects using multi-frame motion history images, the system comprising:

one or more processors and a non-transitory computer-readable medium having instructions encoded thereon such that when the instructions are executed, the one or more processors perform operations of;

receiving an input video sequence of consecutive registered image frames, wherein the sequence of consecutive registered image frames comprises forward and backward registered image frames registered to a coordinate system of a reference image frame;

computing frame differences between each of the consecutive registered image frames and the reference image frame;

accumulating the frame differences based on characteristics of the input video sequence to compute a corresponding motion response value; and

applying a selected threshold value to the motion response value to produce at least one binary image used for detection of moving objects in the input video sequence, wherein the one or more processors further perform operations of;

accumulating polarized differences between the reference image frame and the consecutive registered image frames for forward and backward registered image frames separately to generate a motion response value for each of the forward and backward registered image frames;

applying the selected threshold to each of the motion response values; and

computing the intersection of the motion response values according to the following

D=Th[Σ

_i=0<sup>k−

1</sup>(F_k−

F_i)]∩

Th[Σ

_i=k+1^{2 k}(F_k−

F_i)],

where Th denotes threshold, Σ

represents a summation, k is an index of the reference image frame, 2 k is the total number of registered image frames registered to the coordinate system of the reference image frame, abs represents absolute value, F_k represents the reference image frame, F_i represents a registered image frame, and ∩

represents an intersection.

4. A system for detecting moving objects using multi-frame motion history images, the system comprising:

one or more processors and a non-transitory computer-readable medium having instructions encoded thereon such that when the instructions are executed, the one or more processors perform operations of;

receiving an input video sequence of consecutive registered image frames, wherein the sequence of consecutive registered image frames comprises forward and backward registered image frames registered to a coordinate system of a reference image frame;

computing frame differences between each of the consecutive registered image frames and the reference image frame;

accumulating the frame differences based on characteristics of the input video sequence to compute a corresponding motion response value; and

applying a selected threshold value to the motion response value to produce at least one binary image used for detection of moving objects in the input video sequence, wherein the one or more processors further perform operations of;

calculating squared differences between the reference image frame and the consecutive registered image frames;

accumulating the squared differences for forward and backward registered image frames separately to generate forward and backward accumulations;

determining a pixel-wise minimum value between the forward and backward accumulations; and

applying the selected threshold to the pixel-wise minimum value according to the following;

D=Th[Min{Σ

_i=0<sup>k−

1</sup>(F_k−

F_i)²,Σ

_i=k+1^{2 k}(F_k−

F_i)²}],

where Th denotes threshold, Σ

represents a summation, k is an index of the reference image frame, 2 k is the total number of registered image frames registered to the coordinate system of the reference image frame, abs represents absolute value, F_k represents the reference image frame, F_i represents a registered image frame, and Min represents minimization.

5. A system for adaptive parameter optimization by input image characterization, the system comprising:

one or more processors and a non-transitory computer-readable medium having instructions encoded thereon such that when the instructions are executed, the one or more processors perform operations of;

receiving an input image sequence comprising a plurality of image frames;

detecting at least one characteristic of an input source related to the input image sequence;

setting at least one parameter based on the at least one characteristic of the input source;

using homography estimation, registering consecutive image frames in the plurality of image frames into a reference image frame;

wherein the at least one parameter is used to determine at least one of;

a number of consecutive image frames to use for registration; and

a selected threshold value to apply to accumulated frame differences between each of the consecutive registered image frames and the reference image frame;

using the at least one parameter to select a motion detection method; and

detecting at least one moving object in the input image sequence using the selected motion detection method,

wherein in using the selected motion detection method,

frame differences between each of the consecutive registered image frames and the reference image frame are computed;

the frame differences are accumulated to compute a motion response value; and

a threshold function is applied to the motion response value, wherein the threshold function is a sum of absolute differences between the reference image frame and the registered consecutive image frames.

16. The system as set forth in claim 5, wherein the one or more processors further perform an operation of:

applying the selected threshold value to the motion response value according to the following;

D=Th[Σ

_i=0^{2 k}abs(F_k−

F_i)],

where Th denotes threshold, Σ

represents a summation, 2 k is the total number of registered image frames registered to a coordinate system of the reference image frame, abs represents absolute value, F_k represents the reference image frame, and F_i represents a registered image frame.

6. A computer-implemented method for detecting moving objects using multi-frame motion history images, comprising an act of:

causing one or more processors to execute instructions stored on a non-transitory memory such that upon execution, the one or more processors performs operations of;

receiving an input video sequence of consecutive registered image frames, wherein the sequence of consecutive registered image frames comprises forward and backward registered image frames registered to a coordinate system of a reference image frame;

computing frame differences between each of the consecutive registered image frames and the reference image frame;

accumulating the frame differences based on characteristics of the input video sequence to compute a corresponding motion response value; and

applying a selected threshold value to the motion response value to produce at least one binary image used for detection of moving objects in the input video sequence, wherein the data processor further perform operations of;

accumulating absolute values of the frame differences to generate the motion response value; and

applying the selected threshold to the motion response value according to the following;

D=Th[Σ

_i=0^{2 k}abs(F_k−

F_i)],

where Th denotes threshold, Σ

represents a summation, 2 k is the total number of registered image frames registered to the coordinate system of the reference image frame, abs represents absolute value, F_k represents the reference image frame, and F_i represents a registered image frame.

7. A computer-implemented method for detecting moving objects using multi-frame motion history images, comprising an act of:

causing one or more processors to execute instructions stored on a non-transitory memory such that upon execution, the one or more processors performs operations of;

receiving an input video sequence of consecutive registered image frames, wherein the sequence of consecutive registered image frames comprises forward and backward registered image frames registered to a coordinate system of a reference image frame;

computing frame differences between each of the consecutive registered image frames and the reference image frame;

accumulating the frame differences based on characteristics of the input video sequence to compute a corresponding motion response value; and

applying a selected threshold value to the motion response value to produce at least one binary image used for detection of moving objects in the input video sequence, wherein the data processor further performs operations of;

calculating, then accumulating absolute values of the frame differences for forward and backward registered image frames separately to generate a motion response value for each of the forward and backward registered image frames;

applying the selected threshold to each of the motion response values; and

computing the intersection of the motion response values according to the following;

D=Th[Σ

_i=0<sup>k−

1</sup>abs(F_k−

F_i)]∩

Th[Σ

_i=k+1^{2 k}abs(F_k−

F_i)],

where Th denotes threshold, Σ

represents a summation, k is an index of the reference image frame, 2 k is the total number of registered image frames registered to the coordinate system of the reference image frame, abs represents absolute value, F_k represents the reference image frame, F_i represents a registered image frame, and ∩

represents an intersection.

8. A computer-implemented method for detecting moving objects using multi-frame motion history images, comprising an act of:

causing one or more processors to execute instructions stored on a non-transitory memory such that upon execution, the one or more processors performs operations of;

receiving an input video sequence of consecutive registered image frames, wherein the sequence of consecutive registered image frames comprises forward and backward registered image frames registered to a coordinate system of a reference image frame;

computing frame differences between each of the consecutive registered image frames and the reference image frame;

accumulating the frame differences based on characteristics of the input video sequence to compute a corresponding motion response value; and

applying a selected threshold value to the motion response value to produce at least one binary image used for detection of moving objects in the input video sequence, wherein the data processor further performs operations of;

accumulating polarized differences between the reference image frame and the consecutive registered image frames for forward and backward registered image frames separately to generate a motion response value for each of the forward and backward registered image frames;

applying the selected threshold to each of the motion response values; and

computing the intersection of the motion response values according to the following;

D=Th[Σ

_i=0<sup>k−

1</sup>(F_k−

F_i)]∩

Th[Σ

_i=k+1^{2 k}(F_k−

F_i)],

where Th denotes threshold, Σ

represents a summation, k is an index of the reference image frame, 2 k is the total number of registered image frames registered to the coordinate system of the reference image frame, abs represents absolute value, F_k represents the reference image frame, F_i represents a registered image frame, and ∩

represents an intersection.

9. A computer-implemented method for detecting moving objects using multi-frame motion history images, comprising an act of:

causing one or more processors to execute instructions stored on a non-transitory memory such that upon execution, the one or more processors performs operations of;

receiving an input video sequence of consecutive registered image frames, wherein the sequence of consecutive registered image frames comprises forward and backward registered image frames registered to a coordinate system of a reference image frame;

computing frame differences between each of the consecutive registered image frames and the reference image frame;

accumulating the frame differences based on characteristics of the input video sequence to compute a corresponding motion response value; and

applying a selected threshold value to the motion response value to produce at least one binary image used for detection of moving objects in the input video sequence, wherein the data processor further performs operations of;

calculating squared differences between the reference image frame and the consecutive registered image frames;

accumulating the squared differences for forward and backward registered image frames separately to generate forward and backward accumulations;

determining a pixel-wise minimum value between the forward and backward accumulations; and

applying the selected threshold to the pixel-wise minimum value according to the following;

D=Th[Min{Σ

_i=0<sup>k−

1</sup>(F_k−

F_i)²,Σ

_i=k+1^{2 k}(F_k−

F_i)²}],

where Th denotes threshold, Σ

represents a summation, k is an index of the reference image frame, 2 k is the total number of registered image frames registered to the coordinate system of the reference image frame, abs represents absolute value, F_k represents the reference image frame, F_i represents a registered image frame, and Min represents minimization.

10. A computer-implemented method for adaptive parameter optimization by input image characterization, comprising:

an act of causing a data processor to execute instructions stored on a non-transitory memory such that upon execution, the data processor performs operations of;

receiving an input image sequence comprising a plurality of image frames;

detecting at least one characteristic of an input source related to the input image sequence;

setting at least one parameter based on the at least one characteristic of the input source;

using homography estimation, registering consecutive image frames in the plurality of image frames into a reference image frame;

wherein the at least one parameter is used to determine at least one of;

a number of consecutive image frames to use for registration; and

a selected threshold value to apply to accumulated frame differences between each of the consecutive registered image frames and the reference image frame;

using the at least one parameter to select a motion detection method; and

detecting at least one moving object in the input image sequence using the selected motion detection method,

wherein in using the selected motion detection method,

frame differences between each of the consecutive registered image frames and the reference image frame are computed;

the frame differences are accumulated to compute a motion response value; and

a threshold function is applied to the motion response value, wherein the threshold function is a sum of absolute differences between the reference image frame and the registered consecutive image frames.

17. The method as set forth in claim 10, wherein the one or more processors further perform an operation of:

applying the selected threshold value to the motion response value according to the following;

D=Th[Σ

_i=0^{2 k}abs(F_k−

F_i)],

where Th denotes threshold, Σ

represents a summation, 2 k is the total number of registered image frames registered to a coordinate system of the reference image frame, abs represents absolute value, F_k represents the reference image frame, and F_i represents a registered image frame.

11. A computer program product for detecting moving objects using multi-frame motion history images, the computer program product comprising computer-readable instructions stored on a non-transitory computer-readable medium that are executable by a computer having a processor for causing the processor to perform operations of:

receiving an input video sequence of consecutive registered image frames, wherein the sequence of consecutive registered image frames comprises forward and backward registered image frames registered to a coordinate system of a reference image frame;

computing frame differences between each of the consecutive registered image frames and the reference image frame;

accumulating the frame differences based on characteristics of the input video sequence to compute a corresponding motion response value; and

applying a selected threshold value to the motion response value to produce at least one binary image used for detection of moving objects in the input video sequence, wherein the computer program product further comprises instructions for causing the processor to perform operations of;

accumulating absolute values of the frame differences to generate the motion response value; and

applying the selected threshold to the motion response value according to the following;

D=Th[Σ

_i=0^{2 k}abs(F_k−

F_i)],

where Th denotes threshold, Σ

represents a summation, 2 k is the total number of registered image frames registered to the coordinate system of the reference image frame, abs represents absolute value, F_k represents the reference image frame, and F_i represents a registered image frame.

12. A computer program product for detecting moving objects using multi-frame motion history images, the computer program product comprising computer-readable instructions stored on a non-transitory computer-readable medium that are executable by a computer having a processor for causing the processor to perform operations of:

receiving an input video sequence of consecutive registered image frames, wherein the sequence of consecutive registered image frames comprises forward and backward registered image frames registered to a coordinate system of a reference image frame;

computing frame differences between each of the consecutive registered image frames and the reference image frame;

accumulating the frame differences based on characteristics of the input video sequence to compute a corresponding motion response value; and

applying a selected threshold value to the motion response value to produce at least one binary image used for detection of moving objects in the input video sequence, wherein the computer program product further comprises instructions for causing the processor to perform operations of;

calculating, then accumulating absolute values of the frame differences for forward and backward registered image frames separately to generate a motion response value for each of the forward and backward registered image frames;

applying the selected threshold to each of the motion response values; and

computing the intersection of the motion response values according to the following;

D=Th[Σ

_i=0<sup>k−

1</sup>abs(F_k−

F_i)]∩

Th[Σ

_i=k+1^{2 k}abs(F_k−

F_i)],

where Th denotes threshold, Σ

represents a summation, k is an index of the reference image frame, 2 k is the total number of registered image frames registered to the coordinate system of the reference image frame, abs represents absolute value, F_k represents the reference image frame, F_i represents a registered image frame, and ∩

represents an intersection.

13. A computer program product for detecting moving objects using multi-frame motion history images, the computer program product comprising computer-readable instructions stored on a non-transitory computer-readable medium that are executable by a computer having a processor for causing the processor to perform operations of:

receiving an input video sequence of consecutive registered image frames, wherein the sequence of consecutive registered image frames comprises forward and backward registered image frames registered to a coordinate system of a reference image frame;

computing frame differences between each of the consecutive registered image frames and the reference image frame;

accumulating the frame differences based on characteristics of the input video sequence to compute a corresponding motion response value; and

applying a selected threshold value to the motion response value to produce at least one binary image used for detection of moving objects in the input video sequence, wherein the computer program product further comprises instructions for causing the processor to perform operations of;

accumulating polarized differences between the reference image frame and the consecutive registered image frames for forward and backward registered image frames separately to generate a motion response value for each of the forward and backward registered image frames;

applying the selected threshold to each of the motion response values; and

computing the intersection of the motion response values according to the following;

D=Th[Σ

_i=0<sup>k−

1</sup>(F_k−

F_i)]∩

Th[Σ

_i=k+1^{2 k}(F_k−

F_i)],

where Th denotes threshold, Σ

represents a summation, k is an index of the reference image frame, 2 k is the total number of registered image frames registered to the coordinate system of the reference image frame, abs represents absolute value, F_k represents the reference image frame, F_i represents a registered image frame, and ∩

represents an intersection.

14. A computer program product for detecting moving objects using multi-frame motion history images, the computer program product comprising computer-readable instructions stored on a non-transitory computer-readable medium that are executable by a computer having a processor for causing the processor to perform operations of:

receiving an input video sequence of consecutive registered image frames, wherein the sequence of consecutive registered image frames comprises forward and backward registered image frames registered to a coordinate system of a reference image frame;

computing frame differences between each of the consecutive registered image frames and the reference image frame;

accumulating the frame differences based on characteristics of the input video sequence to compute a corresponding motion response value; and

applying a selected threshold value to the motion response value to produce at least one binary image used for detection of moving objects in the input video sequence, wherein the computer program product further comprises instructions for causing the processor to perform operations of;

calculating squared differences between the reference image frame and the consecutive registered image flames;

accumulating the squared differences for forward and backward registered image frames separately to generate forward and backward accumulations;

determining a pixel-wise minimum value between the forward and backward accumulations; and

applying the selected threshold to the pixel-wise minimum value according to the following;

D=Th[Min{Σ

_i=0<sup>k−

1</sup>(F_k−

F_i)²,Σ

_i=k+1^{2 k}(F_k−

F_i)²}],

where Th denotes threshold, Σ

represents a summation, k is an index of the reference image frame, 2 k is the total number of registered image frames registered to the coordinate system of the reference image frame, abs represents absolute value, F_k represents the reference image frame, F_i represents a registered image frame, and Min represents minimization.

15. A computer program product for adaptive parameter optimization by input image characterization, the computer program product comprising computer-readable instructions stored on a non-transitory computer-readable medium that are executable by a computer having a processor for causing the processor to perform operations of:

receiving an input image sequence comprising a plurality of image frames;

detecting at least one characteristic of an input source related to the input image sequence;

setting at least one parameter based on the at least one characteristic of the input source;

using homography estimation, registering consecutive image frames in the plurality of image frames into a reference image frame;

wherein the at least one parameter is used to determine at least one of;

a number of consecutive image frames to use for registration; and

a selected threshold value to apply to accumulated frame differences between each of the consecutive registered image frames and the reference image frame;

using the at least one parameter to select a motion detection method; and

detecting at least one moving object in the input image sequence using the selected motion detection method,

wherein in using the selected motion detection method,

frame differences between each of the consecutive registered image frames and the reference image frame are computed;

the frame differences are accumulated to compute a motion response value; and

a threshold function is applied to the motion response value, wherein the threshold function is a sum of absolute differences between the reference image frame and the registered consecutive image frames.

18. The computer program product as set forth in claim 15, further comprising instructions for causing the one or more processors to perform an operation of:

applying the selected threshold value to the motion response value according to the following;

D=Th[Σ

_i=0^{2 k}abs(F_k−

F_i)],

where Th denotes threshold, Σ

represents a summation, 2 k is the total number of registered image frames registered to a coordinate system of the reference image frame, abs represents absolute value, F_k represents the reference image frame, and F_i represents a registered image frame.