Target moving object tracking device

• US 20080100709A1
• Filed: 08/31/2007
• Published: 05/01/2008
• Est. Priority Date: 10/27/2006
• Status: Active Grant

• Current Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Original Assignee: Matsushita Electric Works Ltd (Panasonic Holdings Corporation)
• Inventors: Furukawa, Satoshi
• Granted Patent: US 8,194,134 B2
• Time in Patent Office: Days
• US Class Current: 348/169
• CPC Class Codes: G01S 3/7865 using correlation of the live video image with a stored image
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20132 Image cropping
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/248 involving reference images or patches

Abstract

A target moving object tracking device takes a time series of picture images of a target moving object, and track a movement of the moving object in the picture images for displaying an enlarged view of the moving object. The device includes a template memory storing a template image which is compared with each one of time-series outline images given from the picture images to determine a partial area in match with the template image for each of the outline images and extract the partial areas as a moving object outline image. The template image is constantly updated by being replaced with a combination of the previous moving object outline images to accurately reflect the moving object.

Claims

1. A target moving object tracking device comprising:

a picture image memory configured to store a time series of real picture images taken by a video camera with regard to an inspection zone covering a possible target moving object;

a display configured to display selective one or ones of said real picture images at a desired magnification;

an outline image processor configured to provide outline images respectively from said real picture images;

a template memory configured to store a template image identifying said target moving object;

a moving object locator configured to compare each one of said outline images with said template image to detect a partial area of each outline image that matches with the template image, said moving object locator obtaining positional data of the target moving object within the inspection zone based upon the partial area detected to be in match with said template image;

enlarged picture generation means configured to extract, based upon said positional data, an enlarged image from a portion of said real picture image corresponding to the partial area of said outline image detected to match with the template image, and to display the enlarged picture image on said display;

wherein

said moving object locator is configured to extract a moving object outline image from each of said outline images in correspondence to said partial area detected to be in match with said template image,

template updating means is provided to update said template image by replacing it with a combination of a current one of said moving object outline images with previous one or ones of said moving object outline image.

2. A target moving object tracking device as set forth in claim 1, wherein said outline image processor is configured to provide said outline image defined by binary data.

3. A target moving object tracking device as set forth in claim 1, wherein said outline image processor is configured to provide said outline image defined by discrete gray scale data.

4. A target moving object tracking device as set forth in claim 1, wherein said outline image processor is configured to obtain a contrast of said template image so as to provide said outline image defined by binary data when said contrast exceeds a predetermined reference, and provide said outline image defined by gray scale data when said contrast is lower than said reference.

5. A target moving object tracking device as set forth in claim 4, wherein said outline image processor is configured to detect an average pixel value which is an average of pixel values given respectively to pixels within each of a plurality of divisions of said template image, and to judge that said contrast is below said reference when any one of said divisions is detected to have said average pixel value lower than a threshold or when said average pixel value detected for any one of said divisions is lower than that for another of said divisions by more than a predetermined extent.

6. A target moving object tracking device as set forth in claim 2, wherein said outline image processor is configured to provide a variable threshold for converting the real picture image into said outline image of the binary data, said outline image processor being configured to obtain an average gray scale value of said template image, and lower the threshold as said average gray scale value is lower than a predetermined limit.

7. A target moving object tracking device as set forth in claim 1, further including:

a moving object outline image memory is provided to store a time series of the moving object outline images,

said template updating means is configured to read from said moving object outline image memory a predetermined number of the previous moving object outline images, combine these outline images with the current moving object outline image, and update the previous template image by replacing it with said combination.

8. A moving object tracking device as set forth in claim 7, wherein said template updating means is configured to update said template image each time a fresh consecutive set of said moving object outline images are accumulated up to the predetermined number.

9. A target moving object tracking device as set forth in claim 7 or 8, wherein said template updating means is configured to combine only the moving object outline image which is determined to be valid in accordance with a predetermined criterion.

10. A target moving object tracking device as set forth in claim 9, wherein said template updating means is configured to calculate a pixel index which is a number of pixels that are included in each of the moving object outline images and have a pixel value of more than zero, and provide said criterion which determines the current moving object outline image as valid when the pixel index of the current moving object outline image varies from that of the previous moving object outline image by more than a predetermined extent.

11. A target moving object tracking device as set forth in claim 9, wherein said template updating means is configured to calculate a standard deviation of the pixel value of one of the current moving object outline image and the corresponding real picture image, and to provide said criterion which determines the current moving object outline image as valid when said standard deviation varies by more than a predetermined extent from the standard deviation calculated with regard to the previous moving outline object image.

12. A target moving object tracking device as set forth in claim 9, wherein said template updating means is configured to calculate the number of pixels constituting an outline of said moving object in each of said moving object outline images, and to provide said criterion which determines the current moving object outline image as valid when said number of the pixels with regard to the current moving object outline image varies by more than a predetermined extent from the number of the pixels with regard to the previous moving object outline image.

13. A target moving object tracking device as set forth in claim 1, wherein said moving object locator comprises matching means which is configured to collect different partial areas out of said outline image, each areas being of the same size of said template image, to calculate a correlation with regard to each of said different areas, and to determine the partial area having a maximum correlation as said moving object outline image in match with said template image, and

said template updating means is configured to obtain a pixel value for each of pixels in said moving object outline image so as to add the pixel value to each corresponding one of the pixels in the previous moving outline image.

14. A target moving object tracking device as set forth in claim 13, wherein said matching means is configured to define said correlation as a sum of pixel values obtained from said template image with regard to pixels selected to correspond to the pixels constituting an outline within each of said partial areas of said outline image.

15. A target moving object tracking device as set forth in claim 13, wherein said matching means is configured to define said correlation as a power sum of the pixel values obtained from said template image with regard to the pixels selected to correspond to the pixels constituting an outline within each of said partial areas of said outline image.

16. A target moving object tracking device as set forth in claim 13, wherein said outline image processor is configured to give the outline image of binary data in which the pixel value of “

1”

is assigned to the pixels constituting an outline of the outline image and the pixel value of “

0”

is assigned to the remainder pixels of the outline image, said matching means is configured to select the pixels from said template image corresponding to the pixels constituting the outline within each of said partial areas of said outline image, and obtain a number of the pixels having a pixel value of greater than “

0”

around each of the selected pixels so as to weight the pixel value for each of the selected pixels in accordance with thus obtained number of the pixels, said matching means being configured to define said correlation as the sum of thus weighted pixel values of the selected pixels of the template image.

17. A target moving object tracking device as set forth in claim 13, wherein said outline image processor is configured to give the outline image of binary data in which the pixel value of “

1”

is assigned to the pixels constituting an outline of the outline image and the pixel value of “

0”

is assigned to the remainder pixels of the outline image, said matching means is configured to obtain a first number of the pixels for each of said partial areas satisfying that both of the pixels in said partial area and the corresponding pixels in said template image have the pixel value of “

1”

or greater, and a second number of the pixels in said template image having the pixel value of “

1”

or greater, said matching means defining said correlation for each of said partial areas that is a ratio of the first number to said second number.

18. A target moving object tracking device as set forth in claim 13, wherein said outline image processor is configured to give the outline image of binary data in which the pixel value of “

1”

is assigned to the pixels constituting an outline of the outline image and the pixel value of “

0”

is assigned to the remainder pixels of the outline image, said matching means is configured to obtain a first number of the pixels for each of said partial areas satisfying that both of the pixels in said partial area and the corresponding pixels in said template image have the pixel value of “

1”

or greater, a second number of the pixels for each of said partial areas satisfying that both of the pixels in said partial area and the corresponding pixels in said template image have the pixel value of “

0”

, and a third number of the pixels in said template image having the pixel value of “

1”

or greater,

said matching means defining said correlation for each of said partial areas that is a ratio of the first number plus said second number to said third number.

19. A target moving object tracking device as set forth in claim 13, wherein said outline image processor is configured to give the outline image of binary data in which the pixel value of “

1”

is assigned to the pixels constituting an outline of the outline image and the pixel value of “

0”

is assigned to the remainder pixels of the outline image, said matching means is configured to obtain a maximum of said pixel values from a set of pixels arranged around a selected pixel in said template image which corresponds to each of pixels constituting an outline within each of said partial areas of the outline image, and to define said correlation as a sum of said maximum values obtained respectively for partial areas.

20. A target moving object tracking device as set forth in claim 13, wherein said outline image processor is configured to give the outline image of binary data in which the pixel value of “

1”

is assigned to the pixels constituting an outline of the outline image and the pixel value of “

0”

is assigned to the remainder pixels of the outline image, said matching means is configured to obtain

a first row index which is the number of the pixels arranged in each row of each said partial area and having the pixel value of more than “

0”

,

a first column index which is the number of the pixels arranged in each column of each said partial area of the outline image and having the pixel value of more than “

0”

,

a second row index which is the number of the pixels arranged in each row of said template image and having the pixel value of more than “

0”

,

a second column index which is the number of the pixels arranged in each column of said template image and having the pixel value of more than “

0”

,

a row number difference which is a difference between said first row index and said second row index for each row, and

a column number difference which is a difference between said first column index and said second column index for each column,

a total row value which is a sum of said row number differences obtained respectively with respect to the rows;

a total column value which is a sum of said column number differences obtained respectively with respect to the columns;

said matching means defining said correlation as an inverse of a sum of said total row value and said total column values for each of said partial areas.

21. A target moving object tracking device as set forth in claim 1, further including:

position estimating means for estimating a limited search region within said outline image for detection of the target moving object, and

moving object extracting means configured to detect at least one possible moving object based upon a time-dependent difference between two or more of said consecutive outline images and provide at least one mask section of reduced size covering said moving object;

said position estimating means being configured to obtain a time series data of said positional data which is stored in a position data memory each time said moving object locator gives the positional data, to calculate an estimated position of the target moving object based upon two or more consecutive time series data of said positional data, to set a prospected zone of a predetermined size around said estimated position, and to provide said limited search region which is a minimum region including said at least one said mask section which overlaps said prospected zone,

said moving object locator being configured to select said partial areas only within said limited search region.

22. A target moving object tracking device as set forth in claim 21, wherein said position estimating means is configured to calculate an estimated moving speed of the moving object based upon two or more consecutive time series data of said positional data, and to provide said prospected zone of which size is proportional to said estimated speed of the moving object.

23. A target moving object tracking device as set forth in claim 21, wherein said position estimating means is configured to determine the size of said prospected zone which is a function of the size of said template image.

24. A target moving object tracking device as set forth in claim 22 or 23, wherein said position estimating means is configured to

obtain a row index which is the number of the pixels arranged along each row of said limited search region and having the pixel value of “

1”

or greater, select a group of consecutive rows each having the row index greater than a predetermined row threshold,

obtain a column index which is the number of the pixels arranged along each column of said limited search region and having the pixel value of “

1”

or greater, select a group of consecutive columns each having the column index greater than a predetermined column threshold, and

further delimit said limited search region to a region confined by the selected group of the consecutive rows and the selected group of the consecutive columns.

25. A target moving object tracking device as set forth in claim 24, wherein said position estimating means is configured to validate, when two or more groups of the consecutive rows are selected, only one of said groups which is closer to said estimated position of the target moving object, and to validate, when two or more groups of the consecutive columns are selected, only one of said group which is closer to said estimated position of the target moving object.

26. A target moving object tracking device as set forth in claim 22 or 23, wherein said position estimating means is configured to

obtain a row index which is the number of the pixels arranged along each row of said limited search region and having the pixel value of “

1”

or greater,

select at least one group of consecutive rows each having the row index greater than a predetermined row threshold,

validate only one group of the consecutive rows which is closer to said estimated position of the target moving object, when two or more groups of the consecutive rows are selected,

obtain a column index which is the number of the pixels arranged along each row of said limited search region only within a range confined by said validated group of the consecutive rows and having the pixel value of “

1”

or greater,

select a group of consecutive columns each having the column index greater than a predetermined column threshold, and

further delimit said limited search region to a region confined by the selected group of the consecutive columns and said validated group of the consecutive rows.

27. A target moving object tracking device as set forth in claim 22 or 23, wherein said position estimating means is configured to

obtain a column index which is the number of the pixels arranged along each column of said limited search region and having the pixel value of “

1”

or greater,

select at least one group of consecutive columns each having the column index greater than a predetermined column threshold,

validate only one group of the consecutive columns which is closer to said estimated position of the target moving object, when two or more groups of the consecutive columns are selected,

obtain a row index which is the number of the pixels arranged along each row of said limited search region only within a range confined by said validated group of the consecutive columns and having the pixel value of “

1”

or greater,

select a group of consecutive rows each having the row index greater than a predetermined row threshold, and

further delimit said limited search region to a region confined by the selected group of the consecutive rows and the validated group of the consecutive columns.