Tracking apparatus for tracking image in local region

• US 5,838,365 A
• Filed: 08/29/1995
• Issued: 11/17/1998
• Est. Priority Date: 09/20/1994
• Status: Expired due to Term

• Current Assignee: Fujitsu Limited
• Original Assignee: Fujitsu Limited
• Inventors: Sato, Masahiko, Sawasaki, Naoyuki
• Primary Examiner(s): PEESO, THOMAS R
• Application Number:
  US08/521,076
• Time in Patent Office: 1,176 Days
• Field of Search: 364/516, 382/103, 342/189, 342/378, 348/169
• US Class Current: 348/169
• CPC Class Codes: G06F 17/153 Multidimensional correlation or convolution
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/231 using full search
• CPC Class Codes: G06T 7/248 involving reference images or patches

Abstract

A local region image tracking apparatus continuously tracks an arbitrary local region within a search image that is described by search image data by comparing the search image data to be subjected to a tracking process out of input image data picked up by image pickup means with reference image data related to the arbitrary local region. The local region image tracking apparatus includes an image data bus transferring the input image data from the image pickup means, and a plurality of correlation tracking processors coupled in parallel to the image data bus and respectively capable of independently carrying out the tracking process with respect to at least one search image, where the tracking process with respect to one or a plurality of search images is distributively carried out by the correlation tracking processors.

Claims

1. A local region image tracking apparatus, comprising:

an image data bus coupled to an image pickup unit and transferring therefrom input image data representing an input image; and

a plurality of correlation tracking processors, coupled in parallel to said image data bus, each correlation tracking processor capable of independently tracking a respective local region of the input image each correlation tracking processor including;

at least three image memories;

a selector that selects, from the at least three image memories, an input image memory that stores received input image data included in the input image data, a search image memory that stores search image data representing a search image included in the input image, and a reference image memory that stores reference image data, the search image including a local region to be tracked and the reference image data being related to the local region;

an address generator that generates a search image address to identify the search image memory and a reference image address to identify the reference image memory;

a correlation computing unit that computes a correlation value indicating a correlation between the reference image data and the search image data; and

a peak position detector that detects a peak position within the input image, the peak position having a correlation value indicating a highest correlation.

2. The local region image tracking apparatus as claimed in claim 1, wherein the input image data includes a plurality of frames and wherein the selector controls, for each of the correlation tracking processors, in accordance with a predetermined storage sequence:

storage of the received input image data to the input image memory for tracking with each frame of the input image data;

storage of each frame of the search image data to the search image memory; and

updating of the reference image data at arbitrary frame intervals.

3. The local region image tracking apparatus as claimed in claim 1, wherein:

each of the correlation tracking processors further includes a fourth image memory; and

wherein the selector controls, for each of the correlation processors, in accordance with a predetermined storage sequence;

storage of the received input image data to the input image memory for tracking with each frame of the received input image data;

storage of each frame of the search image data to the search image memory; and

storage of the reference image data to the reference image memory and successive reference image data to at least another image memory.

4. The local region image tracking apparatus as claimed in claim 1, wherein:

the reference image data represents a reference image consisting of a square array of pixels having 16 pixels on each side thereof; and

the search image is larger than the reference image and includes a plurality of partial region images.

5. The local region image tracking apparatus as claimed in claim 1, wherein the image pickup unit generates the input image data in accordance with a first number of processing bits, and further comprising a converter that converts the input image data into converted image data in accordance with a second number of processing bits, the second number corresponding to processing by said plurality of correlation tracking processors.

6. The local region image tracking apparatus as claimed in claim 5, wherein:

the first number is larger than the second number;

the input image data is color image data with each color represented by the first number of processing bits; and

the converted image data is pseudo-color data with each pseudo-color represented by the second number of processing bits.

7. The local region image tracking apparatus as claimed in claim 5, wherein said converter includes a memory that prestores a conversion table.

8. The local region image tracking apparatus as claimed in claim 5, wherein:

the image pickup unit includes a memory that stores a conversion table; and

said converter converts the input image data in accordance with the conversion table.

13. The local region image tracking apparatus as claimed in claim 8, wherein for each of the correlation tracking processors the control computer interface comprises

a first first-in-first-out memory that successively stores instructions from said control computer, the instructions controlling the correlation tracking processor to track the local region without intervention from said control computer.

14. The local region image tracking apparatus as claimed in claim 13, wherein for each of the correlation tracking processors:

the control computer interface comprises a second first-in-first-out memory that successively stores at least one of the correlation value from the correlation computing unit and the peak position from the peak position detector; and

said control computer reads information stored in the second first-in-first-out memory at a second arbitrary time.

9. A local region image tracking apparatus, comprising:

an image data bus coupled to an image pickup unit and transferring therefrom input image data representing an input image; and

a plurality of correlation tracking processors coupled in parallel to said image data bus and distributively tracking the local region by comparing, for each local region, search image data representing a search image including the local region with reference image data related to the local region, each correlation tracking processor including;

at least three image memories;

a selector that selects from the image memories, for each of a successive series of iterations, an input memory that stores received input image data included in the input image data, a search image memory that stores search image data representing a search image included in the input image, and a reference image memory that stores reference image data, the search image including a local region to be tracked and the reference image data being elated to the local region;

a first-in-first-out memory that temporarily and successively stores a plurality of successive reference image data read from the image memories and a plurality of successive search image data read from the image memories; and

a correlation computing unit that computes a correlation value indicating a correlation between a successive reference image data read from the first-in-first-out memory and a corresponding successive search image data read from the first-in-first-out-memory.

10. A local region image tracking apparatus for distributive local region image tracking, comprising:

an image data bus coupled to an image pickup unit and transferring therefrom input image data representing an input image;

a plurality of correlation tracking processors coupled in parallel to said image data bus, each correlation tracking processor including a plurality of image memories, a selected image memory thereof storing reference image data relating to a local region included in the input image and to be tracked;

a control computer bus having said plurality of correlation tracking processors coupled thereto in parallel; and

a control computer coupled to said control computer bus and communicating therethrough with each of the correlation tracking processors to control updating of the reference image data and selection and switching of the image memories in the correlation tracking processor.

11. The local region image tracking apparatus as claimed in claim 10, wherein for each of the correlation tracking processors the plurality of image memories includes at least three image memories, and each of the correlation tracking processors further comprises:

a selector that selects from the image memories an input image memory that stores received input image data included in the input image data, a search image memory that stores search image data representing a search image included in the input image, and a reference image memory that stores the reference image data;

a correlation computing unit that computes a correlation value indicating a correlation between the reference image data and the search image data;

a peak position detector that detects a peak position within the input image, the peak position having a correlation value indicating a highest correlation; and

a control computer interface, coupled to the selector means and to said control computer bus.

12. The local region image tracking apparatus as claimed in claim 11, wherein for each of the correlation tracking processors said control computer registers, through said control computer bus, an instruction in the control computer interface to update the reference image date at a first arbitrary time.

15. The local region image tracking apparatus as claimed in claim 11, wherein for each of the correlation tracking processors the control computer interface comprises:

a storage device that stores a correlation value distribution comprising a plurality of correlation values obtained from the correlation computing unit during tracking of the local region; and

an adder that adds the first correlation value distribution to a second correlation value distribution obtained with respect to tracking the local region.

16. The local region image tracking apparatus as claimed in claim 15, wherein for each of the correlation tracking processors the control computer interface is controlled by an instruction from said control computer indicating whether to add the first and second correlation value distributions.

17. The local region image tracking apparatus as claimed in claim 11, wherein for each of the correlation tracking processors said control computer controls the control computer interface, for tracking of the local region with respect to a reference image having an arbitrary magnifications, by:

generating, for each of a plurality of fixed-size adjacent reference images, a correlation value distribution obtained from the correlation computing unit in accordance with the adjacent reference image, the adjacent reference image being represented by the reference image data;

obtaining a sum of all of the correlation value distributions for the respective adjacent reference images; and

obtaining a peak position of the sum.

18. The local region image tracking apparatus as claimed in claim 11, wherein for each of the correlation tracking processors said control computer registers, through said control computer bus, an instruction in the control computer interface to read at a first arbitrary time data representing the local region from the search image memory and the reference image memory.

19. The local region image tracking apparatus as claimed in claim 11, wherein for each of the correlation tracking processors the control computer interface comprises a storage that stores at least one of the correlation value from the correlation computing unit and the peak position from the peak position detector, said control computer reading information stored in the storage at a first arbitrary time through said control computer bus.

20. The local region image tracking apparatus as claimed in claim 10, wherein each of the correlation tracking processors tracking processors includes:

a first-in-first-out memory that temporarily and successively stores a plurality of successive reference image data read from the image memories and a plurality of successive search image data read from the image memories; and

a correlation computing unit that computes a correlation value on the basis of the successive search image data and the successive reference image data successively stored in the first-in-first-out memory.

21. A local region image tracking apparatus which continuously tracks an arbitrary local region within a search image that is described by search image data by comparing the search image data to be subjected to a tracking process out of input image data picked up by image pickup means with reference image data related to the arbitrary local region, said local region image tracking apparatus comprising:

at least three image memories;

selector means for selectively switching said image memories to an input image memory which stores the input image data, a search image memory which stores the search image data, and a reference image memory which stores the reference image data;

correlation computing means for computing a correlation value which indicates a correlation between the reference image data stored in the reference image memory and the search image data stored in the search image memory by carrying cut a correlation computing process;

peak position detecting means for obtaining a peak position of the correlation value from said correlation computing means; and

image memory managing means for controlling said selector means based on the peak position from said peak position detecting means and successively storing a history of the reference image data in said image memory, so as to update the reference image data based on a history of the peak position of the correlation value stored within the search image memory.

22. The local region image tracking apparatus as claimed in claim 21, which further comprises:

correlation comparing means for updating the reference image data by setting the image memory which stores the reference image data having a maximum correlation value as a result of the tracking process as the reference image memory via said image memory managing means based on the peak position from said peak position detecting means.

23. The local region image tracking apparatus as claimed in claim 21, which further comprises:

correlation evaluating means for evaluating the correlation of the reference image data stored in the reference image memory as a result of the tracking process based on the peak position from said peek position detecting means, and updating the reference image data based on the evaluated correlation value.

24. The local region image tracking apparatus as claimed in claim 23, wherein said correlation evaluating means updates the reference image data to most recent reference image data only when the correlation value becomes less than or equal to a predetermined threshold value during the tracking process.

25. The local region image tracking apparatus as claimed in claim 23, which further comprises:

correlation comparing means for updating the reference image data by setting the image memory which stores the reference image data having a maximum correlation value as a result of the tracking process as the reference image memory via said image memory managing means based on the peak position from said peak position detecting means only when said correlation evaluating means detects that the correlation value has become less than a predetermined threshold value during the tracking process.

27. The local region image tracking apparatus as claimed in claim 25, wherein said memory means stores the input image data, and said reference image data generating means compares the input image data and a preset background image data and generates the reference image data based on a difference between the compared image data.

26. A local region image tracking apparatus which continuously tracks an arbitrary local region within a search image that is described by search image data by comparing the search image data to be subjected to a tracking process out of input image data picked up by image pickup means that is supported on moving means with reference image data related to the arbitrary local region, said local region image tracking apparatus comprising:

memory means for storing at least the search image data and the reference image data;

reference image data generating means for automatically generating the reference image data from the input image data;

address generating means for generating addresses for reading the search image data and the reference image data from said memory means;

correlation computing means for computing a correlation value which indicates a correlation between the reference image data and the search image data read from said memory means by carrying out a correlation computing process; and

control means for outputting a moving instruction which moves said image pickup means by said moving means so as to track the arbitrary local region based on a peak position of the correlation value obtained from said correlation computing means.

28. The local region image tracking apparatus as claimed in claim 26, wherein said reference image data generating means divides the input image data at predetermined time intervals into a plurality of regions, obtains the regions in which motion is generated by carrying out the correlation computing process in said correlation computing means with respect to each of the divided regions, and generates the reference image data based on whether or not a number of regions in which the motion is generated exceeds a predetermined value.

29. The local region image tracking apparatus as claimed in claim 26, wherein said control means updates the reference image data by setting an address of the search image data into said address generating means as an address of the reference image data when the peak value of the correlation value exceeds a predetermined value.

30. The local region image tracking apparatus as claimed in claim 26, wherein said image pickup means comprises zoom means, and said control means controls a zoom value of said zoom means so that the arbitrary local region is gradually picked up with a preset zoom setting value.

31. The local region image tracking apparatus as claimed in claim 30, wherein said control means outputs a signal indicating one of an alarm and a start of recording of the input image data based on the motion in the arbitrary local region and the zoom value.

32. The local region image tracking apparatus as claimed in claim 26, wherein said control means comprises:

means for assigning a label to each of a plurality of local regions to identify each of the local regions; and

means for carrying out the tracking process with respect to a largest local region out of the plurality of labeled local regions.

33. The local region image tracking apparatus as claimed in claim 26, wherein said image pickup means comprises:

a first image pickup device movably supported on said moving means; and

a second image pickup device having zoom means,

said control means outputting a moving instruction which controls said first image pickup device to track the arbitrary local region and controls a direction of said second image pickup device depending on a zoom ratio.

34. The local region image tracking apparatus as claimed in claim 33, wherein said control means comprises:

means for assigning a label to each of a plurality of local regions to identify each of the local regions; and

means for controlling said first image pickup device to carry out the tracking process with respect to the plurality of labeled local regions and controlling said second image pickup means to zoom up on each of the labeled local regions in sequence.

35. A local region image tracking apparatus which continuously tracks an arbitrary local region within a search image that is described by search image data by comparing the search image data to be subjected to a tracking process out of input image data picked up by image pickup means with reference image data related to the arbitrary local region, said local region image tracking apparatus comprising:

memory means for storing at least the search image data and the reference image data;

reference image data generating means for automatically generating the reference image data related to a plurality of local regions from the input image data;

labeling means for labeling the plurality of local regions to identify the plurality of local regions;

address generating means for generating addresses for reading the search image data and the reference image data from said memory means;

correlation computing means for computing a correlation value which indicates a correlation between the reference image data and the search image data read from said memory means by carrying out a correlation computing process with respect to each of the plurality of labeled local regions; and

control means for tracking the plurality of labeled local regions based on a peak position of the correlation value obtained from said correlation computing means.

36. The local region image tracking apparatus as claimed in claim 35, wherein said memory means stores the input data, and said reference image data generating means compares a preset background image data and the input image data and generates the reference image data based on a difference between the compared image data.

37. The local region image tracking apparatus as claimed in claim 35, wherein said reference image data generating means divides the input image data at predetermined time intervals into a plurality of regions, obtains the regions in which motion is generated by carrying out the correlation computing process in said correlation computing means with respect to each of the divided regions, and generates the reference image data based on whether or not a number of regions in which the motion is generated exceeds a predetermined value.

38. The local region image tracking apparatus as claimed in claim 35, wherein said control means updates the reference image data for each of the local regions by setting an address of the search image data into said address generating means as an address of the reference image data when the peak value of the correlation value exceeds a predetermined value.

39. The local region image tracking apparatus as claimed in claim 35, wherein said control means outputs a signal indicating one of an alarm and a start of recording of the input image data based on the motion in each of the local regions.

40. A local region image tracking apparatus for continuously tracking a plurality of local regions within a search image represented by search image data, said local region image tracking apparatus comprising:

at least four image memories;

a selector that selects, from the at least four image memories, an input image memory that stores input image data, a search image memory that stores the search image data, and a reference image memory that stores reference image data relating to a selected local region in the plurality of local regions;

a correlation computing unit that computes a correlation value indicating a correlation between the reference image data and the search image data representing a specified local region included in the plurality of local regions;

a peak position detector that obtains a peak position when the correlation value indicates a highest correlation within the input image; and

an image memory manager that controls said selector based on the peak position from said peak position detector.

41. The local region image tracking apparatus as claimed in claim 40, wherein said image memory manager identifies a plurality of successive reference image data stored in at least one image memory included in said at least four image memories, and further comprising a correlation comparator that designates next reference image data in the plurality of successive reference image data when the designated successive reference image data has a corresponding peak position with a maximum correlation value relative to the reference image data and the other successive reference image data.

42. The local region image tracking apparatus as claimed in claim 40, further comprising a correlation evaluator that designates the input image data as next reference image data when the correlation value of the peak position of the search image data satisfies a predetermined condition.

43. The local region image tracking apparatus as claimed in claim 42, wherein the predetermined condition includes the condition that the correlation value of the peak position is greater than or equal to a predetermined threshold value.

44. A local region image tracking apparatus for continuously tracking a local region within a search image represented by search image data, said local region image tracking apparatus comprising:

at least one memory that stores the search image data and reference image data related to the local region, the search image data being included in input image data picked up by a motorized image pickup unit;

a motion detector that automatically generates the reference image data on the basis of the input image data;

an address generator that generates addresses for reading the search image data and the reference image data from the at least one image memory;

a correlation computing unit that computes a correlation value indicating a correlation between the reference image data and the search image data; and

a controller that generates a moving instruction controlling the motorized image pickup unit to track the specified local region based on a peak position of the correlation value obtained from said correlation computing unit.

45. The local region image tracking apparatus as claimed in claim 44, wherein the at least one image memory stores the input image data, and the motion detector compares the input image data with a preset background image data and generates the reference image data based on a difference therebetween.

46. The local region image tracking apparatus as claimed in claim 44, wherein the motion detector:

receives the input image data at a predetermined time interval;

divides the input image data into a plurality of regions including the local region;

identifies at least one motion region from among the plurality of regions when said correlation computing unit computes a respective correlation value indicating motion in the at least one motion region;

compares an enumeration of the at least one motion region with a predetermined region value; and

identifies next reference image data in accordance with a result of comparing the enumeration with the predetermined region value.

47. The local region image tracking apparatus as claimed in claim 46, wherein:

the motorized image pickup unit includes a zoom mechanism responsive to a zoom value generated by said controller; and

said controller adjusts the zoom value by at least one zoom increase value in accordance with a target zoom value when the motion detector determines that the enumeration exceeds the predetermined region value.

48. The local region image tracking apparatus as claimed in claim 44, further comprising a peak position detector that obtains a peak position when the correlation value indicates a highest correlation within the input image, and wherein said controller designates the search image data as next reference image data when the correlation value of the peak position exceeds a predetermined threshold value.

49. The local region image tracking apparatus as claimed in claim 44, wherein said controller includes:

a labeling section that assigns an identifying label to each one of a plurality of regions included in the input image, the plurality of regions including the local region; and

a decision section that determines a largest local region from the plurality of regions and controls said motion detector to track the largest region.

50. The local region image tracking apparatus as claimed in claim 44, wherein the motorized image pickup unit includes:

a first image pickup device supported on a movable support; and

a second image pickup device having a zoom mechanism; and

wherein said controller generates a moving instruction to control the first image pickup device to track the local region and to control the second image pickup device in accordance with a zoom ratio.