State estimation apparatus, state estimation method, and integrated circuit

• US 9,792,697 B2
• Filed: 02/05/2015
• Issued: 10/17/2017
• Est. Priority Date: 03/27/2014
• Status: Active Grant

• Current Assignee: Megachips Corporation
• Original Assignee: Megachips Corporation
• Inventors: Nagamine, Kenta, Hasegawa, Hiromu
• Primary Examiner(s): CHEN, XUEMEI G
• Application Number:
  US14/614,800
• Publication Number:
  US 20150278601A1
• Time in Patent Office: 985 Days
• Field of Search: 34235752, 348135, 348169, 348335, 382100, 382103, 382195, 463 1, 704236
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters

Abstract

A state estimation apparatus appropriately estimates the internal state of an observation target by calculating a likelihood from observation data, and tracks, for example, multiple objects in a moving image and detects a new object and adds the object as a tracking target in an appropriate manner. A labeling unit detects a closed area from an observation image, and adds a label number to the closed area to generate a label image. A likelihood obtaining unit generates an object-erased image for new object detection by erasing image areas corresponding to all the currently-tracked objects. The apparatus performs the process for detecting a new object using the object-erased image for new object detection based on label numbers. The apparatus is appropriately prevented from erroneously determining that an area of the object-erased image for new object detection corresponding to the currently-tracked object and remaining after the erasure corresponds to a new object.

Claims

1. A state estimation apparatus for tracking a plurality of objects, the apparatus comprising:

observation obtaining circuitry configured to obtain, at each predetermined time interval, an observation image that is observation data obtained from an observable event, and obtain an extracted-feature image representing a predetermined feature quantity extracted from the obtained observation image;

labeling circuitry configured to extract a closed image area from the observation image obtained by the observation obtaining circuitry, and add a label number to the extracted closed image area to generate a label image;

object information management circuitry configured to add an object number to each target object to be tracked, and manage the object number added to each target object;

prediction circuitry configured to perform, for each object that is to be tracked at a current time t, prediction using posterior probability distribution data, and generate predictive probability distribution data for each object that is to be tracked at the current time t, the posterior probability distribution data indicating a probability distribution of an internal state of the object at a preceding time t−

1, the predictive probability distribution data indicating a probability distribution of an internal state of the object at the current time t;

likelihood obtaining circuitry configured to;

(A) obtain, for each object determined as being tracked at the preceding time t−

1, particles based on the predictive probability distribution data, and determine an area to be erased from the extracted-feature image based on the obtained particles to generate an object erased image for tracking or for a new object detection, the object-erased image for tracking being an image from which an area corresponding to an object other than a processing target object selected from the objects that are to be tracked at the current time t has been erased;

the object-erased image for new object detection being an image from which an area corresponding to each of the objects that are to be tracked at the current time t has been erased,

(B) generate particles for the new object detection, use the particles for the new object detection to process the object-erased image to obtain a new object candidate area, and determine whether the new object candidate area corresponds to a new object that is to be a new tracking target at the current time t based on the obtained new object candidate area and the label image to determine a target object to be tracked at the time t, and

(C) calculate a final likelihood based on a likelihood that is calculated for each object determined to be a tracking target object at the time t by using the particles generated based on the predictive probability distribution data to process the object-erased image, and update the predictive probability distribution data obtained by the prediction circuitry based on the calculated final likelihood to generate updated probability distribution data;

posterior probability distribution estimation circuitry configured to estimate, for each object determined to be a tracking target object at the time t, posterior probability distribution data for a state of the event from the updated probability distribution data and the final likelihood obtained by the likelihood obtaining circuitry; and

prior probability distribution output circuitry configured to output probability distribution data based on the posterior probability distribution data estimated by the posterior probability distribution estimation circuitry as prior probability distribution data at a next time t+1.

2. The state estimation apparatus according to claim 1, wherein

the likelihood obtaining circuitry includes

erasing circuitry configured to obtain, for each object determined as being tracked at the preceding time t−

1, particles based on the predictive probability distribution data, and determine an area to be erased from the extracted-feature image based on the obtained particles to generate an object-erased image for tracking or for a new object detection, the object-erased image for tracking being an image from which an area corresponding to an object other than a processing target object selected from the objects that are to be tracked at the current time t has been erased, the object-erased image for new object detection being an image from which an area corresponding to each of the objects that are to be tracked at the current time t has been erased,

likelihood calculation circuitry configured to obtain, for each object determined as being tracked at the preceding time t−

1, particles based on the predictive probability distribution data, and calculate a likelihood for each object that is to be tracked at the current time t by using the obtained particles to process the object-erased image for tracking, and configured to generate particles for new object detection and calculate a likelihood for a new object by using the obtained particles to process the object-erased image for new object detection,

new object candidate area detection circuitry configured to use the particles for new object detection to process the object-erased image for new object detection to obtain the new object candidate area,

gravity center detection circuitry configured to detect a gravity center of the new object candidate area,

new object detection circuitry configured to determine whether the new object candidate area corresponds to a new object that is to be a new tracking target at the current time t based on the gravity center of the new object candidate area detected by the gravity center detection circuitry and the label image,

tracking object determination circuitry configured to determine a tracking target object at the time t based on a result of determination performed by the new object detection circuitry, and

final determination circuitry configured to calculate a final likelihood based on the likelihood obtained by using the particles generated for each object determined as being tracked at the time t, based on the predictive probability distribution data to process the object-erased image, and update the predictive probability distribution data obtained by the prediction circuitry based on the calculated final likelihood to obtain updated probability distribution data.

3. The state estimation apparatus according to claim 2, wherein

the tracking object determination circuitry obtains, as a label determination area, an area of the object-erased image for tracking corresponding to the object that is being tracked at the time t−

1 for which a sum of likelihoods calculated at the time t exceeds a predetermined value, determines whether an area of the label image corresponding to the label determination area has a label number, and

when determining that the area of the label image corresponding to the label determination area has a label number, the tracking object determination circuitry outputs object information associating an object number of the object and the label number to the object information management circuitry, and

the object information management circuitry stores the object information,

the new object detection circuitry determines whether the label image includes a label number added to a pixel at the same coordinate position as the gravity center of the new object candidate area, and

(A) when the label image includes a label number added to a pixel at the same coordinate position as the gravity center of the new object candidate area, the new object detection circuitry obtains the label number as a first label number, and

determines whether the first label number matches an object number included in the object information stored in the object information management circuitry, and

(A1) determines that no new object is detected when the first label number matches the object number included in the object information, and

(A2) determines that the new object candidate area is a candidate for a new object area when the first label number does not match the object number included in the object information,

(B) when the label image does not include a label number added to a pixel at the same coordinate position as the gravity center of the new object candidate area, the new object detection circuitry sets a search area with a predetermined size including the gravity center, and performs a search process for determining whether an area of the label image corresponding to the search area includes an image area having a label number matching an object number included in the object information stored in the object information management circuitry, and determines that no new object is detected when the area of the label image corresponding to the search area includes an image area having a label number matching an object number included in the object information,

the tracking object determination circuitry calculates a sum of likelihoods obtained by using the particles for new object detection to process the object-erased image for new object detection when the new object detection circuitry determines that the new object candidate area is a new object area, and determines that a new object is detected when the sum of likelihoods exceeds a predetermined threshold, and

the final determination circuitry uses the likelihood obtained by using the particles for new object detection to process the object-erased image for new object detection as a likelihood for a new object in calculating the final likelihood, and uses probability distribution data obtained by using the particles for new object detection to process the object-erased image for new object detection as probability distribution data for a new tracking target in obtaining the updated probability distribution data.

4. The state estimation apparatus according to claim 1, wherein the new object detection circuitry sets an area including the gravity center and including the new object candidate area as the search area.

5. The state estimation apparatus according to claim 1, wherein

the new object detection circuitry sets an oval area including the gravity center as the search area.

6. The state estimation apparatus according to claim 1, wherein

the new object detection circuitry sets a rectangular area including the gravity center as the search area.

7. The state estimation apparatus according to claim 1, wherein

the new object detection circuitry uses the gravity center as a start point of the search process in the search area, and sets image areas to undergo the search process sequentially from an image area with a smaller distance from the gravity center to an image area with a greater distance from the gravity center.

8. The state estimation apparatus according to claim 7, wherein

the new object detection circuitry ends the search process when detecting an image area with a label number matching an object number included in the object information in the area of the label image corresponding to the search area.

9. A state estimation method for tracking a plurality of objects, the method comprising:

(1) obtaining, at each predetermined time interval, an observation image that is observation data obtained from an observable event, and obtaining an extracted-feature image representing a predetermined feature quantity extracted from the obtained observation image;

(2) extracting a closed image area from the observation image obtained in the step of obtaining the observation image, and adding a label number to the extracted closed image area to generate a label image;

(3) adding an object number to each target object to be tracked, and managing the object number added to each target object;

(4) performing, for each object that is to be tracked at a current time t, prediction using posterior probability distribution data, and generating predictive probability distribution data for each object that is to be tracked at the current time t, the posterior probability distribution data indicating a probability distribution of an internal state of the object at a preceding time t−

1, the predictive probability distribution data indicating a probability distribution of an internal state of the object at the current time t;

(5a) obtaining, for each object determined as being tracked at the preceding time t−

1, particles based on the predictive probability distribution data, determining an area to be erased from the extracted-feature image based on the obtained particles to generate an object-erased image for tracking or for a new object detection;

the object-erased image for tracking being an image from which an area corresponding to an object other than a processing target object selected from the objects that are to be tracked at the current time t has been erased, the object-erased image for new object detection being an image from which an area corresponding to each of the objects that are to be tracked at the current time t has been erased, (5b) generating particles for the new object detection, using the particles for the new object detection to process the object-erased image to obtain a new object candidate area, and determining whether the new object candidate area corresponds to a new object that is to be a new tracking target at the current time t based on the obtained new object candidate area and the label image to determine a target object to be tracked at the time t, and (5c) calculating a final likelihood based on a likelihood that is calculated for each object determined to be a tracking target object at the time t by using the particles generated based on the predictive probability distribution data to process the object-erased image, and updating the predictive probability distribution data obtained in the step of performing the prediction based on the calculated final likelihood to generate updated probability distribution data;

(6) estimating, for each object determined to be a tracking target object at the time t, posterior probability distribution data for a state of the event from the updated probability distribution data and the final likelihood obtained in the step of calculating the final likelihood; and

(7) outputting probability distribution data based on the posterior probability distribution data estimated in the step of estimating the posterior probability distribution data as prior probability distribution data at a next time t+1.

10. An integrated circuit for performing a state estimation process for tracking a plurality of objects, the integrated circuit comprising:

observation obtaining circuitry configured to obtain, at each predetermined time interval, an observation image that is observation data obtained from an observable event, and obtain an extracted-feature image representing a predetermined feature quantity extracted from the obtained observation image;

labeling circuitry configured to extract a closed image area from the observation image obtained by the observation obtaining circuitry, and add a label number to the extracted closed image area to generate a label image;

object information management circuitry configured to add an object number to each target object to be tracked, and manage the object number added to each target object;

prediction circuitry configured to perform, for each object that is to be tracked at a current time t, prediction using posterior probability distribution data, and generate predictive probability distribution data for each object that is to be tracked at the current time t, the posterior probability distribution data indicating a probability distribution of an internal state of the object at a preceding time t−

1, the predictive probability distribution data indicating a probability distribution of an internal state of the object at the current time t;

likelihood obtaining circuitry configured to;

(A) obtain, for each object determined as being tracked at the preceding time t−

1, particles based on the predictive probability distribution data, and determine an area to be erased from the extracted-feature image based on the obtained particles to generate an object-erased image for tracking or for a new object detection, the object-erased image for tracking being an image from which an area corresponding to an object other than a processing target object selected from the objects that are to be tracked at the current time t has been erased, the object-erased image for new object detection being an image from which an area corresponding to each of the objects that are to be tracked at the current time t has been erased,

(B) generate particles for the new object detection, use the particles for the new object detection to process the object-erased image to obtain a new object candidate area, and determine whether the new object candidate area corresponds to a new object that is to be a new tracking target at the current time t based on the obtained new object candidate area and the label image to determine a target object to be tracked at the time t, and

(C) calculate a final likelihood based on a likelihood that is calculated for each object determined to be a tracking target object at the time t by using the particles generated based on the predictive probability distribution data to process the object-erased image, and update the predictive probability distribution data obtained by the prediction circuitry based on the calculated final likelihood to generate updated probability distribution data;

posterior probability distribution estimation circuitry configured to estimate, for each object determined to be a tracking target object at the time t, posterior probability distribution data for a state of the event from the updated probability distribution data and the final likelihood obtained by the likelihood obtaining circuitry; and

prior probability distribution output circuitry configured to output probability distribution data based on the posterior probability distribution data estimated by the posterior probability distribution estimation circuitry as prior probability distribution data at a next time t+1.