STATE ESTIMATION APPARATUS, STATE ESTIMATION METHOD, AND INTEGRATED CIRCUIT

• US 20150278601A1
• Filed: 02/05/2015
• Published: 10/01/2015
• Est. Priority Date: 03/27/2014
• Status: Active Grant

• Current Assignee: Megachips Corporation
• Original Assignee: Megachips Corporation
• Inventors: Hasegawa, Hiromu, Nagamine, Kenta
• Granted Patent: US 9,792,697 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters

Abstract

A state estimation apparatus appropriately estimates the internal state of an observation target by calculating a likelihood from observation data, and tracks, for example, multiple objects in a moving image and detects a new object and adds the object as a tracking target in an appropriate manner. A labeling unit detects a closed area from an observation image, and adds a label number to the closed area to generate a label image. A likelihood obtaining unit generates an object-erased image for new object detection by erasing image areas corresponding to all the currently-tracked objects. The apparatus performs the process for detecting a new object using the object-erased image for new object detection based on label numbers. The apparatus is appropriately prevented from erroneously determining that an area of the object-erased image for new object detection corresponding to the currently-tracked object and remaining after the erasure corresponds to a new object.

Claims

1. A state estimation apparatus for tracking a plurality of objects, the apparatus comprising:

an observation obtaining unit configured to obtain, at predetermined time intervals, an observation image that is observation data obtained from an observable event, and obtain an extracted-feature image representing a predetermined feature quantity extracted from the observation image;

a labeling unit configured to extract a closed image area from the observation image obtained by the observation obtaining unit, and add a label number to the extracted area to generate a label image;

an object information management unit configured to add an object number to each target object to be tracked, and manage the object number added to each target object;

a prediction unit configured to perform, for each object that is currently being tracked, prediction using posterior probability distribution data, and generate predictive probability distribution data for each currently-tracked object, the posterior probability distribution data indicating a probability distribution of an internal state of the object at a preceding time t−

1, the predictive probability distribution data indicating a probability distribution of an internal state of the object at a current time t; and

a likelihood obtaining unit configured to

obtain, for each object determined as currently being tracked at the preceding time t−

1, particles based on the predictive probability distribution data, determine an area to be erased from the extracted-feature image based on the obtained particles to generate an object-erased image for tracking or an object-erased image for new object detection, the object-erased image for tracking being an image from which an area corresponding to an object other than a processing target object selected from the currently-tracked objects has been erased, the object-erased image for new object detection being an image from which an area corresponding to each of the currently-tracked objects has been erased,

generate particles for new object detection, use the particles for new object detection to process the object-erased image to obtain a new object candidate area, and determine whether the new object candidate area corresponds to a new object that is to be a new tracking target at the current time t based on the obtained new object candidate area and the label image to determine a target object to be tracked at the time t, and

calculate a final likelihood based on a likelihood that is calculated for each object determined to be a tracking target object at the time t by using the particles generated based on the predictive probability distribution data to process the object-erased image, and update the predictive probability distribution data obtained by the prediction unit based on the calculated final likelihood to generate updated probability distribution data;

a posterior probability distribution estimation unit configured to estimate, for each object determined to be a tracking target object at the time t, posterior probability distribution data for a state of the event from the updated probability distribution data and the final likelihood obtained by the likelihood obtaining unit; and

a prior probability distribution output unit configured to output probability distribution data based on the posterior probability distribution data estimated by the posterior probability distribution estimation unit as prior probability distribution data at a next time t+1.

2. The state estimation apparatus according to claim 1, wherein

the likelihood obtaining unit includes

an erasing unit configured to obtain, for each object determined as currently being tracked at the preceding time t−

1, particles based on the predictive probability distribution data, determine an area to be erased from the extracted-feature image based on the obtained particles to generate an object-erased image for tracking or an object-erased image for new object detection, the object-erased image for tracking being an image from which an area corresponding to an object other than a processing target object selected from the currently-tracked objects has been erased, the object-erased image for new object detection being an image from which an area corresponding to each of the currently-tracked objects has been erased,

a likelihood calculation unit configured to obtain, for each object determined as currently being tracked at the preceding time t−

1, particles based on the predictive probability distribution data, and calculate a likelihood for each object that is currently being tracked by using the obtained particles to process the object-erased image for tracking, and configured to generate particles for new object detection and calculate a likelihood for a new object by using the obtained particles to process the object-erased image for new object detection,

a new object candidate area detection unit configured to use the particles for new object detection to process the object-erased image for new object detection to obtain the new object candidate area,

a gravity center detection unit configured to detect a gravity center of the new object candidate area,

a new object detection unit configured to determine whether the new object candidate area corresponds to a new object that is to be a new tracking target at the current time t based on the gravity center of the new object candidate area detected by the gravity center detection unit and the label image,

a tracking object determination unit configured to determine a tracking target object at the time t based on a result of determination performed by the new object detection unit, and

a final determination unit configured to calculate a final likelihood based on the likelihood obtained by using the particles generated for each object determined as currently being tracked at the time t, based on the predictive probability distribution data to process the object-erased image, and update the predictive probability distribution data obtained by the prediction unit based on the calculated final likelihood to obtain updated probability distribution data.

3. The state estimation apparatus according to claim 2, wherein

the tracking object determination unit obtains, as a label determination area, an area of the object-erased image for tracking corresponding to the object that is being tracked at the time t−

1 for which a sum of likelihoods calculated at the time t exceeds a predetermined value, determines whether an area of the label image corresponding to the label determination area has a label number, and

when determining that the label image area corresponding to the label determination area has a label number, the tracking object determination unit outputs object information associating an object number of the object and the label number to the object information management unit, and

the object information management unit stores the object information,

the new object detection unit determines whether the label image includes a label number added to a pixel at the same coordinate position as the gravity center of the new object candidate area, and

(A) when the label image includes a label number added to a pixel at the same coordinate position as the gravity center of the new object candidate area, the new object detection unit obtains the label number as a first label number, and

determines whether the first label number matches an object number included in the object information stored in the object information management unit, and

(A1) determines that no new object is detected when the first label number matches the object number included in the object information, and

(A2) determines that the new object candidate area is a candidate for a new object area when the first label number does not match the object number included in the object information,

(B) when the label image does not include a label number added to a pixel at the same coordinate position as the gravity center of the new object candidate area, the new object detection unit sets a search area with a predetermined size including the gravity center, and performs a search process for determining whether an area of the label image corresponding to the search area includes an image area having a label number matching an object number included in the object information stored in the object information management unit, and determines that no new object is detected when the area of the label image corresponding to the search area includes an image area having a label number matching an object number included in the object information,

the tracking object determination unit calculates a sum of likelihoods obtained by using the particles for new object detection to process the object-erased image for new object detection when the new object detection unit determines that the new object candidate area is a new object area, and determines that a new object is detected when the sum of likelihoods exceeds a predetermined threshold, and

the final determination unit uses the likelihood obtained by using the particles for new object detection to process the object-erased image for new object detection as a likelihood for a new object in calculating the final likelihood, and uses probability distribution data obtained by using the particles for new object detection to process the object-erased image for new object detection as probability distribution data for a new tracking target in obtaining the updated probability distribution data.

4. The state estimation apparatus according to claim 3, wherein the new object detection unit sets an area including the gravity center and including the new object candidate area as the search area.

5. The state estimation apparatus according to claim 3, wherein

the new object detection unit sets an oval area including the gravity center as the search area.

6. The state estimation apparatus according to claim 3, wherein

the new object detection unit sets a rectangular area including the gravity center as the search area.

7. The state estimation apparatus according to claim 3, wherein

the new object detection unit uses the gravity center position as a start point of the search process in the search area, and sets image areas to undergo the search process sequentially from an image area with a smaller distance from the gravity center to an image area with a greater distance from the gravity center.

8. The state estimation apparatus according to claim 7, wherein

the new object detection unit ends the search process when detecting an image area with a label number matching an object number included in the object information in the area of the label image corresponding to the search area.

9. A state estimation method for tracking a plurality of objects, the method comprising:

obtaining, at predetermined time intervals, an observation image that is observation data obtained from an observable event, and obtaining an extracted-feature image representing a predetermined feature quantity extracted from the observation image;

extracting a closed image area from the observation image obtained in the step of obtaining the observation image, and adding a label number to the extracted area to generate a label image;

adding an object number to each target object to be tracked, and managing the object number added to each target object;

performing, for each object that is currently being tracked, prediction using posterior probability distribution data, and generating predictive probability distribution data for each currently-tracked object, the posterior probability distribution data indicating a probability distribution of an internal state of the object at a preceding time t−

1, the predictive probability distribution data indicating a probability distribution of an internal state of the object at a current time t;

obtaining, for each object determined as currently being tracked at the preceding time t−

1, particles based on the predictive probability distribution data, determining an area to be erased from the extracted-feature image based on the obtained particles to generate an object-erased image for tracking or an object-erased image for new object detection, the object-erased image for tracking being an image from which an area corresponding to an object other than a processing target object selected from the currently-tracked objects has been erased, the object-erased image for new object detection being an image from which an area corresponding to each of the currently-tracked objects has been erased, generating particles for new object detection, using the particles for new object detection to process the object-erased image to obtain a new object candidate area, and determining whether the new object candidate area corresponds to a new object that is to be a new tracking target at the current time t based on the obtained new object candidate area and the label image to determine a target object to be tracked at the time t, and calculating a final likelihood based on a likelihood that is calculated for each object determined to be a tracking target object at the time t by using the particles generated based on the predictive probability distribution data to process the object-erased image, and updating the predictive probability distribution data obtained in the step of performing the prediction based on the calculated final likelihood to generate updated probability distribution data;

estimating, for each object determined to be a tracking target object at the time t, posterior probability distribution data for a state of the event from the updated probability distribution data and the final likelihood obtained in the step of calculating the final likelihood; and

outputting probability distribution data based on the posterior probability distribution data estimated in the step of estimating the posterior probability distribution data as prior probability distribution data at a next time t+1.

10. An integrated circuit for performing a state estimation process for tracking a plurality of objects, the integrated circuit comprising:

an observation obtaining unit configured to obtain, at predetermined time intervals, an observation image that is observation data obtained from an observable event, and obtain an extracted-feature image representing a predetermined feature quantity extracted from the observation image;

a labeling unit configured to extract a closed image area from the observation image obtained by the observation obtaining unit, and add a label number to the extracted area to generate a label image;

an object information management unit configured to add an object number to each target object to be tracked, and manage the object number added to each target object;

a prediction unit configured to perform, for each object that is currently being tracked, prediction using posterior probability distribution data, and generate predictive probability distribution data for each currently-tracked object, the posterior probability distribution data indicating a probability distribution of an internal state of the object at a preceding time t−

1, the predictive probability distribution data indicating a probability distribution of an internal state of the object at a current time t; and

a likelihood obtaining unit configured to

obtain, for each object determined as currently being tracked at the preceding time t−

1, particles based on the predictive probability distribution data, determine an area to be erased from the extracted-feature image based on the obtained particles to generate an object-erased image for tracking or an object-erased image for new object detection, the object-erased image for tracking being an image from which an area corresponding to an object other than a processing target object selected from the currently-tracked objects has been erased, the object-erased image for new object detection being an image from which an area corresponding to each of the currently-tracked objects has been erased,

generate particles for new object detection, use the particles for new object detection to process the object-erased image to obtain a new object candidate area, and determine whether the new object candidate area corresponds to a new object that is to be a new tracking target at the current time t based on the obtained new object candidate area and the label image to determine a target object to be tracked at the time t, and

calculate a final likelihood based on a likelihood that is calculated for each object determined to be a tracking target object at the time t by using the particles generated based on the predictive probability distribution data to process the object-erased image, and update the predictive probability distribution data obtained by the prediction unit based on the calculated final likelihood to generate updated probability distribution data;

a posterior probability distribution estimation unit configured to estimate, for each object determined to be a tracking target object at the time t, posterior probability distribution data for a state of the event from the updated probability distribution data and the final likelihood obtained by the likelihood obtaining unit; and

a prior probability distribution output unit configured to output the prior probability distribution data based on the posterior probability distribution data estimated by the posterior probability distribution estimation unit as prior probability distribution data at a next time t+1.