SYSTEM AND METHOD FOR REAL-TIME OBJECT RECOGNITION AND POSE ESTIMATION USING IN-SITU MONITORING

• US 20090190798A1
• Filed: 02/15/2008
• Published: 07/30/2009
• Est. Priority Date: 01/25/2008
• Status: Active Grant

• Current Assignee: Sungkyunkwan University Foundation For Corporate Collaboration (SungKyunKwan University)
• Original Assignee: Sungkyunkwan University Foundation For Corporate Collaboration (SungKyunKwan University)
• Inventors: LEE, Jangwon, BAEK, Seung-Min, LEE, Sukhan, LEE, Jeihun
• Granted Patent: US 8,503,760 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06F 18/254 of classification results, e.g. of results related to same input data
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/35 using statistical methods
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/77 using statistical methods
• CPC Class Codes: G06V 10/809 of classification results, e.g. where the classifiers operate on the same input data
• CPC Class Codes: G06V 20/64 Three-dimensional objects

Abstract

Provided are a system and method for real-time object recognition and pose estimation using in-situ monitoring. The method includes the steps of: a) receiving 2D and 3D image information, extracting evidences from the received 2D and 3D image information, recognizing an object by comparing the evidences with model, and expressing locations and poses by probabilistic particles; b) probabilistically fusing various locations and poses and finally determining a location and a pose by filtering inaccurate information; c) generating ROI by receiving 2D and 3D image information and the location and pose from the step b) and collecting and calculating environmental information; d) selecting an evidence or a set of evidences probabilistically by receiving the information from the step c) and proposing a cognitive action of a robot for collecting additional evidence; and e) repeating the steps a) and b) and the steps c) and d) in parallel until a result of object recognition and pose estimation is probabilistically satisfied.

Claims

1. A method for real time object recognition and pose estimation using in-situ monitoring, comprising the steps of:

a) receiving 2D and 3D real time image information, extracting single or multiple evidences from the received 2D and 3D real time image information, recognizing an object by comparing the extracted evidences with model information, and expressing locations and poses of the object by probabilistic particles in a space;

b) probabilistically fusing various locations and poses of the object, which are generated in a particle form, and finally determining a location and a pose of the object by filtering inaccurate information;

c) generating a region of interest (ROI) by receiving 2D and 3D real time image information and the location and pose of the object from the step b) and collecting and calculating real time environmental information;

d) selecting an evidence or a set of evidences probabilistically by receiving information from the step c) and proposing a cognitive action of a robot for collecting additional evidence if selected evidence is not sufficient; and

e) repeating the steps a) and b) and the steps c) and d) in parallel until a result of object recognition and pose estimation is probabilistically satisfied.

2. The method of claim 1, wherein in the step a), the evidence is a feature obtained from image information and a combination of two or more features, where the feature is color, line, and scale invariant feature transform (SIFT).

3. The method of claim 2, wherein in case of using a line feature among extracted evidences, the step a) includes steps of:

a-1) extracting all of lines from input 2D images;

a-2) transforming a 2D line to a 3D line by mapping 3D points corresponding to the extracted lines; and

a-3) generating various locations and poses of the object by comparing the 3D lines with 3D line information on the model pre-stored in a database.

4. The method of claim 2, wherein in the step a) if a SIFT feature is used among extracted evidences, the location and the pose of the object is generated by calculating transformation between the SIFT feature measured from a current frame and a corresponding SIFT feature in a database.

5. The method of claim 2, wherein in case of using a combination of two or more features among extracted evidences, the step a) includes the step of:

generating various locations and poses of the object by comparing various features extracted from a current input image with features of the model stored in a database.

6. The method of claim 1, wherein the step b) includes the steps of:

b-1) generating an observation likelihood from the pose generated using various evidences at the step a);

b-2) estimating particles denoting poses of the object by propagating particles from a previous state using motion information and updating the estimated particles by the observation likelihood; and

b-3) re-sampling the particles according to a weight for expressing particles with different weights fused in the step b-2) as particles with uniform weights.

7. The method of claim 6, wherein in the step b-1), the observation likelihood is generated by an equation:

$p<br><br>\left(Z_t|O_t^{\left[i\right]}\right)=\underset{j=1}{\overset{m}{\sum <br><br>}}<br><br>w_j·<br><br>\exp <br><br>\left[\frac{-1}{2}·<br><br>\underset{i=1}{\overset{4}{\sum <br><br>}}<br><br>\left\{\times <br><br>S_t^{-1}·<br><br>\begin{array}{c}{\left({\mathrm{Ob\_TP}}_t^j-{\mathrm{St\_TP}}_t^i\right)}^T\\ \left({\mathrm{Ob\_TP}}_t^j-{\mathrm{St\_TP}}_t^i\right)\end{array}\right\}\right],$

where O^[j] is a homogeneous transform matrix denoting a pose of an object generated from an observation time t, O_t^[i] is a homogeneous transform matrix denoting a pose of an object, which is estimated from a time t-1 to a time t, w_j is a similarity weight for O^[j] which is a particle generated from a time t, m denotes the number of particles generated at an observation time t, {Ob_TP₁^j, Ob_TP₂^j, Ob_TP₃^j, Ob_TP₄^j} denotes transformed four points expressing O^[j], {St_TP₁ⁱ, St_TP₂ⁱ, St_TP₃ⁱ, St_TP₄ⁱ} denotes transformed four points expressing O_t^[i], and S_j denotes a matching error covariance for points that express O^[j].

8. The method of claim 7, wherein the similarity weight w_j is calculated by an equation:

w_j=p(O_t,Object|E)=p(O_t,id,O_t,pose|E)

, where O_t,Object denotes an object to recognize, O_t,id means whether recognized object is correct or not, O_t,pose means a precision level of an estimated object pose, and E denotes a measured feature.

9. The method of claim 8, wherein if a pose of the object is generated using a SIFT feature in the step a) and if O_id and O_pose are independent events, w_j is calculated by equation:

$\begin{array}{c}{w}_j=<br><br>p<br><br>\left(O_{t,\mathrm{Object}}|E_{\mathrm{SIFT}}\right)=p<br><br>\left(O_{t,\mathrm{id}},O_{t,\mathrm{pose}}|E_{\mathrm{SIFT}}\right)\\ =<br><br>p<br><br>\left(O_{t,\mathrm{id}}|E_{\mathrm{SIFT}}\right)<br><br>p<br><br>\left(O_{t,\mathrm{pose}}|E_{\mathrm{SIFT}}\right),\end{array}$

where the p(O_t,id|E_SIFT) is calculated using a sigmoide function based on the number of corresponding SIFTs through measurement, and the p(O_t,pose|E_SIFT) is defined as an average distance error.

10. The method of claim 8, wherein in the step a), if a pose of the object is generated using a line feature in the step a) and if O_id and O_pose are independent events, w_j is calculated by equation:

w_j=p(O_t,Object|E_Line)=p(O_t,id,O_t,pose|E_Line)=p(O_t,id|E_Line)p(O_t,pose|E_Line)

, where the p(O_t,id|E_Line) is defined by an equation ‘

Coverage=Matched_line_length/Total_line_length_of_model’ and

P(O_t,pose|E_Line) is defined as a matching error.

11. The method of claim 8, wherein if a location of the object is generated using a color feature in the step a), the similarity weight w_j is expressed in consideration of the number of color pixels matched with a model or as a predefined constant.

12. The method of claim 6, wherein a probability distribution of particles denoting an object pose at an estimated time t in the step b-2) is expressed as:

O_t^[i]˜

p(O_t|O_t-1^[i],u_t), (i=1, . . . , k)

, where u_t denotes camera motion control between a time t-1 to a time t, and k is the number of particles.

13. The method of claim 1, wherein in the step c), a result of the step b) is received, a ROI is generated from an input image, and illumination, texture density, and a distance between a robot and an expected object in the generated ROI are calculated.

14. The method of claim 13, wherein in the step d), a best evidence for object recognition and pose estimation is selected using information about the illumination, the texture density, and the distance between a robot and an expected object, which are collected in the step c).

15. The method of claim 14, wherein in the step d), an evidence having a highest P(Object|Evidence) is selected and the P(Object|Evidence) is calculated by an equation:

$P<br><br>\left(\mathrm{Object}|\mathrm{Evidence}\right)=\frac{1}{1+\frac{\begin{array}{c}P(\mathrm{Evidence}|P<br><br>\left(\stackrel{\_}{\mathrm{Object}}\right)·<br><br>\\ P<br><br>\left(\stackrel{\_}{\mathrm{Object}}\right)\end{array}}{\begin{array}{c}P<br><br>\left(\mathrm{Evidence}|\mathrm{Object}\right)·<br><br>\\ P<br><br>\left(\mathrm{Object}\right)\end{array}}},$

where P(Evidence|Object) denotes a probability of having corresponding evidence when the object exists (positive information) P(Evidence| Object) denotes a probability of having corresponding evidence when the object does not exists (negative information), P(Object) denotes a probability of having the object in a predetermined space (prior probability), and P( Object) denotes a probability of not having the object in a predetermined space.

16. The method of claim 15, wherein since the collected information about the illumination, the texture density, and the distance between a robot and an expected object in the step c) are independent events, P(Object|Evidence) is calculated by an equation:

P(E|O)=P(E_distance|O)·

P(E_texture|O)·

P(E_illumination|O)

, where P(E_distance|O) P(E_texture|O), and P(E_illumination|O) for each evidence are calculated by predefined probability distribution of each evidence according to illumination information, texture density information, and information about a distance between a robot and an expected object.

17. The method of claim 16, wherein the predefined probability distribution for each evidence has experimental probability distribution according to illumination information variation, texture density information variation, and variation of an estimated distance between a robot and an object.

18. The method of claim 16, wherein since the predefined probability distribution for each feature is an independent event,

a probability distribution for an evidence using a combination of a color feature and a SIFT feature is calculated by an equation:

P(E_Color+SIFT|Object)=P(E_Color|Object)·

P(E_SIFT|Object),

a probability distribution for an evidence using a combination of a color feature and a line feature is calculated by an equation;

P(E_Color+Line|Object)=P(E_Color|Object)·

P(E_Line|Object), and

a probability distribution for an evidence using a combination of a feature 1, a feature 2, and a feature 3 is calculated by an equation;

P(E_{Feature1+Feature2+Feature3+ . . .} |Object)=P(E_Feature1|Object)·

P(E_Feature2|Object)·

P(E_Feature3|Object)×

. . . .

19. The method of claim 15, wherein the P(Evidence| Object) (negative information) is a probability of having each feature of image in a space with no expected object and is calculated using background texture information and background color information, and

wherein if object recognition and pose estimation fail although object identification mission is performed using selected evidences, the negative information of an image feature used by the selected evidence increases not to exceed an upper bound of positive information so as to feedback a recognition result of a previous state to an evidence selecting and collecting unit in order to enable the evidence selecting and collecting unit to select an evidence proper to a corresponding environment.

20. The method of claim 14, wherein in the step d), an evidence having a minimum entropy value is selected and the entropy is calculated by an equation:

Entropy(i)=−

P(i)log₂P(i)−

(1−

P(i)log₂(1−

P(i)

,where P(i)=α

·

P(Object|Evidence)+(1−

α

)·

(1−

ExpTime_i/MaxTime_i) where α

denotes a weight for time consumption, ExpTime is a real execution time, MaxTime is a maximum consumption time based on an experimental recognition result, P(i) is an utility function, and P(Object|Evidence) denotes a probability of having the object when the evidence exists.

21. The method of claim 14, wherein if additional evidence is required because collected evidence is insufficient, a cognitive action of a robot is proposed to collect additional evidence by an evidence selecting and collecting unit.

22. A system for real time object recognition and pose estimation using in-situ monitoring, comprising:

an image capturing unit for continuously capturing an object to recognize with a real peripheral environment in various view points;

a real time environment monitoring unit for receiving 2D and 3D image information from the image capturing unit and calculating and collecting real time environmental information;

a multiple evidence extracting unit for extracting various evidences from the real time image information and generating various locations and poses of the object by comparing the extracted evidences with model information;

an evidence selecting and collecting unit for selecting a best evidence for the object and the peripheral environment thereof using the real time environmental information and the model information and proposing a predetermined action of a robot for collecting additional evidences; and

a probabilistic information fusion unit for estimating a location and a pose of the object through particle filtering of the various generated locations and poses and expressing the estimated location and pose in arbitrary distribution of particles.