Visually tracking an object in real world using 2D appearance and multicue depth estimations

• US 8,711,221 B2
• Filed: 12/09/2008
• Issued: 04/29/2014
• Est. Priority Date: 12/11/2007
• Status: Expired due to Fees

• Current Assignee: Honda Research Institute Europe GmbH (Honda Motor Company)
• Original Assignee: Honda Research Institute Europe GmbH (Honda Motor Company)
• Inventors: Eggert, Julian, Rebhan, Sven, Willert, Volker, Zhang, Chen
• Primary Examiner(s): Nguyen, Thuong
• Application Number:
  US12/331,202
• Publication Number:
  US 20090213219A1
• Time in Patent Office: 1,967 Days
• Field of Search: 348/148, 348/143, 348/E07.088, 348/E07.085, 382/103, 382/107, 382/104
• US Class Current: 348/148
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06T 7/285 using a sequence of stereo image pairs

Abstract

Estimating the dynamic states of a real-world object over time using a camera, two dimensional (2D) image information and a combination of different measurements of the distance between the object and the camera. The 2D image information is used to track a 2D position of the object as well as its 2D size of the appearance and change in the 2D size of the appearance of the object. In addition, the distance between the object and the camera is obtained from one or more direct depth measurements. The 2D position, the 2D size, and the depth of the object are coupled to obtain an improved estimation of three dimensional (3D) position and 3D velocity of the object. The object tracking apparatus uses the improved estimation to track real-world objects. The object tracking apparatus may be used on a moving platform such as a robot or a car with mounted cameras for a dynamic visual scene analysis.

Claims

1. A method for visually tracking a moving real-world object by estimating a three dimensional position and a two dimensional velocity of the moving object, the method comprising:

capturing a first image at a camera at a first time;

processing an image region of the first captured image to obtain first visual cues representing first input features of the moving object to be tracked, the image region in the first image being the region the moving object is expected to be located at the first time;

initializing a 2D tracker template based on the first input features;

capturing a second image at the camera at a second time subsequent to the first time;

processing the second captured image to obtain second visual cues representing second input features;

at a processor, determining a two dimensional position of the moving object to be tracked from the second captured image by estimating parameters of a two dimensional transformation on the determined object in the first captured image or the first input features that best match the determined object in the second captured image or the second input features;

determining a two dimensional velocity of the object to be tracked based on the position of the moving object in the first and second captured images;

estimating a depth of the moving object to be tracked at the first time based on a third cue obtained at the first time;

estimating a change of the depth of the moving object to be tracked between the first time and the second time based on the third cue and a fourth cue obtained at the second time;

determining a relative change of size of the moving object to be tracked from the first captured image and the determined object from the second captured image by performing a two dimensional transformation on the determined object in the first and second captured images or the first and second input features;

updating the estimated depth and the change of the depth of the moving object to be tracked based on the relative change of the size of the moving object to be tracked;

combining the updated depth of the moving object to be tracked, the change of the depth of the moving object to be tracked, two dimensional position of the moving object to be tracked, and the two dimensional velocity of the moving object to be tracked to obtain a three dimensional coordinate of the object to be tracked at the second time; and

outputting the three dimensional coordinate of the object to be tracked and tracking the moving object.

2. The method of claim 1, further comprising estimating the size of the object based on the three dimensional coordinate of the object.

3. The method of claims 1, wherein estimating the change of the depth of the object comprises integrating measurements associated with depth over time.

4. The method of claim 1, where estimating the change of the depth of the object comprises processing a plurality of depth estimations, each depth estimation obtained from different cues or measurement techniques.

5. The method of claims 1, wherein the two dimensional transformation is based on a change in a size of the appearance of the object indicating the change of depth in the object.

6. The method of claim 1, further comprising compensating motions of the camera or a platform on which the camera is mounted by taking into account changes in a position and an orientation of the camera in global space.

7. The method of claim 1, wherein the two dimensional position and the two dimensional velocity of the object are estimated using a probabilistic method.

8. The method of claims 1, further comprising processing higher-order derivatives of state parameters of the object.

9. The method of claim 1, further comprising sending the three dimensional coordinate of the object to a visual servoing unit controlling a field of view of the camera by controlling actuators.

10. The method of claim 1, wherein the depth of the object and the depth change of the object are estimated using Bayesian filters or Kalman filters.

11. The method of claim 1, wherein the two dimensional position of the object in the second image is determined by using a multicue Dynamic Bayesian Tracker.

12. A tracking apparatus including a processor and a memory storing instruction, the instructions when executed by the processor cause the processor to:

capture a first image at a camera at a first time;

processing an image region of the first captured image to obtain first visual cues representing first input features of the moving object to be tracked, the image region in the first image being the region the moving object is expected to be located at the first time;

initialize a 2D tracker template based on the first input features;

capture a second image at the camera at a second time subsequent to the first time;

process the second captured image to obtain second visual cues representing second input features;

determine a two dimensional position of the moving object to be tracked from the second captured image by estimating parameters of a two dimensional transformation on the determined object in the first captured image or the first input features that best match the determined object in the second captured image or the second input features;

estimate a depth of the moving object to be tracked at the first time based on a third cue obtained at the first time;

estimate a change of the depth of the moving object between the first time and the second time based on the third cue and a fourth cue obtained at the second time;

determine a relative change of size of the moving object to be tracked from the first captured image and the determined object from the second captured image by performing a two dimensional transformation on determined object in the first and second images or the first and second input features;

update the estimated depth and the change of the depth of the moving object to be tracked based on the relative change of the size of the moving object to be tracked;

combine the updated depth of the moving object to be tracked, the change of the depth of the moving object to be tracked, two dimensional position of the moving object to be tracked, and the two dimensional velocity of the moving object to obtain a three dimensional coordinate of the object at the second time; and

output the three dimensional coordinate of the object to be tracked and tracking the moving object.

13. A humanoid robot including a processor and a memory storing instruction, the instructions when executed by the processor cause the processor to:

capture a first image at a camera at a first time;

process an image region of the first captured image to obtain first visual cues representing first input features of the moving object to be tracked, the image region in the first image being the region the moving object is expected to be located at the first time;

initialize a 2D tracker template based on the first input features;

capture a second image at the camera at a second time subsequent to the first time;

process the second captured image to obtain second visual cues representing second input features;

determine a two dimensional position of the moving object to be tracked from the second captured image by estimating parameters of a two dimensional transformation on the determined object in the first captured image or the first input features that best match the determined object in the second captured image or the second input features;

determine a two dimensional velocity of the object to be tracked based on the position of the moving object in the first and second captured images;

estimate a depth of the moving object to be tracked at the first time based on a third cue obtained at the first time;

estimate a change of the depth of the moving object to be tracked between the first time and the second time based on the third cue and a fourth cue obtained at the second time;

determine a relative change of size of the moving object to be tracked from the first captured image and the determined object from the second captured image by performing a two dimensional transformation on the determined object in the first and second captured images or the first and second input features;

update the estimated depth and the change of the depth of the moving object to be tracked based on the relative change of the size of the moving object to be tracked;

combine the updated depth of the moving object to be tracked, the change of the depth of the moving object to be tracked, two dimensional position of the moving object to be tracked, and the two dimensional velocity of the moving object to be tracked to obtain a three dimensional coordinate of the object to be tracked at the second time; and

output the three dimensional coordinate of the object to be tracked and tracking the moving object.

14. The tracking apparatus of claim 13, wherein the depth of the object and the depth change of the object are estimated using Bayesian filters or Kalman filters.

15. An automobile including a processor and a memory storing instruction, the instructions when executed by the processor cause the processor to:

capture a first image at a camera at a first time;

process an image region of the first captured image to obtain first visual cues representing first input features of the moving object to be tracked, the image region in the first image being the region the moving object is expected to be located at the first time;

initialize a 2D tracker template based on the first input features;

capture a second image at the camera at a second time subsequent to the first time;

process the second captured image to obtain second visual cues representing second input features;

determine a two dimensional position of the moving object to be tracked from the second captured image by estimating parameters of a two dimensional transformation on the determined object in the first captured image or the first input features that best match the determined object in the second captured image or the second input features;

determine a two dimensional velocity of the object to be tracked based on the position of the moving object in the first and second captured images;

estimate a depth of the moving object to be tracked at the first time based on a third cue obtained at the first time;

estimate a change of the depth of the moving object to be tracked between the first time and the second time based on the third cue and a fourth cue obtained at the second time;

determine a relative change of size of the moving object to be tracked from the first captured image and the determined object from the second captured image by performing a two dimensional transformation on the determined object in the first and second captured images or the first and second input features;

update the estimated depth and the change of the depth of the moving object to be tracked based on the relative change of the size of the moving object to be tracked;

combine the updated depth of the moving object to be tracked, the change of the depth of the moving object to be tracked, two dimensional position of the moving object to be tracked, and the two dimensional velocity of the moving object to be tracked to obtain a three dimensional coordinate of the object to be tracked at the second time; and

output the three dimensional coordinate of the object to be tracked and tracking the moving object.

16. The automobile of claim 15, wherein the depth of the object and the depth change of the object are estimated using Bayesian filters or Kalman filters.

17. A non-transitory computer readable storage medium configured to store instructions, the instructions when executed by a processor cause the processor to:

capture a first image at a camera at a first time;

process an image region of the first captured image to obtain first visual cues representing first input features of the moving object to be tracked, the image region in the first image being the region the moving object is expected to be located at the first time;

initialize a 2D tracker template based on the first input features;

capture a second image at the camera at a second time subsequent to the first time;

process the second captured image to obtain second visual cues representing second input features;

determine a two dimensional position of the moving object to be tracked from the second captured image by estimating parameters of a two dimensional transformation on the determined object in the first captured image or the first input features that best match the determined object in the second captured image or the second input features;

determine a two dimensional velocity of the object to be tracked based on the position of the moving object in the first and second captured images;

estimate a depth of the moving object to be tracked at the first time based on a third cue obtained at the first time;

estimate a change of the depth of the moving object to be tracked between the first time and the second time based on the third cue and a fourth cue obtained at the second time;

determine a relative change of size of the moving object to be tracked from the first captured image and the determined object from the second captured image by performing a two dimensional transformation on the determined object in the first and second captured images or the first and second input features;

update the estimated depth and the change of the depth of the moving object to be tracked based on the relative change of the size of the moving object to be tracked;

combine the updated depth of the moving object to be tracked, the change of the depth of the moving object to be tracked, two dimensional position of the moving object to be tracked, and the two dimensional velocity of the moving object to be tracked to obtain a three dimensional coordinate of the object to be tracked at the second time; and

output the three dimensional coordinate of the object to be tracked and tracking the moving object.

18. The computer readable storage medium of claim 17, wherein the depth of the object and the depth change of the object are estimated using Bayesian filters or Kalman filters.