Visual tracking using depth data

• US 7,372,977 B2
• Filed: 05/28/2004
• Issued: 05/13/2008
• Est. Priority Date: 05/29/2003
• Status: Active Grant

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: Fujimura, Kikuo, Nanda, Harsh
• Primary Examiner(s): Bella; Matthew C.
• Assistant Examiner(s): WANG, CLAIRE X
• Application Number:
  US10/857,581
• Publication Number:
  US 20050031166A1
• Time in Patent Office: 1,446 Days
• Field of Search: None
• US Class Current: 382/103
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/107 Static hand or arm

Abstract

Real-time visual tracking using depth sensing camera technology, results in illumination-invariant tracking performance. Depth sensing (time-of-flight) cameras provide real-time depth and color images of the same scene. Depth windows regulate the tracked area by controlling shutter speed. A potential field is derived from the depth image data to provide edge information of the tracked target. A mathematically representable contour can model the tracked target. Based on the depth data, determining a best fit between the contour and the edge of the tracked target provides position information for tracking. Applications using depth sensor based visual tracking include head tracking, hand tracking, body-pose estimation, robotic command determination, and other human-computer interaction systems.

Claims

1. A computer based method for tracking a target using a depth sensing camera, the method comprising the steps of:

receiving a depth image of an area including the target to be tracked;

determining an edge of the target in the depth image; and

determining a best fit of a mathematically representable contour with respect to the determined edge corresponding to the target to be tracked, wherein determining a best fit comprises;

dividing the mathematically representable contour into a plurality of segments, each contour segment comprising an inner side corresponding to lower depth values and an outer side corresponding to higher depth values; and

matching an orientation of each contour segment against a corresponding edge segment of the determined edge in the depth image by comparing the depth values on each side of the determined edge segment and determining that the depth values on a side of the determined depth segment that overlaps with the inner side of the matched contour segment are lower than the depth values of a side of the determined edge segment overlapping with the outer side of the matched contour segment.

2. The computer based method of claim 1, wherein determining the edge comprises comparing depth values of a plurality of depth image sections to find edge depth image sections having depth variations above a maximum threshold with respect to neighboring depth image sections.

3. The method of claim 2, wherein determining the edge further comprises:

assigning a reference value to the edge depth image sections; and

assigning a value to neighboring depth image sections by applying a transform to calculate a metric from the closest edge depth image section.

4. The method of claim 3, wherein the depth image sections are depth image pixels.

5. The method of claim 1, wherein determining the edge comprises calculating a depth gradient vector.

6. The method of claim 1, wherein the mathematically representable contour is a graphically representable mathematical function.

7. The method of claim 6, wherein the mathematically representable contour is one of the group consisting of a parallelogram, an oval, an ellipse, a circle, and a curve.

8. The method of claim 1, wherein the target includes one or more body parts.

9. The method of claim 8, wherein the one or more body parts include a human head.

10. The method of claim 1, wherein the determining a best fit further comprises:

calculating a plurality of match values, one match value for each state of a set of possible states, the match values corresponding to an overall distance metric between each contour segment of the mathematically representable contour and the corresponding segment of the determined edge of the target in the depth image;

comparing the plurality of match values with matched orientations; and

selecting the state with the match value corresponding the distance metric signifying the smallest distance between the contour and the edge of the target.

11. The method of claim 1 further comprising:

determining a search area of the depth image in which to try different locations for the outline for determining the best fit.

12. The method of claim 11, wherein determining the search area comprises predicting a target position for a next image frame based on a constant velocity assumption for movement of the target.

13. A computer readable medium for tracking a target using a depth sensing camera comprising a computer program that when executed by a computer processor implements the steps of:

receiving a depth image of an area including the target to be tracked;

determining an edge of the target in the depth image; and

determining a best fit of a mathematically representable contour with respect to the determined edge corresponding to the target to be tracked, wherein determining a best fit comprises;

dividing the mathematically representable contour into a plurality of segments, each contour segment comprising an inner side corresponding to lower depth values and an outer side corresponding to higher depth values; and

matching an orientation of each contour segment against a corresponding edge segment of the determined edge in the depth image by comparing the depth values on each side of the determined edge segment and determining that the depth values on a side of the determined depth segment that overlaps with the inner side of the matched contour segment are lower than the depth values of a side of the determined edge segment overlapping with the outer side of the matched contour segment.

14. A system for tracking a target using a depth sensing camera, the system comprising:

means for receiving a depth image of an area including the target to be tracked;

means for determining an edge of the target in the depth image; and

means for determining a best fit of a mathematically representable contour with respect to the determined edge corresponding to the target to be tracked, wherein determining a best fit comprises;

dividing the mathematically representable contour into a plurality of segments, each contour segment comprising an inner side corresponding to lower depth values and an outer side corresponding to higher depth values; and

matching an orientation of each contour segment against a corresponding edge segment of the determined edge in the depth image by comparing the depth values on each side of the determined edge segment and determining that the depth values on a side of the determined depth segment that overlaps with the inner side of the matched contour segment are lower than the depth values of a side of the determined edge segment overlapping with the outer side of the matched contour segment.

15. A human-computer interaction system for visually tracking human movement comprising:

an active depth sensor for capturing depth images of human body parts;

a processing unit coupled to the active depth sensor for receiving the depth images, the processing unit comprising;

an edge detection module; and

a tracking module coupled to the edge detection module to determine a best fit of a mathematically representable contour with respect to one or more body parts to determine a position of the one or more body parts within the captured depth images, the tracking module further comprising computer instructions for;

dividing the mathematically representable contour into a plurality of segments, each contour segment comprising an inner side corresponding to lower depth values and an outer side corresponding to higher depth values; and

matching an orientation of each contour segment against a corresponding edge segment of the determined edge in the depth image by comparing the depth values on each side of the determined edge segment and determining that the depth values on a side of the determined depth segment that overlaps with the inner side of the matched contour segment are lower than the depth values of a side of the determined edge segment overlapping with the outer side of the matched contour segment.