Method and apparatus for robustly tracking objects

• US 7,317,812 B1
• Filed: 07/12/2003
• Issued: 01/08/2008
• Est. Priority Date: 11/15/2002
• Status: Active Grant

• Current Assignee: F Poszat Hu LLC (Intellectual Ventures LLC)
• Original Assignee: Videomining Corporation
• Inventors: Krahnstoever, Nils, Schapira, Emilio, Sharma, Rajeev
• Primary Examiner(s): Wu; Jingge
• Assistant Examiner(s): DULANEY, KATHLEEN YUAN
• Application Number:
  US10/618,393
• Time in Patent Office: 1,641 Days
• Field of Search: 382/154, 382/103
• US Class Current: 382/103
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 7/285 using a sequence of stereo image pairs
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

The present invention pertains generally to the field of computer graphics user interfaces. More specifically, the present invention discloses a video image based tracking system that allows a computer to robustly locate and track an object in three dimensions within the viewing area of two or more cameras. The preferred embodiment of the disclosed invention tracks a person'"'"'s appendages in 3D allowing touch free control of interactive devices but the method and apparatus can be used to perform a wide variety of video tracking tasks. The method uses at least two cameras that view the volume of space within which the object is being located and tracked. It operates by maintaining a large number of hypotheses about the actual 3D object location.

Claims

1. A method for visually tracking a target object in three dimensions, comprising the steps of:

a) capturing the target object with two or more imaging sensors,

b) maintaining a large number of 3D target location hypotheses,

c) initially distributing a first set of 3D target location hypotheses randomly in spaces viewed by said imaging sensors,

d) calculating locations of a second set of 3D target location hypotheses by adding offsets to locations of the first set of 3D target location hypotheses, wherein the offsets are randomly drawn between two numbers,

e) projecting each of the second set of 3D target location hypotheses from 3D space to 2D image spaces of said imaging sensors,

f) measuring confidences about a presence of the target object based on a combination of color and motion cues in each image captured by said imaging sensors, wherein the color and motion cues are spatially spread out using averaging before the color and motion cues are combined, and

g) combining the measured confidences of the second set of 3D target location hypotheses to obtain a 3D location of the target object,

wherein the 3D location of the target object is given by a weighted mean of all the second set of 3D target location hypotheses.

2. The method according to claim 1, wherein the step of e) projecting each of the second set of 3D target location hypotheses further comprises a step of utilizing projections that are obtained by calibrating said imaging sensors with respect to a reference coordinate system.

3. The method according to claim 1, wherein the method further comprises a step of calculating the color cues using a color model of the target object, wherein the color model of the target object is represented by a histogram that is estimated by collecting color samples of the target object.

4. The method according to claim 1, wherein the method further comprises a step of calculating the motion cues by measuring differences between the images captured sequentially by said imaging sensors.

5. The method according to claim 1, wherein the step of b) maintaining a large number of 3D target location hypotheses further comprises a step of creating a set of 3D target location hypotheses at each time step,

wherein weights in the weighted mean of all 3D target location hypotheses are given by said confidences.

6. An apparatus for visually tracking a target object in three dimensions, comprising:

a) means for capturing the target object with two or more imaging sensors,

b) means for maintaining a large number of 3D target location hypotheses,

c) means for initially distributing a first set of 3D target location hypotheses randomly in spaces viewed by said imaging sensors,

d) means for calculating locations of a second set of 3D target location hypotheses by adding offsets to locations of the first set of 3D target location hypotheses, wherein the offsets are randomly drawn between two numbers,

e) means for projecting each of the second set of 3D target location hypotheses from 3D space to 2D image spaces of said imaging sensors,

f) means for measuring confidences about a presence of the target object based on a combination of color and motion cues in each image captured by said imaging sensors, wherein the color and motion cues are spatially spread out using averaging before the color and motion cues are combined, and

g) means for combining the measured confidences of the second set of 3D target location hypotheses to obtain a 3D location of the target object,

wherein the 3D location of the target object is given by a weighted mean of all the second set of 3D target location hypotheses.

7. The apparatus according to claim 6, wherein the e) means for projecting each of the second set of 3D target location hypotheses further comprises means for utilizing projections that are obtained by calibrating said imaging sensors with respect to a reference coordinate system.

8. The apparatus according to claim 6, wherein the apparatus further comprises means for calculating the color cues using a color model of the target object, wherein the color model of the target object is represented by a histogram that is estimated by collecting color samples of the target object.

9. The apparatus according to claim 6, wherein the apparatus further comprises means for calculating the motion cues by measuring differences between the images captured sequentially by said imaging sensors.

10. The apparatus according to claim 6, wherein the b) means for maintaining a large number of 3D target location hypotheses further comprises means for creating a set of 3D target location hypotheses at each time step,

wherein weights in the weighted mean of all 3D target location hypotheses are given by said confidences.