Robust multi-modal method for recognizing objects

• US 6,118,887 A
• Filed: 10/10/1997
• Issued: 09/12/2000
• Est. Priority Date: 10/10/1997
• Status: Expired due to Term

• Current Assignee: AT&T Corporation (AT&T, Inc.)
• Original Assignee: AT&T Corporation (AT&T, Inc.)
• Inventors: Potamianos, Gerasimos, Graf, Hans Peter, Cosatto, Eric
• Primary Examiner(s): Au, Amelia
• Assistant Examiner(s): AHMED, SAMIR ANWAR
• Application Number:
  US08/948,750
• Time in Patent Office: 1,068 Days
• Field of Search: 382/277, 382/230, 382/304, 382/118, 382/103, 382/107, 382/218, 382/173, 348/169
• US Class Current: 382/103
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation

Abstract

A method for tracking heads and faces is disclosed wherein a variety of different representation models can be used to define individual heads and facial features in a multi-channel capable tracking algorithm. The representation models generated by the channels during a sequence of frames are ultimately combined into a representation comprising a highly robust and accurate tracked output. In a preferred embodiment, the method conducts an initial overview procedure to establish the optimal tracking strategy to be used in light of the particular characteristics of the tracking application.

Claims

1. A method for tracking heads and faces, comprising the steps of:

activating a channel for collecting data comprising perceived locations of designated features of one of heads and faces;

collecting the data for each feature during a sequence of frames;

generating, for each feature, one or more representation models based on the collected data, wherein for at least one feature, complementary representation models are generated, and wherein each complementary representation model comprises data reflecting the perceived location of the feature to which it corresponds;

comparing the complementary representation models corresponding to the at least one feature to generate correlated data; and

combining the correlated data into a single representation, wherein said comparing step comprises the steps of;

defining a distance metric for each of the complementary representation models corresponding to the at least one feature;

positioning the complementary representation models adjacent a common interface;

measuring the mutual overlap of the complementary representation models; and

collecting, based on the overlap, information representing areas of correlation between the complementary representation models.

2. The method of claim 1, wherein said combining step comprises an n-gram search.

3. The method of claim 1, wherein the single representation is further combined with at least one representation from a second active channel.

4. The method of claim 1, wherein the complementary representation models comprise a bounding box.

5. The method of claim 4, wherein the complementary representation models further comprise a pixel map.

6. The method of claim 1, further comprising the step of:

generating, for each feature lacking corresponding complementary models, a unitary representation model based on the data collected for each such feature, wherein each unitary model comprises data reflecting the perceived location of the feature to which it corresponds;

combining the single representation with each unitary model to form a tracked output.

7. The method of claim 6, wherein said combining of the single representation with each unitary model comprises an n-gram search.

8. The method of claim 1, wherein the complementary models are predetermined pursuant to an optimal tracking strategy.

9. The method of claim 6, wherein the complementary models are predetermined pursuant to an optimal tracking strategy.

10. The method of claim 6, wherein each unitary model is predetermined pursuant to an optimal tracking strategy.

11. The method of claim 8, wherein the optimal tracking strategy is determined by representation models obtained from an initial overview sequence.

12. The method of claim 10, wherein the optimal tracking strategy is determined by representation models obtained from an initial overview sequence.

13. A method for locating heads and faces in a sequence of frames of images, comprising the steps of:

activating a plurality of channels for tracking the heads and faces;

gathering, by each channel, data from the tracked images during a sequence of frames;

generating, from data gathered by a first channel, a first group of complementary representation models comprising perceived locations of head and facial features;

comparing the first group of complementary representation models to generate a first intermediate representation comprising correlated data, and

combining the correlated data into a single representation, wherein said comparing step comprises the steps of;

positioning the complementary representation models adjacent a common interface;

retrieving a comparison function from memory;

selecting, based on the identity of the representation models, one or more distances metric;

measuring the mutual overlap between the representation models; and

storing the data correlating to the representation models.

14. The method of claim 13, wherein said combining step comprises an n-gram search.

15. The method of claim 13, further comprising the step of:

generating, for a second channel, a second group of complementary representation models comprising perceived locations of head and facial features to which the second group of complementary models corresponds.

16. The method of claim 15, further comprising the step of

comparing the second group of complementary models to generate a second intermediate representation comprising correlated data, wherein said second intermediate representation corresponds to head and facial features represented by the second complementary group;

combining the first intermediate representation with the second intermediate representation.

17. The method of claim 16, wherein said combining of the first and second intermediate representations comprises a tracked output.

18. A method for tracking facial features in images, comprising the steps of:

activating a fast channel;

collecting a first set of complementary representation models, by the first channel, of designated candidate facial features;

determining correlated data between the first set of complementary representation models;

generating a first intermediate representation based on the correlated data;

activating a second channel;

collecting a second set of complementary representation models, by the second channel, of designated candidate facial features;

measuring the correlated data between the second set of complementary representation models;

generating a second representation based on the correlated data; and

combining the first intermediate and second representations to from a tracked output.

19. The method of claim 18, wherein said determining step further comprises the steps of:

overlapping the first set of complementary representation models on a common interface;

computing the mutual overlap between the overlapping models; and

gathering correlated data based upon the overlap.

20. A method for tracking facial features in complex images, comprising the steps of:

activating a plurality of channels for performing an initial overview sequence;

generating, based on data gathered from the overview sequence, one or more representations comprising facial feature candidates;

terminating activity on the plurality of channels;

determining, based on the one or more representations, an optimal tracking strategy for the images to be tracked by selecting, for one or more additional facial features, representation models which correspond to each additional feature; and

reactivating selected channels of the plurality of channels for gathering data from the images to be tracked, wherein said determining step further comprises the steps of selecting, for designated facial features, complementary representation models which correspond to each designated feature, and for one or more additional facial features, unitary representation models which correspond to each additional feature; and

generating a first representation from the unitary models;

comparing the complementary representation models to generate a second representation comprising correlated data; and

combining the first and second representations.

21. The method of claim 20, wherein one channel of the plurality performs a shape analysis.

22. The method of claim 20, wherein another channel of the plurality performs a motion analysis.