Audience measurement system and method

• US 5,550,928 A
• Filed: 12/15/1992
• Issued: 08/27/1996
• Est. Priority Date: 12/15/1992
• Status: Expired due to Term

• Current Assignee: Nielsen Company (US), LLC
• Original Assignee: ACNielsen Corporation (MOOD Media Corporation)
• Inventors: Lee, Morris, Thomas, William L., Bernard, Bruce, Lu, Daozheng, Zhang, Jia, Shagrin, Ceril T.
• Primary Examiner(s): Razavi, Michael T.
• Assistant Examiner(s): NOT, DEFINED
• Application Number:
  US07/992,383
• Time in Patent Office: 1,351 Days
• Field of Search: 382/2, 382/1, 382/16, 382/25, 382/30, 382/115, 382/116, 382/118, 382/203, 382/209, 382/190, 382/103, 358/84, 358/108, 340/825.34, 348/1, 348/2, 348/5
• US Class Current: 382/116
• CPC Class Codes: G06V 20/53 Recognition of crowd images, e.g. recognition of crowd congestion
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: H04H 60/45 for identifying users
• CPC Class Codes: H04N 21/42201 biosensors, e.g. heat sensor for presence detection, EEG sensors or any limb activity sensors worn by the user input arrangements for interaction with the human body based on nervous system activity detection G06F3/015

Abstract

In a passive identification apparatus for identifying a predetermined individual member of a television viewing audience in a monitored viewing area, a video image of a monitored viewing area is captured. A template matching score is provided for an object in the video image. A Eigenface recognition score is provided for an object in the video image. These scores may be provided by comparing objects in the video image to reference files. The template matching score and the Eigenface recognition score are fused to form a composite identification record from which a viewer may be identified. Body shape matching, viewer tracking, viewer sensing, and/or historical data may be used to assist in viewer identification. The reference files may be updated as recognition scores decline.

Claims

1. An image recognition apparatus for passively identifying individuals in a monitored area comprising:

means for storing a first set of reference facial image signatures wherein each reference facial image signature in the first set corresponds to a predetermined one of said individuals and is formed from an initial image of a predetermined individual by a first facial recognition methodology;

means for storing a second set of reference facial image signatures wherein each reference facial image signature in the second set corresponds to a predetermined one of said individuals and is formed from an initial image of a predetermined individual by a second facial recognition methodology which is different from the first facial recognition methodology;

image capturing means for capturing video images of a monitored area;

means for extracting a first current facial image signature from the video image by processing the video images and by utilizing the first facial recognition methodology and for providing a first set of identity-indicating scores by comparing the first current facial image signature to each reference facial image signature of the first set of reference facial image signatures;

means for extracting a second current facial image signature from the video image by processing the video images and by utilizing the second facial recognition methodology and for providing a second set of identity-indicating scores by comparing the second current facial image signature to each reference facial image signature of the second set of reference facial image signatures; and

,

means for fusing the first and second sets of identity-indicating scores to form a set of composite identity-indicating scores from which individuals may be identified.

2. The image recognition apparatus of claim 1 wherein the first set of reference facial image signatures are reference template signatures and wherein the means for extracting a first current facial image signature comprises means for extracting a current image template signature and for providing the first set of identity-indicating scores by comparing the current image template signature to each of the reference template signatures.

3. The image recognition apparatus of claim 1 wherein the second set of reference facial image signatures are reference Eigenface signatures and wherein the means for extracting a second current facial image signature comprises means for extracting a current image Eigenface signature and for providing the second set of identity-indicating scores by comparing the current image Eigenface signature to each of the reference Eigenface signatures.

4. The image recognition apparatus of claim 3 wherein the means for fusing comprises means for employing a linear discriminant function to form the set of composite identity-indicating scores.

5. The image recognition apparatus of claim 1 wherein the means for fusing comprises means for employing at least a quadratic discriminant function to form the set of composite identity-indicating scores.

6. The image recognition apparatus of claim 1 further comprising:

means for storing reference body shape signatures, wherein each reference body shape signature corresponds to a predetermined one of the individuals and is formed from an initial image of a predetermined individual by a body shape recognition methodology; and

,

means for extracting a current body shape signature from the video images by utilizing the body shape recognition methodology and for providing a third set of identity-indicating scores by comparing the current body shape signature to each reference body shape signature.

7. The image recognition apparatus of claim 6 wherein the first set of reference facial image signatures are reference template signatures and wherein the means for extracting a first current facial image signature comprises means for extracting a current image template signature and for providing the first set of identity-indicating scores by comparing the current image template signature to each of the reference template signatures.

8. The image recognition apparatus of claim 6 wherein the second set of reference facial image signatures are reference Eigenface signatures and wherein the means for extracting a second current facial image signature comprises means for extracting a current image Eigenface signature and for providing the second set of identity-indicating scores by comparing the current image Eigenface signature to each of the reference Eigenface signatures.

9. The image recognition apparatus of claim 6 wherein the means for fusing comprises means for employing a polynomial discriminant function to form the set of composite identity-indicating scores.

10. An image recognition system for identifying an individual in a monitored area comprising:

means for storing a plurality of reference facial image signatures and a plurality of reference body shape signatures, each stored reference facial image signature and each reference body shape signature corresponding to a predetermined individual;

video camera apparatus adapted to capture a current image of an individual in the monitored area;

means responsive to the video camera apparatus for extracting a current facial image signature from the current image, for extracting a current body shape signature from the current image, for comparing the current facial image signature with the stored reference facial image signatures to thereby generate a first set of scores wherein each score of the first set of scores represents a degree of agreement between the current facial image signature and a corresponding stored reference facial signature, for comparing the current body shape signature with the stored reference body shape signatures to thereby generate a second set of scores wherein each score of the second set of scores represents a degree of agreement between the current body shape signature and a corresponding stored reference body shape signature, for forming a composite set of scores from the first and second sets of scores, and for selecting a maximum score from the composite set of scores.

11. The image recognition system of claim 10 further comprising:

an historical record related to a probability that the individual is present in the monitored area; and

,

means for estimating, from the historical record, a probability that the individual is present in the monitored area;

wherein the means responsive to the video camera apparatus for extracting a current facial image signature from the current image determines the identity of the predetermined individual from the first and second sets of scores and from the probability that the predetermined individual is present in the monitored area.

12. The image recognition system of claim 10 further comprising:

sensing means for sensing the presence of an individual in the monitored area; and

,

means responsive to the sensing means for estimating the number of individuals present in the monitored area;

wherein the means responsive to the video camera apparatus for extracting a current facial image signature from the current image determines the identity of the predetermined individual from the first and second sets of scores and from the estimated number of individuals present in the monitored area.

13. A system for identifying predetermined persons in a monitored area, the persons having established viewing or listening habits during past days, the system comprising:

an historical record of the viewing or listening habits of the predetermined persons during past days;

means for forming, from the historical record, a first probability estimate that predetermined persons are present in the monitored area;

means for storing a plurality of reference facial image signatures wherein each of the reference facial image signatures corresponds to a predetermined person;

means for capturing a current image of the monitored area;

means for extracting a current facial image signature from the current image;

means for comparing the current facial image signature with the reference facial image signatures to form a second probability estimate that predetermined persons are present in the monitored area; and

,

means for identifying predetermined persons from the first and second probability estimates.

14. The system of claim 13 further comprising means for locating and tracking a person in the monitored area.

15. A method for determining that a predetermined individual is present in a monitored area during a predetermined time interval, the method comprising the following steps:

a) forming a first set of reference facial image signatures wherein each reference facial image signature of the first set is extracted from an initial image of a plurality of individuals according to a first methodology;

b) forming a second set of reference facial image signatures wherein each reference facial image signature of the second set is extracted from an initial image of the plurality of individuals according to a second methodology which is different from the first methodology;

c) capturing a current image of the monitored area;

d) locating a face of an individual from the current image;

e) extracting a first current facial image signature from the located face by use of the first methodology;

f) comparing the first current facial image signature with the first set of reference facial image signatures to generate a first set of scores;

g) extracting a second current facial image signature from the located face by use of the second methodology;

h) comparing the second current facial image signature with the second set of reference facial image signatures to generate a second set of scores;

i) combining the first and the second sets of scores to form a composite set of scores;

j) determining if the predetermined individual is present in the monitored area from the composite set of scores.

16. The method of claim 15 further comprising the steps of locating and tracking an individual in the monitored area.

17. A method for tracking an individual within a monitored area comprising the following steps:

a) forming a first reference facial image signature related to the individual according to a first methodology;

b) forming a second reference facial image signature related to the individual according to a second methodology which is different from the first methodology;

c) obtaining a current image and a set of subsequent images of the monitored area;

d) locating a current facial image of the individual in the current image;

e) extracting a first current facial image signature from the current facial image by use of the first methodology;

f) comparing the first current facial image signature with the first reference facial image signature to generate a first score;

g) extracting a second current facial image signature from the current facial image by use of the second methodology;

h) comparing the second current facial image signature with the second reference facial image signature to generate a second score;

i) identifying the individual from the first and second scores; and

,

j) tracking the identified individual from the current image through at least some of the subsequent images.

18. The method of claim 17 wherein the step of forming a first reference facial image signature related to the individual according to a first methodology comprises the step of forming a reference template signature, wherein the step of extracting a first current facial image signature from the current facial image by use of the first methodology comprises the step of extracting a current image template signature, and wherein the step of comparing the first current facial image signature with the first reference facial image signature to generate a first score comprises the step of comparing the current image template signature to the reference template signature.

19. The method of claim 17 wherein the step of forming a second reference facial image signature related to the individual according to a second methodology comprises the step of forming a reference Eigenface signature, wherein the step of extracting a second current facial image signature from the current facial image by use of the second methodology comprises the step of extracting a current image Eigenface signature, and wherein the step of comparing the second current facial image signature with the second reference facial image signature to generate a second score comprises the step of comparing the current image Eigenface signature to the reference Eigenface signature.

20. The method of claim 17 wherein the step of identifying the individual from the first and second scores comprises the step of employing a polynomial discriminant function to form a third composite score from the first and second scores.

21. The method of claim 20 comprising the further following steps;

detecting declining scores as current facial image signatures are compared with the first reference facial image signature; and

,

replacing the first reference facial image signature with a current facial image signature when the declining scores are below a predetermined value.

22. The method of claim 17 comprising the further following steps;

detecting declining scores as current facial image signatures are compared with the first reference facial image signature; and

,

replacing the first reference facial image signature with a current facial image signature when the declining scores are below a predetermined value.